Automating web search and research

Consider a company that needs to find new suppliers of a particular type of equipment.

An employee receives the criteria: country, product type, technical specifications, price range, delivery terms, certification, and minimum order quantity. They then begin a conventional internet search.

They write several search engine queries and browse directories, manufacturers' websites, marketplaces, industry platforms, and publications. On each website, they have to find the right section, open product listings, read company descriptions, study PDF catalogs, and record details manually.

Some information is on the home page, some in technical documentation, and some in PDFs or spreadsheets. Prices may be missing or listed in different currencies. The same company may appear in several sources.

The employee copies the findings into Excel: company name, website, country, products, price, contact details, and comments. They then check the records, remove duplicates, clarify disputed data, and compare the options.

After several hours or days, a table or analytical brief is ready. But prices, product ranges, and terms may change a month later, requiring much of the research to be repeated.

A typical process looks like this:

The user gives the system a business task rather than a single search query. For example:

Find manufacturers of industrial equipment in EU countries.

Criteria:

• the required equipment category
• specific technical parameters
• delivery availability
• a listed price or the ability to request one
• an official website
• contact details

text-LLM-model can use these requirements to develop a search strategy: which queries to run, which types of sources to look for, and which data to extract.

The system then performs web searches automatically and obtains a set of potentially relevant pages. A browser or python-script opens the websites and extracts their content. If the required information is in a PDF, Word file, or another document, that file is processed as well.

text-LLM-model analyzes the content and converts unstructured information into a predefined structure. For example:

• Company
• Country
• Website
• Product type
• Model
• Specifications
• Price
• Currency
• Minimum order
• Delivery terms
• Contact details
• Source

A python-script then normalizes the data, bringing currencies, country names, measurement units, and other values into a consistent format.

The results are saved in PostgreSQL together with source links. This matters: the system must store not only the conclusion but also where each fact came from.

The next stage is comparing the results. Duplicates are removed, records for the same company are merged, and the options are checked against the specified criteria.

text-LLM-model can then prepare a comparative analysis: which options meet the requirements, how they differ, and which information needs additional verification.

The final output is generated automatically in Excel, Word, PDF, or another convenient format.

Instead of running dozens of searches and copying information, a person reviews the results that have already been collected and structured.

A company is looking for suppliers of a particular type of equipment.

Instead of several days of manual searching, the pipeline can produce a single consolidated table in a few minutes:

• Found: 47 companies
• After deduplication: 31
• Meet all criteria: 8
• Partially meet the criteria: 12
• Need additional verification: 11

Links to the original pages and documents are saved for each result.

• finding suppliers and manufacturers;
• competitor analysis;
• price monitoring;
• finding properties;
• selecting tenders and procurement opportunities;
• finding job openings or specialists;
• market analysis;
• collecting product specifications;
• finding company contact details;
• monitoring industry news;
• finding regulatory and technical documentation;
• collecting information from catalogs;
• analyzing scientific and professional publications;
• preparing reviews;
• regularly monitoring website changes.

Automation is especially useful when research has to be carried out continuously.

For example:

Or:

Web research then becomes a continuously running information pipeline rather than a one-off research task.

Automated research needs to do more than find information. It must also preserve the ability to verify it.

The pipeline can store:

• source URL
• retrieval date
• page title
• extracted passage
• identified value
• confidence level

If the same fact appears in several independent sources, the system can take this into account during verification.

If the data conflicts, instead of automatically choosing one value, the pipeline can flag the record as “verification required”.

Automation should therefore identify uncertainty rather than conceal it, and pass only the cases that require a decision to a person.