Web enrichment architecture¶

Status: implemented (#842). Decision: ADR-0029. Reference: company pass and operations.

Goal¶

For a company with a CVR, collect evidenced information from search, Google AI Mode, and the company's own website: the website and social profiles, company contacts, what the company does and for whom, the people it names, and the companies it names as customers, suppliers, partners, or competitors. Every value keeps its source and an exact evidence locator.

Runtime view¶

flowchart LR
    CALLER[Backend or web-starter job] -->|POST /company-requests| SVC[web-enrichment service]
    SVC -->|runs + receipt| DB[(Postgres)]
    SVC -->|one task per run| Q1[web-enrich-batch]
    SVC -->|one task per run| Q2[web-enrich-ad-hoc]
    Q1 -->|POST /tasks/runs/ID| SVC
    Q2 -->|POST /tasks/runs/ID| SVC
    SCHED[Cloud Scheduler, every 10 min] -->|POST /tasks/sweep| SVC
    SVC --> DFS[DataForSEO live search and AI Mode]
    SVC --> OR[OpenRouter model]
    SVC --> OAI[OpenAI embeddings]
    SVC -->|through Webshare| WEB[Company websites]
    SVC --> GCS[(GCS raw evidence)]

One service does everything. A company pass runs inside the task request that delivers its run; each instance runs up to eight passes at the same time and shares one headless Chromium between them. Cloud Run adds instances when the queues hold work and scales to zero when they are empty.

The service reaches company websites through the Webshare proxy, which gives Danish addresses: many Danish web firewalls refuse cloud addresses, and a direct fetch spends the reputation of our own address. The direct route is used only when the proxy itself fails. The paid APIs are called directly.

One company pass¶

flowchart TD
    FACTS[Registry facts] --> LEADS[Leads: registry website and email domain]
    FACTS --> AI[Live AI Mode question]
    FACTS --> ORG[Live organic search]
    LEADS --> VISIT[Visit site: HTTP, plain HTTP, browser]
    AI -->|site named as the website| VISIT
    ORG -->|non-directory results| VISIT
    VISIT --> DECIDE{Ownership from registry facts on the site}
    DECIDE -->|clear| ACCEPT[Verified website]
    DECIDE -->|unclear| JUDGE[Model judges with the same facts]
    JUDGE --> ACCEPT
    DECIDE -->|refused| VISIT
    ACCEPT --> PAGES[Select up to 8 pages by topic]
    PAGES --> RETAIN[Store raw pages and search responses]
    RETAIN --> READ[Deterministic contacts, socials, technologies]
    RETAIN --> SITE[One model call: profile, classes, mentions, people]
    AI --> SOURCES[One model call: AI Mode signals]
    READ --> PUBLISH[One transaction to the published tables]
    SITE --> PUBLISH
    SOURCES --> PUBLISH
    PUBLISH --> EMBED[Profile embedding]

The AI Mode question starts at once. The organic search also starts at once when the registry names no website; otherwise it runs only when the registry leads fail. A pass checks at most four sites.

A site is checked at its root, not at the page a search returned, so a directory profile page cannot stand in for a website. The home page is fetched first; the contact and about pages linked from it are fetched only when the home page alone does not decide ownership.

Module map¶

Layer	Module	Responsibility
adapters	`adapters/web/fetch.py`	HTTP and browser fetching, redirects, TLS, proxy fallback
adapters	`adapters/web/dataforseo.py`, `llm.py`, `embeddings.py`	Provider calls with bounded retries
enrichment	`enrichment/web/facts.py`	Registry facts, name folding, registrable domains
enrichment	`enrichment/web/leads.py`	Leads from registry, AI Mode, search; directory exclusion
enrichment	`enrichment/web/identity.py`, `ownership.py`	Registry facts on a site; the ownership decision and judge prompt
enrichment	`enrichment/web/page.py`, `page_selection.py`	Text projection, link and sitemap page selection
enrichment	`enrichment/web/extraction.py`, `sources.py`	Model schemas and prompts, quote checks, evidence locators
orchestration	`orchestration/web/company_pass.py`	The pass and its log
orchestration	`orchestration/web/store.py`	Registry reads and the publication transaction
orchestration	`orchestration/web/runner.py`, `requests.py`, `tasks.py`	Claims and retries, admission and progress, Cloud Tasks
runtime	`runtime/web_service.py`, `web_starter.py`, `web_auth.py`	The service, the operator job, caller identity

The existing domain rules for contacts, people attribution and privacy, company mentions, technology signatures, and the embedding text recipe are reused unchanged (enrichment/contacts.py, people.py, mentions.py, technology.py, embedding_text.py).

Data¶

A run writes to the tables the read contracts already publish:

Table	Content
`company_enrichment_run`	Status, attempts, outcome, reason, and the pass log with costs
`web_evidence_bundle`, `web_evidence_bundle_page`	The sealed manifest of retained pages for the verified site, with the ownership decision
`web_company_presence`	Website and social profiles
`web_company_contact`	Company contacts with exact locators
`web_company_profile_signal`	Profile fields with quotes and locators
`company_web_filter_evaluation`, `web_extractor_result`, `web_extraction_application`	Classification and technology filters, extractor records
`web_entity_mention`, `entity_relationships`	Named companies and, after a unique active-company match, observed edges
`person_mentions`, `person_mention_contact`	Named people and the work contacts the page attributes to them
`source_retrieval`, `web_source_extraction`, `web_source_signal`	Search responses and the AI Mode signals read from them
`company_embedding`	The profile embedding

Raw evidence is stored create-only in GCS under raw/web/pages/ and raw/web/search/ before any model reads it. Locators name the object generation, digest, projection version, and character range.

Failure and retry¶

A duplicate or stale delivery finds the run claimed or finished and does nothing.
A provider fault inside a pass is retried by its adapter. A fault that remains becomes a named step error in the log; the pass continues with what it has.
An unexpected error releases the claim and returns HTTP 500; Cloud Tasks retries with backoff. The third failed attempt ends the run as failed.
A lost task is found by the sweep: a queued run untouched for ten minutes, or a claim older than twenty minutes, is enqueued again.
A retry may repeat searches and model calls (about USD 0.01).

Scale¶

A pass takes 10 to 60 seconds: on the deployed service a mean of 23 to 38 seconds and a 95th percentile of 41 to 64 seconds. The batch queue runs at most 40 passes at a time and dispatches at most four per second; each instance runs up to eight. A 300-company batch finished in 246 seconds (1.22 companies per second). Admission takes about one second for 500 companies. Queue rates and concurrency, instance concurrency, and the maximum instance count are Terraform values in infra/web_service.tf; raise the queue concurrency for more throughput, within the provider and proxy limits.

Provider cost per company is about USD 0.004 to 0.006 for search and AI Mode, plus one to three small model calls and one embedding.