PurveyorsBot

What the crawler fetches

The crawler reads publicly available collection feeds and product pages from green-coffee suppliers. It extracts product URLs, availability, price, origin, processing, certifications, and other product facts used to normalize listings across suppliers. It does not sign in to merchant accounts, add products to carts, or access checkout and customer routes.

Request policy

• Within each scraper process, Shopify requests use one shared queue with a maximum concurrency of one and a randomized 3–6 second baseline between request starts.
• The registered fleet is audited for an applicable Crawl-delay for PurveyorsBot or the wildcard user-agent. Any applicable delay is configured as a global minimum when it is more conservative than that baseline.
• HTTP 429 stops further Shopify traffic for the run. The crawler honors Retry-After and does not replay the same limited request through another transport.
• Durable fleet state carries a rate-limit hold across later runs and fails closed when that control-plane state is unavailable.
• Web Bot Auth signatures are generated after queue admission and refreshed for each request authority.

Broader robots.txt path-policy enforcement is being introduced in stages. Until that work is complete, an operator opt-out is treated as a deny rule and removed from the active supplier set.

How the data is used

Normalized product facts may appear in the public Purveyors catalog, analytics, and API. Raw storefront responses remain within Purveyors infrastructure and are not redistributed as source documents. Purveyors does not use crawler access to place orders or impersonate customers.