Price monitoring at scale without tripping a single alarm

Price monitoring sounds simple: fetch a product page, read the price, store it. At ten products it is. At ten thousand products across twelve countries, refreshed every few hours, it becomes an exercise in looking unremarkable.

Here's how teams run it reliably.

#See the price a local shopper sees

Retailers localise more than currency. Prices, stock, delivery promises and promotions all change with the visitor's country — sometimes with their region. A request from a German exit can show a different promotion than one from France for the same product URL.

So the first rule is to fetch each market from inside that market:

curl -x "http://acme-net-residential-country-de:PASSWORD@gate.riftip.com:7777" https://store.example.com/p/air-runner

curl -x "http://acme-net-residential-country-fr:PASSWORD@gate.riftip.com:7777" https://store.example.com/p/air-runner

#Spread the load like real traffic

Retail sites watch for patterns, not just volume. A thousand requests for product pages in the same minute, from the same few IPs, in alphabetical order, is a pattern. The same thousand requests spread across an hour, from many exits, in shuffled order, is just traffic.

• Shuffle your URL list on every run so you never walk a catalogue in order.
• Rotate exits per request. Product pages don't need a session.
• Pace each domain. A steady few requests per second per retailer beats bursts.
• Back off on errors. A spike in 429s or challenges is a signal to slow down, not to retry harder.

#Pick the network per retailer

Not every retailer needs residential IPs. Many smaller stores serve product pages to anyone — datacenter exits work fine and cost a fraction as much. Large marketplaces are a different story: they invest heavily in bot detection, and residential exits are what keep success rates high.

A practical setup is to start every new retailer on datacenter, watch the success rate for a day, and promote only the ones that need it to residential.

#Validate what you parse

The most dangerous failure in price monitoring isn't a block — it's a wrong number that looks right. A challenge page, a regional redirect or a "price on request" placeholder can all parse into a plausible value.

1. Check that the page is the product you asked for (match the SKU or title).
2. Check the currency matches the market you targeted.
3. Flag price changes above a threshold for review before they reach a report.

#When volume grows

At a few hundred gigabytes a month of residential traffic, look at Unlimited Residential: a flat daily price with unmetered bandwidth, sized by concurrent threads. For steady, always-on monitoring it usually works out cheaper than paying per GB — and it removes the temptation to cut corners on refresh rate to save bandwidth.

Good price monitoring is boring by design: steady pace, local exits, validated data. Boring is what keeps it running every day for years.