GuidesIntegrations

Scrapy

Route a Scrapy project through RiftIP with a few lines of settings.

Scrapy's built-in HttpProxyMiddleware reads the proxy from each request's meta. Set it once in a small downloader middleware:

Python

# middlewares.py
import os

class RiftIPProxy:
    def __init__(self):
        user, pw = os.environ["RIFTIP_USER"], os.environ["RIFTIP_PASS"]
        self.proxy = f"http://{user}-net-residential:{pw}@gate.riftip.com:7777"

    def process_request(self, request, spider):
        request.meta.setdefault("proxy", self.proxy)

Python

# settings.py
DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.RiftIPProxy": 350,
    "scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware": 400,
}
CONCURRENT_REQUESTS = 32
DOWNLOAD_TIMEOUT = 30
RETRY_TIMES = 3
RETRY_HTTP_CODES = [429, 502, 503, 504, 522]

#Per-request options

Override the proxy for a specific request — for example, to keep a session while walking a paginated list:

Python

yield scrapy.Request(url, meta={
    "proxy": f"http://{USER}-session-list_{list_id}-ttl-10m:{PASS}@gate.riftip.com:7777",
})

Tip

Turn on AUTOTHROTTLE_ENABLED for targets you haven't measured yet. It adapts your pace to the target's response times automatically.