Python Async Requests: Speed Up Web Scraping Without Breaking Rate Limits
Python web scrapers often become slow for a simple reason: a synchronous script waits for each response before it starts the next request. Async requests reduce that idle time by keeping several network requests in progress at once.
Speed has a limit. Raising concurrency too quickly can overload the target, trigger 429 rate limits, create connection errors, or return incomplete pages. In the Apify and The Web Scraping Club State of Web Scraping Report 2026, 65.8% of surveyed scraping professionals said they used more proxies in 2025 than the previous year. That reflects a common production problem: scaling a scraper also means managing the network route each request uses. For public pages, rotating residential proxies can avoid concentrating every request on one IP. For price checks, local listings, and other workflows that need one consistent region during a run, sticky sessions keep the connection more stable.
This guide follows one step-by-step aiohttp tutorial: send one request, run several requests concurrently, cap concurrency, add retries, and save results. Requests, asyncio.to_thread(), HTTPX, and NodeMaven Scraping Browser appear later as clearly labelled alternatives for different workflows.
When 429 errors, unstable routes, or one-IP limits appear, add clean residential proxies instead of pushing concurrency higher. Test NodeMaven with 750 MB of residential and mobile traffic for $3.50 and keep every request on a more reliable route
Python Async Requests: Quick Answer
- Follow the
aiohttptutorial below when you are building a new async scraper. - Keep Python Requests for a short local script or a small number of URLs.
- Pick
asyncio.to_thread()when you already have a working Requests scraper and want concurrent runs without rewriting it immediately. - Choose HTTPX when you prefer a Requests-like API with native async support.
- Move to browser automation when the target requires JavaScript, scrolling, clicks, forms, screenshots, or persistent browser state.
Start with a small concurrency limit and validate the returned pages before increasing it.
Why Normal Python Requests Get Slow
A normal Requests script is synchronous. It sends a request, waits for the response, and only then starts the next URL.
This example is not part of the async tutorial. It shows the traditional Requests approach, where URLs are downloaded one by one.
requests.get() sends one request. The for loop does not start the next URL until the previous response arrives. The aiohttp tutorial below changes that behavior by keeping several requests in progress at once.
Choose the Right Python Approach
Requests fits a one-off check, a small local script, or a short list of URLs.
asyncio.to_thread() runs existing blocking Requests code in threads. It gives an older synchronous scraper a path to concurrency without a full rewrite. Python documents it primarily for I/O-bound blocking functions, such as HTTP requests.
aiohttp is the main library in this tutorial. It is built around asyncio, handles many in-flight HTTP requests, and uses a reusable ClientSession with connection pooling. The aiohttp Client Quickstart advises against creating a new session for every request.
HTTPX provides another async route with an API that resembles Requests. Its async documentation and resource limit settings cover reusable clients, connection limits, and keep-alive connections.
NodeMaven Scraping Browser is for browser-based collection. It runs a real cloud browser for pages that need JavaScript rendering, scrolling, clicks, forms, screenshots, or saved browser sessions. A NodeMaven account can run up to 50 active browser sessions at the same time by default.
For a wider introduction to Requests, parsing, selectors, and browser automation, read NodeMaven’s Python web scraping guide.
When async requests cannot render the page, start a NodeMaven proxy trial and run up to 50 cloud browser sessions with Scraping Browser. The browser, profiles, and CAPTCHA solver have no separate browser fee: start with 750 MB of residential and mobile traffic for $3.50.
Step-by-Step: Build an Async Web Scraper With aiohttp
This tutorial uses Books to Scrape, a website created for scraping practice. The example first fetches one page, then several pages, then adds concurrency limits and retries.
Step 1: Create a Virtual Environment and Install aiohttp
Open Terminal in the folder where you want to save the project.
Create a file named async_scraper.py.
Step 2: Send One Async Request
Start with one page. This confirms that your environment, connection, and code work before concurrency enters the picture.
Run the file:
The terminal should print the opening part of the page HTML.
ClientSession() keeps a connection pool. Reusing the same session lets the scraper reuse connections instead of opening a new one for every page.
Step 3: Fetch Several Pages With asyncio.gather()
Replace the code with a version that downloads several category pages concurrently.
asyncio.gather() starts the request tasks together and waits until they all finish. This is fine for three URLs, but a large list needs a concurrency limit.
Step 4: Add a Concurrency Limit
A semaphore limits how many requests are active at once. The scraper can still process a large URL list, but only a defined number of connections remain open concurrently.
Here, the scraper has ten URLs but opens only five requests at a time. Start with 3, 5, or 10. Increase the number only after checking response quality and error rates.
Step 5: Add Retries and Save Results to CSV
The final version adds timeouts, exponential backoff for temporary failures, and a CSV output file.
Run it again:
The script creates scrape_results.csv in the same folder. In a production scraper, replace html_length with the fields you extract from each response, such as product title, price, stock status, listing URL, or JSON data.
Test Concurrency Before Scaling
Do not treat the highest possible concurrency number as the correct setting. A reliable scraper tracks successful, complete responses, not just completed requests.
Test a small URL sample at several limits:
| Concurrency | What to Record |
|---|---|
| 1 | Baseline response time and response quality |
| 5 | Success rate, 429 errors, and timeouts |
| 10 | Whether retries or blocks rise |
| 20+ | Whether the target still returns complete, valid pages |
Record total run time, success rate, retries, 403 and 429 responses, timeouts, and incomplete pages. Stop increasing concurrency when error rates rise or the returned content changes.
If the workflow needs more coverage after you have found a safe request rate, do not simply push the semaphore higher. For permitted public-data collection, rotating residential proxies can distribute independent requests across clean IPs. For region-specific checks or multi-page flows with cookies and pagination, use sticky residential sessions so the scraper keeps one consistent location during the job.
A GitHub discussion about high-volume HTTPX requests shows how thousands of concurrent requests can produce connection problems when client limits and connection pooling are not configured carefully.
When 429 errors, unstable routes, or one-IP limits appear, add clean residential proxies instead of pushing concurrency higher. Test NodeMaven with 750 MB of residential and mobile traffic for $3.50 and keep every request on a more reliable route
An Alternative for Existing Requests Scripts: asyncio.to_thread()
You do not need to rewrite a working Requests scraper on day one. asyncio.to_thread() can run blocking Requests functions concurrently.
This is a transition approach, not a replacement for an async HTTP client at large scale. A scraper that already relies on Requests can gain concurrency first, then move to aiohttp or HTTPX when it needs tighter control over connections and request behavior.
HTTPX AsyncClient: A Requests-Like Alternative
HTTPX resembles Requests but supports native async requests. It is a natural choice when your team already knows the Requests API.
Keep one shared AsyncClient for the batch. Creating a new client inside every request discards the benefits of connection pooling. In one HTTPX community benchmark discussion, client reuse produced performance close to aiohttp for that contributor’s test. Treat it as a developer example, not a universal benchmark.
Add Proxies to Async Python Requests
At a small scale, a normal connection may be enough. Repeated requests from a cloud server or a single IP can run into rate limits, location mismatches, or IP reputation checks.
For independent public pages, rotating residential proxies can distribute requests across clean consumer IPs. A residential proxy with a sticky session fits a job that needs a consistent location, cookies, or region during one collection run.
ISP proxies suit recurring monitoring where the same stable IP should be retained over time. For instance, a price tracker that checks a fixed product set every 15 minutes should not randomly change its network identity during an active workflow.
HTTPX can send requests through a proxy:
For async code, pass the same proxy value to httpx.AsyncClient. NodeMaven’s proxies for Python page covers common Python routing patterns, while the proxy authentication guide explains credentials and authentication methods.
Only collect data where access is permitted. Proxies do not grant permission to access private data, bypass paywalls, or ignore a website’s rules.
When Async Requests Are the Wrong Tool
Async HTTP requests work when the target returns the data in HTML or JSON. They cannot interact with a page the way a browser can.
Move to browser automation when the workflow needs:
- JavaScript-rendered content
- Scrolling, clicks, or form submissions
- Screenshots or visual checks
- Login state and saved cookies
- A browser profile that persists between runs
NodeMaven Scraping Browser provides a cloud browser with NodeMaven proxies, managed browser settings, persistent profiles, CAPTCHA support, Live Browser debugging, and session recordings. It has no separate browser usage fee for eligible NodeMaven customers. You pay for the rotating proxy traffic transferred during sessions.
For code-first browser workflows, read the Playwright scraping guide. For pages where rendering and visual interaction are not required, async HTTP requests are lighter to run.
Common Python Async Scraping Mistakes
Creating a new session per request: Create one aiohttp.ClientSession or httpx.AsyncClient for the batch. This preserves connection pooling and avoids repeatedly opening new connections.
Launching every task at once: asyncio.gather() has no built-in rate limit. Add a semaphore before running a large URL list. If a permitted workflow still needs more independent public requests after you have slowed it down, rotating residential proxies can spread requests across clean consumer IPs instead of concentrating them on one route.
Retrying every error: Retry timeouts, connection problems, 429 responses, and temporary 5xx errors with backoff. Do not repeatedly retry 400, 401, 403, or 404 responses without changing the request. A 403 can point to permissions, a blocked route, or anti-bot controls. NodeMaven’s 403 Forbidden guide explains how to diagnose those cases.
Ignoring the response body: A 200 OK response can still contain a CAPTCHA page, login page, access denied message, empty result, or wrong regional content. Validate an expected title, JSON field, or page marker before saving the result. Clean residential proxies can also reduce CAPTCHA and verification prompts on permitted public-data workflows because the traffic comes from consumer-network IP ranges rather than obvious hosting infrastructure. When a dataset needs a specific location, residential proxies can provide country, region, city, and ZIP-level routing.
Changing proxy location during a stateful workflow: Rotation suits independent requests. When the scraper relies on cookies, locations, pagination, or account state, keep one stable session. A sticky residential proxy works well for a defined collection run, while an ISP proxy fits recurring monitoring from one static IP.
Treating browser problems as HTTP problems: If the target needs JavaScript, scrolling, clicks, or form submissions, increasing async concurrency will not reveal the missing content. NodeMaven Scraping Browser runs real cloud browser sessions with managed proxies, persistent profiles, CAPTCHA support, and Live Browser debugging, so you can inspect the rendered workflow rather than guessing from an HTTP response.
When 429 errors, unstable routes, or one-IP limits appear, add clean residential proxies instead of pushing concurrency higher. Test NodeMaven with 750 MB of residential and mobile traffic for $3.50 and keep every request on a more reliable route
Conclusion
Build an async scraper in layers. Start with one async request, then add concurrent tasks, a semaphore, timeouts, retries, and response validation.
The key check is whether the returned pages are complete and correct. A fast scraper that collects block pages, wrong regional results, or missing records only creates more cleanup work.
For HTML and JSON endpoints, aiohttp or HTTPX can run high-volume workloads efficiently. When error rates rise after you have set a safe concurrency limit, rotating residential proxies can distribute independent public requests across clean IPs. Use sticky residential proxies or ISP proxies when cookies, location, pagination, or recurring monitoring require a stable session.
When the target needs JavaScript, persistent browser state, or interactive page behavior, move the workflow to NodeMaven Scraping Browser. It provides managed cloud browsers, proxy routing, CAPTCHA support, persistent profiles, and Live Browser debugging, with no separate browser usage fee beyond the proxy traffic transferred during each session.
