What Affects Web Scraping Success? 24,156 Page Visits Tested
What makes a scraper successfully return the page with content you asked for? We ran 24,156 page visits across Amazon, Walmart, Google Search, and Google Maps to find out.
We couldn’t establish that proxy-quality signals such as fraud score or blacklist status predicted scraping success. Most measurable paid IPs fell into the same low-risk range, while page delivery changed much more with the target website, test window, and browser workflow.
Other operational differences were measurable. Premium routes connected far more reliably than public free proxies. In the budget comparison, the premium pool kept its requested sticky IP more consistently, which supports workflows that need the same identity across several steps.
This report separates those measurements instead of compressing them into one “proxy success rate”. It shows what influenced usable-page delivery, where proxy quality affected the workflow, and how to test the same variables on your own scraping stack.
Key findings at a glance
Across the tests, web scraping success moved with the entire collection setup. Target website, browser, host machine, test window, location, and connection path all changed the result; each percentage belongs to the conditions in which we measured it.
| Factor | What the tests found |
| Target website | Shared premium runs ranged from 6-12% delivery on Google Search to 94-100% on Google Maps |
| Test window | Amazon delivery rose and fell for all premium pools during the same test windows |
| Browser setup | Changing the browser produced some of the largest differences on Walmart and Google |
| IP risk score | Most scored sessions were low risk in both studies: 655 of 677 in the initial test and 938 of 1,002 in the follow-up. The smaller higher-risk groups didn’t show consistently lower success rates. |
| Exit location | In one NodeMaven run, Amazon delivery was 7.5% through California exits and 22–30% through Texas, New York, and Florida. Walmart showed no clear state-level pattern |
| Session persistence | It didn’t improve one-page delivery in the controlled test, but it remains necessary for workflows that must keep one identity |
| Proxy speed | It affected job duration, not whether the requested content arrived |
| Operational quality | Free and paid pools differed sharply in connectivity, technical failures, and sticky-session reliability |
Taken together, these findings cover different parts of scraping. The target, test window, and browser changed how often the requested content arrived, while connection reliability and speed shaped how steadily and quickly the job ran. IP risk scores showed no consistent link to success.
Methodology: What did we test?
Across both studies, we analyzed 24,156 page visits. The initial test compared premium residential proxies with a public free-proxy list, while the follow-up used NodeMaven’s open-source proxy-benchmark to test more websites and configurations.
Phase one: premium and free proxy availability
The first study examined how proxy availability, connection reliability, session stability, speed, and IP reputation related to Amazon page delivery. It tested 774 premium US residential proxy assignments and 500 candidates from a public free-proxy list.
Requests ran through Playwright with headed Google Chrome on one VPS. Scamalytics and IPinfo supplied the supporting IP data.
Premium routes requested sticky sessions; the public endpoints offered no session guarantee.
Phase two: controlled follow-up tests
The follow-up used the open-source NodeMaven proxy-benchmark for 44 runs covering 19,461 page visits and around 4,500 proxy sessions on one Windows server.
It covered Amazon Search, Walmart Search, Google Search, and Google Maps. Most runs used headless Patchright and Cloak, with Camoufox in selected browser tests. The controlled sticky-session test used Chromium with a visible window.
We checked the exit IP at the start, middle, and end of each session, then recorded connection errors, speed, country, reputation data with Scamalytics, and the returned content. Provider requests alternated inside each run, keeping direct comparisons within the same test window.
Which variables changed web scraping success most?
Website, test window, and browser configuration produced the largest shifts in delivered content. When the premium pools ran side by side, their results generally stayed close.
The target website set the baseline
In the 18 follow-up runs shared by three premium pools, page delivery differed much more by website than by pool:
| Target | Observed delivery across premium pools |
| Google Maps | 94-100% |
| Walmart Search | 37-38% |
| Amazon Search | 26-28% |
| Google Search | 6-12% |
For every target, the premium pools finished within a few percentage points of one another. Changing the target from Google Maps to Google Search moved the result by more than 80 percentage points.
Amazon results moved with the test window
Amazon delivery shifted sharply between runs, with all three premium pools moving in the same direction:
| Test date | Premium pool A | Premium pool B | Premium pool C |
| September 22 | 12% | 10% | 9% |
| September 23 | 45% | 46% | 40% |
| September 29 | 54% | 62% | 54% |
Suppose pool A had run on September 22 and pool B on September 29. The result would resemble a large provider gap even though Amazon became easier for every tested premium pool at roughly the same time.
We couldn’t isolate the exact cause of that movement. It may have come from the hour, target-side controls, traffic conditions, or another shared variable.
The browser sometimes outweighed the proxy
Some of the widest gaps appeared when the browser configuration changed while the proxy pools stayed the same.
| Target | Browser | Page delivery |
| Walmart Search | Cloak | 73% |
| Walmart Search | Patchright | 0.2% |
| Google Search | Camoufox | 53% |
| Google Search | Cloak | 12% |
| Google Search | Patchright | 1% |
On Walmart, the gap between Cloak and Patchright reached 72.8 percentage points. Google Search showed a 52-point range between Camoufox and Patchright.
These results belong to the tested configurations and test windows. They show why browser setup should remain fixed when comparing proxies and be tested separately when page delivery is low.
In our comparison of paid Google SERP APIs and Google Browser Scrapert, we shared how Google also reacted to the steps taken before the query. Across four NodeMaven runs covering 703 searches, cold exits returned result pages in 20% of searches. Exits that completed a six-page warm-up returned them in 87% under the tested conditions.
NodeMaven’s open-source Google Browser Scraper uses that tested workflow. It warms a new sticky identity, enters the query through a real browser, keeps exits that Google accepts, and records CAPTCHA or block pages as failures instead of empty result sets.
Our separate browser automation benchmark article covers another experiment that held the surrounding setup steady while comparing eight browser configurations on Amazon.
Where did proxy quality make a measurable difference?
Across NodeMaven’s two proxy studies, the clearest operational differences appeared in connection reliability and session retention. Public free proxies produced technical failures on 66.8% of attempted visits. In matched follow-up runs, the budget pool retained its exit IP in 71.4% of readable sessions, compared with 98.1% for the premium pool.
| Quality measure | Test result | What it means |
| Free-proxy availability | Only 168 of 500 public candidates were usable, or 33.6% | Most candidates couldn’t be used for the Amazon test |
| Technical failures | Public free proxies: 66.8% in the initial study. Premium pools: 0.6–4.2% initially and 1.5–9.9% in the follow-up. Budget pool: 2.5% in the follow-up | Paid routes reached the target more reliably than the public list |
| Session retention | Budget pool: 71.4%. Premium pool tested beside it: 98.1% | An unexpected IP change can interrupt logins, pagination, carts, and longer browser sequences |
| Country accuracy | The requested country matched in 7,551 of 7,557 paid-pool visits with recorded location data | Five mismatches came from the budget pool and one from a premium pool |
The technical-failure figures came from separate test groups, so they don’t form a direct ranking of every paid pool. The session-retention figures provide the cleaner premium-versus-budget comparison because both pools ran during the same test windows.
A 2024 study of free proxy services, “Free Proxies Unmasked,” followed public proxies for 30 months and examined their stability, security, and traffic manipulation. Our test measured a narrower workflow. In that workflow, two-thirds of visits through the public list failed before usable content arrived.
Could reputation scores predict scraping success?
Neither study found a consistent relationship between reputation scores and usable-page delivery. In the initial study, 655 of 677 scored premium sessions were low risk. After accounting for the collection period, the correlation between risk score and delivery was approximately -0.02.
The follow-up produced a similar distribution: 938 of 1,002 scored sessions fell into the low-risk group, and none of the 706 sessions checked against the selected blacklists appeared there. Risk-score correlations with delivery ranged from -0.05 to 0.09 across Amazon, Walmart, and Google.
Too few sessions landed in the higher-risk groups for a reliable comparison. One controlled NodeMaven run also tested three quality-filter settings; Amazon delivery came to 16%, 17%, and 18%, while Walmart returned 62%, 54%, and 62%.
Most paid IPs sat in the same reputation range, leaving little variation for the score to explain. Connection failures, country accuracy, and IP retention described how each pool behaved during a real job more directly. A fraud score doesn’t show how the browser will perform or whether a five-step session will keep the same identity.
Do sticky sessions improve page delivery?
The controlled follow-up didn’t show that keeping one IP improved single-page Amazon delivery. Sticky sessions delivered 21 of 180 pages, or 11.7%; sessions that requested a new IP for every visit delivered 26 of 180, or 14.4%.
Sticky sessions still have a clear job in longer workflows. Logins, pagination, carts, and multi-step browser sequences may need one identity from start to finish. A rotating route can return the first page correctly and switch IPs before the next step.
Proxy speed changed how long the job took
Speed showed the same weak relationship with page delivery in both studies. In the initial study, correlations with content delivery were approximately -0.09 for latency and -0.10 for download time; no threshold separated delivered pages from failures.
The follow-up reached the same general result across more targets. Correlations between download speed and delivery ranged from -0.09 to 0.24 by website; connection-time correlations ranged from -0.24 to 0.11.
The budget pool reached a median of 1.7 Mbit/s in its shared runs with the premium pool, which reached 3.1 Mbit/s. Their one-page delivery rates stayed close, but the speed difference changed how long the work took.
Slow routes keep browser sessions and workers occupied for longer. The difference becomes more expensive on pages that load large scripts, images, or background requests, especially when thousands of sessions are running.
Why HTTP 200 is not a scraping success
Amazon sometimes returned its “too many requests” page with HTTP 200. Any benchmark that counted status codes alone would have recorded that block as a success.
The HTTP Semantics standard, RFC 9110 defines 200 as a successful HTTP request. It doesn’t verify that a scraper received the product listings, search results, or fields its parser expected.
We classified each returned page as one of the following:
- The requested page.
- A CAPTCHA or challenge.
- A consent screen.
- A block or throttle page.
- An empty response.
- A transport, proxy, or browser error.
Success required the expected content. HTTP status and connection health remained separate fields. Our guide to diagnosing a 403 Client Error in Python uses the same approach: a status code identifies the response, while controlled tests locate the source of the failure.
How should you test proxies for your own scraper?
A useful proxy test should resemble the job that will run in production. Use the same target and browser, then change one variable at a time.
- Test the production page type. A homepage, search page, product page, and authenticated account can behave differently on the same website.
- Run providers in the same window. Alternate configurations in small batches instead of completing one pool before starting the next.
- Hold the rest of the stack steady. Keep the browser, machine, location, request sequence, concurrency, and target unchanged.
- Validate the returned content. Define markers for the requested page, challenge, consent screen, throttle page, empty response, and error.
- Separate technical failures from target refusals. A broken tunnel and an anti-bot page require different fixes.
- Check location and session behavior. Verify that the exit matches the requested country and that sticky sessions retain one identity.
- Repeat the run and calculate the real cost. Compare job duration and cost per usable page across several test windows.
NodeMaven’s open-source proxy-benchmark runs these controlled comparisons across direct and proxied paths, providers, countries, browser engines, HTTP clients, settings, and hosts. It saves the configuration and content verdict for every attempt.
The repository works with NodeMaven and custom providers, so you can run the same matrix through your current proxy. For basic connection, location, and speed checks before the target benchmark, follow our step-by-step proxy testing guide.
Get 750 MB of residential and mobile proxy traffic for $3.50. Compare NodeMaven with your current setup using the same target, browser, and location.
Research limitations
One Windows server produced the follow-up data, and most runs used headless browsers. Another host, browser version, or collection window may return different rates.
The runs covered four weeks. Provider comparisons use shared runs because target behavior changed between windows. Several country, state, ISP, and Google Search subsets were small; two premium pools also exposed their exit IP in fewer sessions, leaving smaller stability samples.
Most paid IPs had already received low-risk classifications, so the research couldn’t build a balanced clean-versus-abused comparison. The controlled sticky-session experiment ran while Amazon refused roughly 85% of visits, which limited its ability to detect a small effect.
Apply the figures to the tested conditions. The reusable result is the testing method: interleave providers, inspect page content, and keep the rest of the stack fixed.
Research limitations
The follow-up ran on one Windows server, mostly with headless browsers. Provider comparisons use shared runs because results changed between test windows. Some location and Google Search samples were small, and two premium pools had fewer readable exit IPs for the stability analysis.
Most paid IPs were already low risk, so the research couldn’t form a balanced clean-versus-abused comparison. Amazon also refused about 85% of visits during the controlled sticky-session test, which may have hidden a small effect. Treat the percentages as specific to these conditions; repeat the method with your own target, browser, and proxy setup.
