{"id":41343,"date":"2026-10-16T10:08:55","date_gmt":"2026-10-16T10:08:55","guid":{"rendered":"https:\/\/nodemaven.com\/?p=41343"},"modified":"2026-10-16T10:16:02","modified_gmt":"2026-10-16T10:16:02","slug":"what-affects-web-scraping-success","status":"publish","type":"post","link":"https:\/\/nodemaven.com\/ru\/blog\/what-affects-web-scraping-success\/","title":{"rendered":"What Affects Web Scraping Success? 24,156 Page Visits Tested"},"content":{"rendered":"

What makes a scraper successfully return the page with content you asked for? We ran 24,156 page visits across Amazon, Walmart, Google Search, and Google Maps<\/strong> to find out.<\/p>\n\n\n\n

We couldn\u2019t establish that proxy-quality signals such as fraud score or blacklist status predicted scraping success. Most measurable paid IPs fell into the same low-risk range, while page delivery changed much more with the target website, test window, and browser workflow.<\/strong><\/p>\n\n\n\n

Other operational differences were measurable. Premium routes connected far more reliably than public free proxies.<\/strong> In the budget comparison, the premium pool kept its requested sticky IP more consistently<\/strong>, which supports workflows that need the same identity across several steps.<\/p>\n\n\n\n

This report separates those measurements instead of compressing them into one \u201cproxy success rate\u201d. It shows what influenced usable-page delivery, where proxy quality affected the workflow, and how to test the same variables on your own scraping stack.<\/p>\n\n\n\n

Key findings at a glance<\/strong><\/h2>\n\n\n\n

Across the tests, \u0432\u0435\u0431-\u0441\u043a\u0440\u0435\u0439\u043f\u0438\u043d\u0433<\/a> success moved with the entire collection setup<\/strong>. Target website, browser, host machine, test window, location, and connection path all changed the result; each percentage belongs to the conditions in which we measured it.<\/p>\n\n\n\n

\u0420\u0430\u0437\u043b\u043e\u0436\u0438\u0442\u044c \u043d\u0430 \u043c\u043d\u043e\u0436\u0438\u0442\u0435\u043b\u0438<\/strong><\/td>What the tests found<\/strong><\/td><\/tr>
\u0426\u0435\u043b\u0435\u0432\u043e\u0439 \u0441\u0430\u0439\u0442<\/td>Shared premium runs ranged from 6-12% delivery on Google Search to 94-100% on Google Maps<\/td><\/tr>
Test window<\/td>Amazon delivery rose and fell for all premium pools during the same test windows<\/td><\/tr>
Browser setup<\/td>Changing the browser produced some of the largest differences on Walmart and Google<\/td><\/tr>
IP risk score<\/td>Most scored sessions were low risk in both studies: 655 of 677 in the initial test and 938 of 1,002 in the follow-up. The smaller higher-risk groups didn\u2019t show consistently lower success rates.<\/td><\/tr>
Exit location<\/td>In one NodeMaven run, Amazon delivery was 7.5% through California exits and 22\u201330% through Texas, New York, and Florida. Walmart showed no clear state-level pattern<\/td><\/tr>
\u0421\u043e\u0445\u0440\u0430\u043d\u0435\u043d\u0438\u0435 \u0441\u0435\u0441\u0441\u0438\u0438<\/td>It didn\u2019t improve one-page delivery in the controlled test, but it remains necessary for workflows that must keep one identity<\/td><\/tr>
Proxy speed<\/td>It affected job duration, not whether the requested content arrived<\/td><\/tr>
Operational quality<\/td>Free and paid pools differed sharply in connectivity, technical failures, and sticky-session reliability<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

Taken together, these findings cover different parts of scraping. The target, test window, and browser changed how often the requested content arrived, while connection reliability and speed shaped how steadily and quickly the job ran.<\/strong> IP risk scores showed no consistent link to success.<\/p>\n\n\n\n

\"Summary<\/figure>\n\n\n\n

Methodology: What did we test?<\/strong><\/h2>\n\n\n\n

Across both studies, we analyzed 24,156 page visits.<\/strong> The initial test compared premium residential proxies with a public free-proxy list, while the follow-up used NodeMaven\u2019s open-source proxy-benchmark to test more websites and configurations.<\/p>\n\n\n\n

Phase one: premium and free proxy availability<\/strong><\/h3>\n\n\n\n

The first study examined how proxy availability, connection reliability, session stability, speed, and IP reputation<\/strong> related to Amazon page delivery. It tested 774 premium US residential proxy assignments and 500 candidates from a public free-proxy list.<\/p>\n\n\n\n

Requests ran through Playwright with headed Google Chrome on one VPS. Scamalytics and IPinfo supplied the supporting IP data.<\/p>\n\n\n\n

Premium routes requested sticky sessions; the public endpoints offered no session guarantee.<\/p>\n\n\n\n

Phase two: controlled follow-up tests<\/strong><\/h3>\n\n\n\n

The follow-up used the open-source NodeMaven proxy-benchmark<\/a> \u043d\u0430 44 runs covering 19,461 page visits and around 4,500 proxy sessions<\/strong> on one Windows server.<\/p>\n\n\n\n

It covered Amazon Search, Walmart Search, Google Search, and Google Maps. Most runs used headless Patchright and Cloak, with Camoufox in selected browser tests. The controlled sticky-session test used Chromium with a visible window.<\/p>\n\n\n\n

We checked the exit IP at the start, middle, and end of each session, then recorded connection errors, speed, country, reputation data with Scamalytics, and the returned content. Provider requests alternated inside each run<\/strong>, keeping direct comparisons within the same test window.<\/p>\n\n\n\n

Which variables changed web scraping success most?<\/strong><\/h2>\n\n\n\n

Website, test window, and browser configuration produced the largest shifts in delivered content.<\/strong> When the premium pools ran side by side, their results generally stayed close.<\/p>\n\n\n\n

The target website set the baseline<\/strong><\/h3>\n\n\n\n

In the 18 follow-up runs shared by three premium pools, page delivery differed much more by website than by pool:<\/p>\n\n\n\n

Target<\/strong><\/td>Observed delivery across premium pools<\/strong><\/td><\/tr>
Google \u041a\u0430\u0440\u0442\u044b<\/td>94-100%<\/td><\/tr>
Walmart Search<\/td>37-38%<\/td><\/tr>
Amazon Search<\/td>26-28%<\/td><\/tr>
\u041f\u043e\u0438\u0441\u043a Google<\/td>6-12%<\/td><\/tr><\/tbody><\/table>
Source: NodeMaven\u2019s 2026 proxy-quality studies described in the methodology above.<\/em><\/figcaption><\/figure>\n\n\n\n

For every target, the premium pools finished within a few percentage points of one another. Changing the target from Google Maps to Google Search moved the result by more than 80 percentage points.<\/p>\n\n\n\n

Amazon results moved with the test window<\/strong><\/h3>\n\n\n\n

Amazon delivery shifted sharply between runs, with all three premium pools moving in the same direction:<\/p>\n\n\n\n

Test date<\/strong><\/td>Premium pool A<\/strong><\/td>Premium pool B<\/strong><\/td>Premium pool C<\/strong><\/td><\/tr>
September 22<\/td>12%<\/td>10%<\/td>9%<\/td><\/tr>
September 23<\/td>45%<\/td>46%<\/td>40%<\/td><\/tr>
September 29<\/td>54%<\/td>62%<\/td>54%<\/td><\/tr><\/tbody><\/table>
Source: NodeMaven\u2019s 2026 proxy-quality studies described in the methodology above.<\/em><\/figcaption><\/figure>\n\n\n\n

Suppose pool A had run on September 22 and pool B on September 29. The result would resemble a large provider gap even though Amazon became easier for every tested premium pool at roughly the same time<\/strong>.<\/p>\n\n\n\n

We couldn\u2019t isolate the exact cause of that movement. It may have come from the hour, target-side controls, traffic conditions, or another shared variable.<\/strong><\/p>\n\n\n\n

The browser sometimes outweighed the proxy<\/strong><\/h3>\n\n\n\n

Some of the widest gaps appeared when the browser configuration changed while the proxy pools stayed the same.<\/p>\n\n\n\n

Target<\/strong><\/td>\u0411\u0440\u0430\u0443\u0437\u0435\u0440<\/strong><\/td>Page delivery<\/strong><\/td><\/tr>
Walmart Search<\/td>Cloak<\/td>73%<\/td><\/tr>
Walmart Search<\/td>Patchright<\/td>0.2%<\/td><\/tr>
\u041f\u043e\u0438\u0441\u043a Google<\/td>Camoufox<\/td>53%<\/td><\/tr>
\u041f\u043e\u0438\u0441\u043a Google<\/td>Cloak<\/td>12%<\/td><\/tr>
\u041f\u043e\u0438\u0441\u043a Google<\/td>Patchright<\/td>1%<\/td><\/tr><\/tbody><\/table>
Source: NodeMaven\u2019s 2026 proxy-quality studies described in the methodology above.<\/em><\/figcaption><\/figure>\n\n\n\n

On Walmart, the gap between Cloak and Patchright reached 72.8 percentage points<\/strong>. Google Search showed a 52-point range between Camoufox and Patchright.<\/p>\n\n\n\n

These results belong to the tested configurations and test windows. They show why browser setup should remain fixed when comparing proxies and be tested separately when page delivery is low.<\/p>\n\n\n\n

In our comparison of paid Google SERP APIs and Google Browser Scrapert<\/a>, we shared how Google also reacted to the steps taken before the query. Across four NodeMaven runs covering 703 searches, cold exits returned result pages in 20% of searches<\/strong>. Exits that completed a six-page warm-up returned them in 87%<\/strong> under the tested conditions. <\/p>\n\n\n\n

NodeMaven\u2019s open-source Google Browser Scraper<\/a> uses that tested workflow. It warms a new sticky identity, enters the query through a real browser, keeps exits that Google accepts, and records CAPTCHA or block pages as failures instead of empty result sets. <\/p>\n\n\n\n

Our separate browser automation benchmark<\/a> article covers another experiment that held the surrounding setup steady while comparing eight browser configurations on Amazon.<\/p>\n\n\n\n

Where did proxy quality make a measurable difference?<\/strong><\/h2>\n\n\n\n

Across NodeMaven\u2019s two proxy studies, the clearest operational differences appeared in connection reliability and session retention. Public free proxies produced technical failures on 66.8% of attempted visits.<\/strong> In matched follow-up runs, the budget pool retained its exit IP in 71.4% of readable sessions, compared with 98.1% for the premium pool.<\/strong><\/p>\n\n\n\n

Quality measure<\/strong><\/td>Test result<\/strong><\/td>\u0427\u0442\u043e \u044d\u0442\u043e \u0437\u043d\u0430\u0447\u0438\u0442<\/strong><\/td><\/tr>
Free-proxy availability<\/td>Only 168 of 500 public candidates were usable, or 33.6%<\/strong><\/td>Most candidates couldn\u2019t be used for the Amazon test<\/td><\/tr>
Technical failures<\/td>Public free proxies: 66.8%<\/strong> in the initial study. Premium pools: 0.6\u20134.2% initially and 1.5\u20139.9% in the follow-up. Budget pool: 2.5%<\/strong> in the follow-up<\/td>Paid routes reached the target more reliably than the public list<\/td><\/tr>
Session retention<\/td>Budget pool: 71.4%<\/strong>. Premium pool tested beside it: 98.1%<\/strong><\/td>An unexpected IP change can interrupt logins, pagination, carts, and longer browser sequences<\/td><\/tr>
Country accuracy<\/td>The requested country matched in 7,551 of 7,557 paid-pool visits<\/strong> with recorded location data<\/td>Five mismatches came from the budget pool and one from a premium pool<\/td><\/tr><\/tbody><\/table>
Source: NodeMaven\u2019s 2026 proxy-quality studies described in the methodology above.<\/em><\/figcaption><\/figure>\n\n\n\n

The technical-failure figures came from separate test groups, so they don\u2019t form a direct ranking of every paid pool. The session-retention figures provide the cleaner premium-versus-budget comparison<\/strong> because both pools ran during the same test windows.<\/p>\n\n\n\n

A 2024 study of free proxy services<\/a>, \u201cFree Proxies Unmasked,\u201d followed public proxies for 30 months and examined their stability, security, and traffic manipulation. Our test measured a narrower workflow. In that workflow, two-thirds of visits through the public list failed before usable content arrived<\/strong>.<\/p>\n\n\n\n

Could reputation scores predict scraping success?<\/strong><\/h2>\n\n\n\n

Neither study found a consistent relationship between reputation scores and usable-page delivery.<\/strong> In the initial study, 655 of 677 scored premium sessions were low risk. After accounting for the collection period, the correlation between risk score and delivery was approximately -0.02.<\/p>\n\n\n\n

The follow-up produced a similar distribution: 938 of 1,002 scored sessions fell into the low-risk group, and none of the 706 sessions checked against the selected blacklists appeared there. Risk-score correlations with delivery ranged from -0.05 to 0.09 across Amazon, Walmart, and Google.<\/p>\n\n\n\n

Too few sessions landed in the higher-risk groups for a reliable comparison. One controlled NodeMaven run also tested three quality-filter settings; Amazon delivery came to 16%, 17%, and 18%, while Walmart returned 62%, 54%, and 62%.<\/p>\n\n\n\n

Most paid IPs sat in the same reputation range, leaving little variation for the score to explain. Connection failures, country accuracy, and IP retention described how each pool behaved during a real job more directly.<\/strong> A fraud score doesn\u2019t show how the browser will perform or whether a five-step session will keep the same identity.<\/p>\n\n\n\n

Do sticky sessions improve page delivery?<\/strong><\/h2>\n\n\n\n

The controlled follow-up didn\u2019t show that keeping one IP improved single-page Amazon delivery. Sticky sessions delivered 21 of 180 pages, or 11.7%<\/strong>; sessions that requested a new IP for every visit delivered 26 of 180, or 14.4%<\/strong>.<\/p>\n\n\n\n

Sticky sessions still have a clear job in longer workflows. Logins, pagination, carts, and multi-step browser sequences may need one identity from start to finish.<\/strong> A rotating route can return the first page correctly and switch IPs before the next step.<\/p>\n\n\n\n

Proxy speed changed how long the job took<\/strong><\/h2>\n\n\n\n

Speed showed the same weak relationship with page delivery in both studies.<\/strong> In the initial study, correlations with content delivery were approximately -0.09 for latency and -0.10 for download time; no threshold separated delivered pages from failures.<\/p>\n\n\n\n

The follow-up reached the same general result across more targets. Correlations between download speed and delivery ranged from -0.09 to 0.24 by website; connection-time correlations ranged from -0.24 to 0.11.<\/p>\n\n\n\n

The budget pool reached a median of 1.7 Mbit\/s in its shared runs with the premium pool, which reached 3.1 Mbit\/s. Their one-page delivery rates stayed close, but the speed difference changed how long the work took<\/strong>.<\/p>\n\n\n\n

Slow routes keep browser sessions and workers occupied for longer. The difference becomes more expensive on pages that load large scripts, images, or background requests, especially when thousands of sessions are running.<\/p>\n\n\n\n

Why HTTP 200 is not a scraping success<\/strong><\/h2>\n\n\n\n

Amazon sometimes returned its \u201ctoo many requests\u201d page with HTTP 200. Any benchmark that counted status codes alone would have recorded that block as a success.<\/p>\n\n\n\n

\u0417\u043e\u043d\u0430 HTTP Semantics standard, RFC 9110<\/a> defines 200 as a successful HTTP request. It doesn\u2019t verify that a scraper received the product listings, search results, or fields its parser expected.<\/p>\n\n\n\n

We classified each returned page as one of the following:<\/p>\n\n\n\n