Что такое веб-скрапинг? Пошаговое руководство на Python

Веб-скрейпинг is the automated collection of information from websites. In this tutorial, you will build a Python scraper that extracts 20 book titles, prices, availability details, and product links, then saves them to a CSV file. Basic Python familiarity helps, but you do not need previous scraping experience.

Automated systems generated more than 53% of web traffic in 2025, according to the Отчет Imperva о плохих ботах 2026. Many websites now respond to automated requests with rate limits, CAPTCHAs, browser checks, and IP reputation controls.

Clean proxies help when a scraper moves from a practice website to larger or location-specific projects. NodeMaven веб-скрейпинг прокси provide pre-filtered residential IPs, flexible rotation, sticky sessions, and precise geo-targeting to reduce failed requests and collect more consistent results. The tutorial shows where proxies fit into the scraper without adding them before they are essential.

Start scraping with cleaner IPs and fewer failed requests

Сократите число CAPTCHA и неудачных запросов благодаря предварительно отфильтрованным резидентским IP. Протестируйте NodeMaven за $3.50 — 750 MB включены.

Попробовать

Что такое веб-скрейпинг и как он работает?

A web scraper visits a page and extracts selected information from its HTML. Instead of copying product prices or article headlines manually, the scraper finds each field and saves it automatically.

A basic scraping workflow has six stages:

1. Send an HTTP request to a URL.
2. Download the page HTML.
3. Parse the HTML into a searchable structure.
4. Find the required elements with CSS selectors or XPath.
5. Extract and clean their contents.
6. Save the records to CSV, JSON, or a database.

In short: Website URL -> HTTP request -> HTML response -> CSS selectors -> selected data -> CSV file.

Suppose an online store places a product title inside an h2 element and the price inside an element with the .price class. A scraper can find those elements on every product page and turn them into spreadsheet rows.

Web scraping, crawling, and data mining

Web crawling and web scraping often appear in the same project, but they do different jobs. A crawler discovers pages. A scraper extracts selected fields from them.

For example, a crawler might find 10,000 product URLs. The scraper then collects the title, price, seller, rating, and stock status from each page.

Data mining examines the collected dataset to find patterns, anomalies, or predictions. This comparison of data mining and web scraping explains where each process fits.

Static and JavaScript-rendered pages

Static pages include the required information in the HTML returned by the server. Python libraries such as Requests and BeautifulSoup can usually handle them.

JavaScript-rendered pages load content after the initial request. Requests may receive HTML without the products, reviews, or listings visible in a browser. Драматург and Selenium solve this by opening a browser, running the page scripts, and extracting data from the rendered page.

Start with a normal HTTP request and inspect the response. If it already contains the required fields, a browser adds unnecessary memory use and slower page loads. Move to browser automation when the content appears only after JavaScript runs, the page scrolls, or a user action takes place.

Developers make the same recommendation in this discussion about moving from a script to scraping infrastructure: start with Requests and BeautifulSoup, then add Playwright only when the target requires it.

How to scrape a website with Python

This example uses Books to Scrape, a demo ecommerce website created for scraping practice. The scraper collects:

• Название книги
• Цена
• наличие
• ссылка на товар

The finished records are saved to books_to_scrape_products.csv. You can adapt the same process to another static website by changing the URL and CSS selectors.

Step 1: Decide what data to collect

Open Books to Scrape and examine one product card. Define the required fields before writing the scraper.

For another website, the fields might include a product name, SKU, discounted price, seller, rating, or review count. Writing this list first prevents the script from collecting fields that the project does not need.

Step 2: Find the CSS selectors

Открыть Books to Scrape in Chrome. Right-click the first book title and select Inspect.

Chrome DevTools highlights the title’s <<code>a```> element. This element contains both values needed by the scraper:

• title contains the full book title.
• href contains the product link.

To copy its selector:

1. Right-click the highlighted a элемент.
2. Открыть копировать.
3. Выберите Copy selector.

4. Repeat the process for the price and availability.

Chrome may return a long selector that points only to the first book. Instead, the scraper will first select every repeated book card and then search for the required fields inside each card.

Use these selectors:

The script uses article.product_pod to find all 20 book cards. Inside each card, h3 a finds the title and link, while .price_color и .availability find the remaining fields.

Step 3: Set up the Python project

Create the project folder:

On macOS or Linux, create and activate a virtual environment:

On Windows PowerShell, use:

Install the required packages:

Requests downloads the HTML. BeautifulSoup parses it and searches for elements using CSS selectors.

Create an empty file named books_scraper.py inside the project folder.

On macOS:

For Linux:

В Windows PowerShell:

Keep the empty file open. Steps 4 through 8 will build the scraper inside it.

Step 4: Download the page

Paste the following code at the top of the empty books_scraper.py file:

requests.get() downloads the page. The timeout stops the script from waiting indefinitely, while raise_for_status() reports HTTP errors.

The UTF-8 setting is correct for Books to Scrape and prevents the pound symbol from being decoded incorrectly. Do not copy this setting blindly to every project. Check the target website’s declared encoding before overriding the value detected by Requests.

To scrape another website, replace URL with its page address.

Keep the file open and continue to Step 5. The scraper is not complete yet.

Step 5: Parse the HTML

Directly below the Step 4 code, convert the response into a BeautifulSoup object:

Find every book card:

select() returns every element matching the CSS selector. Books to Scrape should return 20 cards on the first page.

If the result is 0, inspect the response HTML and check the selector. The website may also load its products through JavaScript.

Step 6: Extract the product fields

Directly below the Step 5 code, create an empty list and loop through the cards:

Keep the four-space indentation inside the на loop. Python uses indentation to determine which instructions should run for each book.

The scraper uses get("title") because the book title is stored in an HTML attribute. Price and availability appear as text between tags, so the script uses get_text().

The fallback values prevent one missing field from stopping the job. They also make missing data visible in the CSV.

Step 7: Build complete product links

Books to Scrape uses relative links such as:

Add this code immediately below the availability code. Keep it indented by four spaces because it belongs inside the same на loop:

The resulting URL is:

Still inside the loop, add the completed record to the list:

Each dictionary becomes one row in the CSV.

Step 8: Save the data to CSV

Add the following code after the loop. Start fieldnames at the left edge with no indentation so Python knows that the loop has ended:

Python’s CSV documentation recommends opening CSV files with newline="". The utf-8-sig encoding helps Excel and similar applications display the pound symbol correctly.

The scraper file is now complete. Save books_scraper.py before running it.

Step 9: Run and check the scraper

Return to the terminal while the virtual environment is still active. Run the file:

The terminal should show:

Check that:

• The CSV contains 20 products.
• Titles and prices appear in the correct columns.
• Currency symbols display properly.
• Product links open the expected pages.

On macOS, open the CSV with:

On Windows, open the books_scraper folder and double-click books_to_scrape_products.csv.

Step 10: Add a proxy when the target requires one

Books to Scrape does not require a proxy. A proxy becomes useful when an approved scraping project needs regional results, session continuity, or more requests than one IP can handle reliably.

For this example, create a NodeMaven residential proxy with the required location in Личный кабинет. Choose rotation for independent pages or a sticky session when pagination, cookies, or related requests must stay together. Select HTTP and copy the generated username and password.

Store the credentials in environment variables instead of placing them in the Python file.

На macOS или Linux:

В Windows PowerShell:

Добавить os to the imports and include quote in the URL utilities:

Replace the original request with:

Requests supports this authenticated proxy format in its official proxy documentation. Encoding the credentials prevents characters such as @ from breaking the proxy URL.

Save the updated file and run it again with python books_scraper.py. The scraper will now send its request through the selected NodeMaven proxy.

Use proxies only where automated access is permitted. A proxy does not repair broken selectors, render JavaScript, or excuse excessive request rates.

Start scraping with cleaner IPs and fewer failed requests

Сократите число CAPTCHA и неудачных запросов благодаря предварительно отфильтрованным резидентским IP. Протестируйте NodeMaven за $3.50 — 750 MB включены.

Попробовать

How to adapt the scraper to another website

To reuse the scraper:

1. Replace the page URL.
2. Find the repeated product or listing card.
3. Replace the card and field selectors.
4. Check whether each value appears as text or an HTML attribute.
5. Update the CSV column names.
6. Run the script and compare its output with the visible page.

Requests and BeautifulSoup work when the required data appears in the downloaded HTML. JavaScript-rendered websites may require Playwright or Selenium.

Зона ChatGPT web scraping guide extends this example with pagination, retries, debugging, and browser rendering.

Web scraping techniques and tools

The appropriate method depends on how the website loads its data, the number of pages involved, and how much maintenance the project can support.

Requests and BeautifulSoup are a sensible starting point because the code is easy to inspect. Scrapy adds request queues, concurrency controls, pipelines, and scheduling for larger jobs.

Драматург is useful when data appears only after JavaScript runs, the page scrolls, or a user clicks a control. No-code tools and scraping APIs can save time when maintaining a custom scraper would cost more than the service. This guide to Инструменты веб-скрейпинга compares the main options.

What is web scraping used for?

Web scraping is useful when information must be collected repeatedly or across more pages than a person could reasonably process by hand.

E-commerce and price monitoring

Retailers collect public prices, discounts, stock status, delivery estimates, ratings, and product launches. A daily scraper can record competitor price changes and support pricing analysis, stock planning, or promotion tracking.

Зона price scraping guide explains how these systems organize pricing data. Amazon needs extra preparation because prices, stock, and shipping details may change by location; the guide to scraping Amazon covers that workflow.

News and search monitoring

News scrapers collect headlines, publication dates, categories, authors, and article URLs. They can support media monitoring and market research. A scheduled scraper may revisit an archive every hour and store only newly published articles. This guide explains how to scrape news websites without collecting the same articles repeatedly.

Search monitoring uses a similar process to collect rankings, featured snippets, local results, and competing domains. Since search results vary by location, a scraper needs to request the market being measured rather than rely on the location of its server.

Social and community research

Public social media data can support trend monitoring, audience research, and brand tracking. These platforms often use JavaScript, authentication, rate limits, and other access controls, so a basic Requests scraper may receive a login page or an incomplete response.

The practical requirements vary by platform. NodeMaven has separate guides to scraping Twitter and building a Парсер Reddit.

Является ли веб-скрейпинг законным?

Web scraping is not automatically legal or illegal. The answer depends on the jurisdiction, the information being collected, the access method, and how the data will be used.

Before scraping a website:

• Check whether it provides an official API.
• Review its terms of service.
• Read its robots.txt file.
• Avoid private or login-protected information without authorization.
• Consider copyright, database rights, and privacy law.
• Keep request rates low enough to avoid disrupting the service.
• Seek legal advice for commercial or high-risk projects.

Зона Robots Exclusion Protocol defines how websites publish crawling instructions in robots.txt. It also states that these rules are not access authorization. An allowed path does not settle questions about copyright, privacy, contracts, or computer-access laws.

The guide to web scraping legality covers these issues in more detail.

How to keep a scraper reliable

A successful HTTP response does not prove that the scraper collected the right information. Real websites change their layouts, return block pages with a 200 OK status, and occasionally fail to respond.

Validate the extracted fields

Define the fields that every valid record must contain. A product record might require a title, price, product ID, and canonical URL. If one of those fields is missing, log the URL instead of saving an incomplete row.

Also check responses for CAPTCHA text, access-denied messages, login pages, and empty product grids.

Retry temporary failures carefully

Timeouts and 429, 502, or 503 responses may be temporary. Retry them two or three times with a longer delay after each attempt. Move URLs that still fail into a separate queue rather than trapping the scraper in an endless loop.

A 2025 research paper on web scraping explains that timeouts and rate limits can create non-random gaps in collected data. If a pricing scraper misses one category more often than another, its final averages may be misleading even when every saved row is correct.

Monitor each run

Track the number of records collected, missing-field rate, duplicate count, retries, block-page detections, and response times. If a job normally returns 500 products and suddenly returns 12, stop the export and inspect the failed pages.

Save the response HTML and, for browser-based jobs, a screenshot when extraction fails. These files help distinguish a broken selector from a block page or JavaScript problem.

When proxies are useful for web scraping

A practice scraper or small job on an open website may not need proxies. They become useful when a project needs regional results, a stable session, or enough requests that one IP begins receiving rate limits.

Apify’s residential proxy documentation explains that residential traffic is harder to distinguish from ordinary user traffic. Its datacenter proxy guide recommends starting with faster, cheaper datacenter proxies and moving to residential addresses when a target heavily blocks hosting networks.

Подобная проблема возникает и в этом обсуждение среди разработчиков веб-скрапинга. One scraper worked locally but triggered immediate bans after moving to a VPS. Commenters identified the центр обработки данных ASN, browser fingerprint, and automated request pattern as possible causes. They recommended residential IPs together with slower requests and better browser configuration.

Выберите Вращение for independent URLs. Use a sticky session when requests share cookies, pagination state, or location settings.

Start scraping with cleaner IPs and fewer failed requests

Сократите число CAPTCHA и неудачных запросов благодаря предварительно отфильтрованным резидентским IP. Протестируйте NodeMaven за $3.50 — 750 MB включены.

Попробовать

Why choose NodeMaven for web scraping

A cheap proxy that repeatedly returns CAPTCHAs or blocks pages creates more retries and leaves gaps in the dataset.

NodeMaven pre-filters residential IPs and offers clean proxies for websites with strict IP reputation checks. Its features map directly to common scraping tasks:

• Вращение distributes product pages, listings, or search requests across different IPs.
• сессии с привязкой к серверу, длящиеся до 24 часов preserve cookies, pagination state, and regional settings.
• страна, город, интернет-провайдер и почтовый индекс supports regional price, inventory, and search-result collection.
• Поддержка HTTP и SOCKS5 works with Requests, Scrapy, Playwright, and other scraping tools.
• Residential and mobile traffic in one plan allows testing across both network types.

For example, a price scraper can rotate IPs across independent product URLs while keeping every request tied to the required city or ZIP code. Eligible proxy issues are also covered by NodeMaven’s quality guarantee and cashback conditions.

Start the NodeMaven proxy trial to test the target, location, and session settings before scaling the scraper.

Start scraping with cleaner IPs and fewer failed requests

Сократите число CAPTCHA и неудачных запросов благодаря предварительно отфильтрованным резидентским IP. Протестируйте NodeMaven за $3.50 — 750 MB включены.

Попробовать

A practical starting point

For a first project, use Requests and BeautifulSoup on a practice website such as Books to Scrape. Confirm that the selectors return the expected records before adding pagination, retries, browser rendering, or proxies. Use Playwright when the required content depends on JavaScript, and introduce proxies only when the project has a legitimate need for regional testing, session continuity, or higher-volume collection. Review the website’s rules and validate the extracted data instead of treating a successful request as proof that the scraper worked.

FAQ