Scrapy Proxy

Create and activate a Python virtual environment:

python3 -m venv scrapy-env

source scrapy-env/bin/activate

Install Scrapy:

python3 -m pip install scrapy

Create the project and example spider:

scrapy startproject nodemaven_scrapy cd nodemaven_scrapy scrapy genspider books books.toscrape.com

If python3 --version also returns “command not found,” install Python 3 from python.org first.

Make sure the terminal is inside the Scrapy project directory. The prompt should contain nodemaven_scrapy.

Open the project’s middlewares.py file with Nano:

nano nodemaven_scrapy/middlewares.py

Move to the bottom of the file, add a blank line, and paste:

import os

from urllib.parse import quote





class NodeMavenProxyMiddleware:

    def __init__(self):

        username = quote(

            os.environ["NODEMAVEN_PROXY_USER"],

            safe="",

        )

        password = quote(

            os.environ["NODEMAVEN_PROXY_PASS"],

            safe="",

        )




        self.proxy_url = (

            f"http://{username}:{password}"

            "@gate.nodemaven.com:8080"

        )




    def process_request(self, request):

        request.meta.setdefault("proxy", self.proxy_url)

Important: The method must be written as __init__, with two underscores before and after init.

Save and close the file:

1. Press Control + O.
2. Press Enter to confirm the filename.
3. Press Control + X.

Check the file for syntax errors:

python3 -m py_compile nodemaven_scrapy/middlewares.py \

  && echo "Middleware syntax check: passed"

If the code is correct, the terminal will display:

Middleware syntax check: passed

The middleware reads the credentials stored in Step 3, creates an authenticated NodeMaven proxy URL, and assigns it to Scrapy requests. The quote() function safely encodes special characters in the username and password.

Make sure the terminal is still inside the nodemaven_scrapy project directory.

Open the project settings file:

nano nodemaven_scrapy/settings.py

Look for this existing setting:

ROBOTSTXT_OBEY = True

If it is already present, leave it unchanged. Do not add it a second time.

Move to the bottom of the file and add:

DOWNLOADER_MIDDLEWARES = {

    "nodemaven_scrapy.middlewares.NodeMavenProxyMiddleware": 700,

}




HTTPPROXY_ENABLED = True

If your project has a different package name, replace nodemaven_scrapy with that name. For the project created in Step 1, keep the value exactly as shown.

Save and close the file:

1. Press Control + O.
2. Press Enter to confirm the filename.
3. Press Control + X.

Check the file for Python syntax errors:

python3 -m py_compile nodemaven_scrapy/settings.py \

  && echo "Settings syntax check: passed"

Confirm that Scrapy loaded the settings:

scrapy settings --get DOWNLOADER_MIDDLEWARES

scrapy settings --get HTTPPROXY_ENABLED

scrapy settings --get ROBOTSTXT_OBEY

The output should include the NodeMaven middleware and show True for both settings:

{'nodemaven_scrapy.middlewares.NodeMavenProxyMiddleware': 700}

True

True

Scrapy’s built-in HttpProxyMiddleware reads the proxy URL from request.meta["proxy"]. The custom middleware added in Step 4 supplies the NodeMaven proxy URL.

Use responsible download delays and concurrency limits based on the target website.

Make sure the terminal is inside the nodemaven_scrapy project directory.

Open the generated spider file:

nano nodemaven_scrapy/spiders/books.py

Nano will display the basic spider created in Step 1.

Remove the existing code:

1. Move the cursor to the first line.
2. Press Control + K repeatedly until every line has been removed.

Paste the following code into the empty file:

import scrapy





class BooksSpider(scrapy.Spider):

    name = "books"

    allowed_domains = ["books.toscrape.com"]

    start_urls = ["https://books.toscrape.com/"]




    def parse(self, response):

        for book in response.css("article.product_pod"):

            yield {

                "title": book.css(

                    "h3 a::attr(title)"

                ).get(),

                "price": book.css(

                    ".price_color::text"

                ).get(),

                "availability": " ".join(

                    book.css(

                        ".availability::text"

                    ).getall()

                ).strip(),

            }

Save and close the file:

1. Press Control + O.
2. Press Enter to confirm the filename.
3. Press Control + X.

Check the spider for syntax errors:

python3 -m py_compile nodemaven_scrapy/spiders/books.py \

  && echo "Spider syntax check: passed"

Confirm that Scrapy recognizes the spider:

scrapy list

The terminal should display:

Spider syntax check: passed

books

This example spider opens Books to Scrape and extracts each book’s:

• Title
• Price
• Availability

We use https://books.toscrape.com/ only as an example. It is a sandbox website created specifically for web-scraping demonstrations.

For a real project, replace the URL and extraction rules with your target website. Make sure you have permission to access and scrape it.

Screenshot: The books.py file showing the example URL and the selectors for title, price, and availability, together with the Spider syntax check: passed confirmation.

From the Scrapy project directory, run:

scrapy crawl books -O books.json

The -O option creates books.json or overwrites the file if it already exists.

Scrapy will route the spider’s requests through gate.nodemaven.com using the location and session settings contained in the generated NodeMaven username.

Wait until the crawl finishes. A successful run should show:

• An HTTP 200 response
• Extracted book items in the logs
• No proxy connection errors
• No 407 Proxy Authentication Required error
• A Spider closed (finished) message
• A books.json file in the project directory

The crawl creates books.json inside the Scrapy project directory.

Confirm that the file exists:

ls -lh books.json

Display the number of extracted records and preview the first three:

python3 -c 'import json; data=json.load(open("books.json")); print(f"Total records: {len(data)}"); print(json.dumps(data[:3], indent=2, ensure_ascii=False))'

The output should contain book records with:

• Title
• Price
• Availability

A completed crawl with exported records and no proxy authentication errors confirms that Scrapy accepted the NodeMaven proxy configuration.

If the spider returns a 407 error, confirm that:

• The complete NodeMaven username was copied
• The password is correct
• The HTTP port matches the dashboard
• The environment variables are available to Scrapy
• Reserved characters in the credentials are URL-encoded

Your NodeMaven proxy is now connected to Scrapy.