Craigslist scraper in Python: step-by-step guide

Craigslist still runs on plain HTML, no heavy JavaScript app wall, no infinite login prompts. That makes it one of the most approachable sites for anyone learning web scraping Craigslist for the first time. But “approachable” doesn’t mean “no rules”. Between rate limits, shifting page markup, and Craigslist’s own stance on automated access, a Python Craigslist scraper needs a bit more care than a basic tutorial script.

This guide walks through building a real Craigslist scraper Python project from the ground up. You’ll write your own Requests and BeautifulSoup code, pull titles, prices, locations, and URLs out of the page, and export everything to a clean CSV file. Along the way we’ll cover the legal questions people always ask before they start scraping Craigslist, and what to do when your requests start getting blocked.

Scrape Craigslist with fewer blocks using high quality residential and mobile proxies.
Start your NodeMaven trial for $3.50 with 750 MB included

Попробовать

Why scrape Craigslist?

Craigslist is one of the largest sources of unfiltered, real-time classified data on the internet. No aggregator sits between you and the actual post.

Some of the most common reasons people build a scraper:

• Маркетинговые исследования — tracking what’s being sold, where, and for how much across a region
• Rental monitoring — a huge use case is web scraping craigslist housing listings to catch new apartments the moment they’re posted, before they get snapped up
• Used car research — comparing asking prices for a specific make and model across cities
• Price tracking — watching a category over weeks or months to spot trends
• Competitive analysis — sellers checking what similar listings in their niche are priced at

A simple script that grabs a search results page every few hours can answer most of these questions on its own.

Is scraping Craigslist legal?

It depends on what you scrape, how you access it, and what you do with the data afterward. Nothing here is legal advice. If you’re building something commercial, talk to an actual lawyer and read Craigslist’s Terms of Use yourself.

Craigslist Terms of Use on automated scraping

Craigslist’s Terms of Use prohibit automated data collection. They cover bots, scripts, and other automated tools, which make it clear that bulk data collection is not allowed. Although many developers scrape Craigslist. Be aware that doing it may violate the site’s Terms of Use, even if it is technically possible.

Robots.txt

Craigslist’s robots.txt file specifies which areas automated crawlers should avoid, including pages for replying to listings and other internal functions. Following the Craigslist robots.txt scraping policy demonstrates responsible practices, even though robots.txt itself is not legally binding. It also reflects the site’s intent to limit automated access.

Craigslist scraping lawsuits

Craigslist has taken legal action against companies that scraped listings at scale and republished the data for commercial purposes. These cases focused on large businesses rather than individuals collecting small amounts of data. If you plan to scrape Craigslist, review the site’s Terms of Use and avoid using scraped content commercially.

Tools You’ll Need

This project uses a short, standard Python stack. Nothing here needs a paid license or a complex install.

• Python 3.9+ — any recent version works fine
• Запросы — handles the HTTP calls and header management
• BeautifulSoup — parses the returned HTML into something you can query
• lxml — a fast parser backend that BeautifulSoup can use instead of Python’s built-in parser
• csv — part of the Python standard library, used to write out results
• NodeMaven residential proxies — optional, but useful once you’re running the scraper regularly or across multiple cities without getting your IP flagged

Related reading: Web scraping with Python: the complete guide

Understanding Craigslist HTML structure

Before writing any parsing code, it helps to know what you’re actually parsing. A Craigslist search results page (for example, an apartments listing page for a specific city) is built from a repeating list of listing cards. Each card is a single HTML list item that wraps:

• A link element that contains the listing’s title text and its full URL
• A price value, when the post has one
• A neighborhood or area label showing roughly where the item is located

Every listing card follows the same repeating pattern, which is exactly what makes Craigslist scrapable with CSS selectors. You find one card, look at how it’s tagged, then tell BeautifulSoup “find me every element that looks like this”.

The tricky part is that Craigslist changes its class names and markup periodically as it updates its front end. A selector that works today might return nothing in six months. That’s normal for scraping any site, and it’s why the code below is written to fail gracefully and print a warning instead of crashing when a field is missing.

Collect Craigslist listings more reliably with premium residential and mobile proxies.
Start your NodeMaven trial for $3.50 with 750 MB included

Попробовать

Build a Craigslist scraper with Python

This is the core of the guide. We’ll build the scraper in small steps so each piece is easy to follow, then put it all together into one script.

Step 1: Install dependencies

Open a terminal and install the two external packages this project needs:

Запросы handles the network call. BeautifulSoup parses the HTML. lxml is the parser engine we’ll hand to BeautifulSoup because it’s noticeably faster than the default.

Step 2: Send an HTTP request

A few things worth noting here. The User-Agent header is the single most important header for this kind of request. Without one, many servers, including Craigslist, will either serve a stripped-down response or block the request outright. The прокси argument is optional and stays empty by default.

Step 3: Parse the HTML

This step is intentionally small. All it does is hand the text to BeautifulSoup and get back a tree structure you can search through with CSS selectors, similar to how you’d target elements in a stylesheet.

Step 4: Extract listing data

Here’s where the actual scraping logic lives. Craigslist wraps each result in a list item, and we grab every one of them on the page.

Notice the fallback. Instead of assuming one selector will always match, the function tries a second, looser pattern if the first one comes back empty. This is a small habit that saves a lot of debugging time when a site’s markup shifts.

Step 5: Extract title, price, location, and URL

Each field is pulled independently and defaults to a safe fallback instead of raising an error. That matters because real listing pages are messy: some posts have no price, some have no clear location label, and a script that crashes on the first missing field is useless for anything you plan to run unattended.

Step 6: Export to CSV

csv.DictWriter matches each dictionary’s keys to the header row automatically, so as long as every listing dictionary has the same four keys, this just works.

Putting it all together

Run this file directly and it will fetch one search page, pull out every listing it can find, print the first five to your terminal, and write the full set to craigslist_listings.csv. From here, you can loop over multiple category or city URLs, add pagination by following the “next” link on the page, or schedule the script to run on a timer for ongoing price or rental tracking.

Важно: Craigslist’s markup changes from time to time. If find_listing_cards() suddenly returns an empty list, open the search page in a browser, inspect a listing card, and update the selector in Step 4 to match what you see.

Reduce scraping interruptions with fast residential and mobile proxies built for automation.
Start your NodeMaven trial for $3.50 with 750 MB included.

Попробовать

Avoid getting blocked while scraping Craigslist

Once you move past a handful of test requests, blocking becomes the real bottleneck. Craigslist doesn’t publish exact rate limits, but sending requests too fast from a single IP is the fastest way to start seeing empty pages.

A few habits go a long way:

• Добавляйте задержки между запросами. A few seconds between page loads mimics normal browsing far better than a tight loop.
• Build in retries with backoff, so a single failed request doesn’t kill an entire scraping run.
• Ротация IP-адресов once you’re making a meaningful number of requests. A single IP hammering the same search page repeatedly stands out.
• Задавайте реалистичные заголовки, not just User-Agent, but Accept-Language and Referer where it makes sense.
• Соблюдайте лимиты even when they’re not published, slow down if you start seeing more errors than usual.

Резидентские прокси can improve scraping reliability by routing requests through real household IP addresses instead of easily identified data center IPs. NodeMaven’s proxies are a good fit for Craigslist scraping, while мобильные прокси. are useful for workflows that also involve account management.

Before running your scraper, you can verify your setup with NodeMaven’s free tools. The bandwidth checker tests proxy connectivity, the IP Lookup tool confirms the IP and location websites see.

Craigslist scraper API vs Python scraper

Craigslist has no official public API for search results, which is part of why so many people end up writing their own tools instead. That leaves a few realistic paths, each with a different tradeoff between effort and control.

Since Craigslist’s search pages are rendered on the server and don’t require JavaScript to display listings, Requests and BeautifulSoup are usually the simplest and fastest route, which is exactly why we built the scraper in this guide around them.

Common Craigslist scraping errors

A few errors show up constantly once you run a scraper for real, rather than against a single test page.

• 403 Запрещено — usually a missing or suspicious User-Agent header, or an IP that’s already been flagged. Double-check your headers first, then consider rotating your IP.
• 429 Слишком много запросов — you’re sending requests faster than the server wants to serve them. Add delays and slow down your loop.
• CAPTCHA-задачи — a sign that your traffic pattern looks automated. Slowing your request rate and rotating IPs both help reduce how often this triggers.
• Empty responses — sometimes the server returns a valid 200 status but a page with no real content, often a soft block. Check the raw HTML length before assuming your parser is broken.
• Missing selectors — your parsing code returns nothing, but the request itself worked fine. This almost always means Craigslist changed its class names; inspect the page manually and update your selectors.

Alternative scraping frameworks

Requests and BeautifulSoup cover most Craigslist scraping needs, but it’s worth knowing the alternatives.

Scrapy is a full scraping framework with built-in request scheduling, retries, and pipelines for exporting data. It’s a better fit once a project grows past a single script into something you’re running on a schedule across many pages.

Selenium drives an actual browser, which matters for sites that build their content with JavaScript after the page loads. Craigslist search results don’t need this, but it’s the right tool if you ever expand into sites that do.

Playwright is a newer alternative to Selenium with a similar purpose: full browser automation, generally faster and easier to work with for modern multi-tab or multi-context scraping.

Заключение

You now have a working Craigslist data scraper: it sends a properly headed request, parses the returned HTML, pulls out title, price, location, and URL for every listing on a page, and writes the results to CSV. That’s a solid foundation whether you’re tracking rental prices, researching used car listings, or keeping an eye on a specific category over time.

The legal and technical sides both matter here. Read Craigslist’s Terms of Use, keep your request volume reasonable, and treat robots.txt as a signal worth respecting even where it doesn’t cover everything you’re interested in.

As your scraping grows past a few manual runs, into scheduled jobs or multi-city pulls, IP rotation stops being optional. That’s the point where proxies earn their place in the stack.

Improve Craigslist scraping success with clean residential and mobile IPs.
Start your NodeMaven trial for $3.50 with 750 MB included

Попробовать

Часто задаваемые вопросы