How to Check Meta Tags for a List of URLs (and What to Do When It Fails)
Reading the meta tags on one page takes a right-click. Checking meta tags across a list of URLs is a different job, and there are four practical ways to do it.
- A browser-based bulk checker. Paste the list, read the results, export a CSV. No setup at all.
- A desktop crawler such as Screaming Frog or Sitebulb. Built to discover URLs across a whole site rather than check the ones you already have.
- A Python script. No URL ceiling, full control over every field, and a scraper that is now yours to look after.
- Spreadsheet formulas.
IMPORTXMLin Google Sheets, free and familiar, right up until the pages stop cooperating.
Each wins in a different situation and breaks in a different way. We will go through all four below, with the strengths and the limits of each, after a quick look at what you are actually collecting.
What lives in a page’s <head>
Everything here comes from one place: the HTML <head> of a public page. These are the fields worth collecting.
| Поле | What it controls | Common problem |
|---|---|---|
<title> | Headline in search results and browser tabs | Missing, duplicated, or truncated past ~60 characters |
meta description | Snippet text under the search result | Missing, or written for a different page |
link rel="canonical" | Declares the preferred version of the page | Points to the wrong URL after a migration |
meta robots | Indexing and crawling directives | An accidental noindex left over from staging |
og:title, og:description, og:image | What Facebook and LinkedIn display | Missing image, so the share card renders grey |
twitter:card, twitter:title | What X displays | Falls back to Open Graph when absent, sometimes badly |
hreflang | Maps language and region variants | Missing return tags, or self-referencing errors |
JSON-LD <script> | Structured data for rich results | Valid syntax describing the wrong entity |
Any one of these is easy to check by hand. The trouble is volume, and it compounds when some of the pages refuse to answer.
Four ways to check meta tags across a list of URLs
1. A browser-based bulk checker
Paste your list, run it, read the results. NodeMaven’s meta tag checker handles up to 50 URLs per run.
It returns titles, descriptions, canonicals, robots directives, schema types, Open Graph and Twitter Card data, plus live previews of how each link renders on Google, Facebook, X и LinkedIn. Results export to CSV, and you can copy a single tag or the whole page head.
The reason to start here is that there is nothing to install and nothing to maintain. For a launch QA pass, a post-migration spot check or a competitor comparison, the setup cost of any other method is larger than the job itself.
Where it stops: 50 URLs per run, no public API, no site discovery. You bring the list, the tool will not go and find one.
2. A desktop crawler
Screaming Frog и Sitebulb solve a different problem. They start from a domain and discover URLs by following links, which is what you want for a full technical audit of a site you have never seen.
That discovery step is also why they are wrong for a known list. Installing a crawler, configuring it and waiting for a crawl, all to check thirty URLs already sitting in a spreadsheet, is work that adds nothing.
Screaming Frog’s free tier caps at 500 URLs. The licence runs around $259 a year at current pricing.
Where it stops: heavy for small jobs, desktop-only, and the free cap arrives quickly on a real site.
3. Python
Twenty lines gets you a repeatable script with no URL ceiling:
python
import csv, httpx
from selectolax.parser import HTMLParser
def get_meta(url):
r = httpx.get(url, follow_redirects=True, timeout=15,
headers={"User-Agent": "Mozilla/5.0"})
tree = HTMLParser(r.text)
def attr(selector, name="content"):
node = tree.css_first(selector)
return node.attributes.get(name, "") if node else ""
title = tree.css_first("title")
return {
"url": url,
"final_url": str(r.url),
"status": r.status_code,
"title": title.text() if title else "",
"description": attr('meta[name="description"]'),
"canonical": attr('link[rel="canonical"]', "href"),
"robots": attr('meta[name="robots"]'),
"og_title": attr('meta[property="og:title"]'),
"og_image": attr('meta[property="og:image"]'),
}
with open("urls.txt") as f:
rows = [get_meta(line.strip()) for line in f if line.strip()]
with open("meta.csv", "w", newline="") as f:
writer = csv.DictWriter(f, fieldnames=rows[0].keys())
writer.writeheader()
writer.writerows(rows)Обратите внимание на final_url field. Recording where the request actually landed costs one line and catches a whole category of silent errors, which the next section covers.
This is the right answer when the check needs to run on a schedule, feed another system, or cover thousands of URLs.
Where it stops: you now own a scraper. Every failure mode below becomes yours to handle, starting with the fact that this script sees only the HTML the server sends and nothing a browser would build afterwards.
4. Spreadsheet formulas
Google Sheets can do this with no tooling at all. Put URLs in column A and:
=IMPORTXML(A2, "//title")
=IMPORTXML(A2, "//meta[@name='description']/@content")
=IMPORTXML(A2, "//link[@rel='canonical']/@href")Free, familiar, and genuinely fine for ten URLs.
Where it stops: IMPORTXML fetches raw HTML, so anything rendered by JavaScript returns empty. Sheets also rate-limits aggressively, and a column of fifty formulas recalculating will start throwing #N/A on its own. There is no status code, so a blocked page and a page with no title look identical.
Which method fits which job
| Метод | Настройка | URL limit | Sees JS content | Handles blocks | Export |
|---|---|---|---|---|---|
| Browser checker | Нет | 50 per run | Yes, with browser mode | Built in | CSV |
| Desktop crawler | Install + config | 500 free | With rendering enabled | Manual config | CSV, Excel |
| Питон | Write + maintain | Нет | Only with a headless browser | You build it | Anything |
| Sheets formulas | Нет | ~10 in practice | Нет | Нет | Native |
Why bulk metadata checks come back wrong
Pick any method above and the mechanics are the same: request a URL, read the <head>, write a row. Five things break that chain, and most of them have nothing to do with the page’s metadata.
One rule covers all five: escalate, do not start heavy. Run the whole list the cheap way, re-run only the failures with browser rendering, then re-run whatever still fails from a different IP. Most URLs never need the expensive path.
Start here: match the symptom to the cause
| What you see | Most likely cause |
|---|---|
| Blank title, page looks fine in Chrome | Рендеринг JavaScript |
| Same title on several unrelated domains | Bot protection answered |
| First rows fine, later rows empty | Ограничение скорости |
| Everything filled, values feel off | Country-specific content |
| Complete data for a page you did not request | Redirect or soft 404 |
The page builds its title in JavaScript
You open the URL in Chrome and the title sits right there in the tab, but your script returns nothing. Both observations are accurate.
The server sent a near-empty HTML shell, and your browser ran the JavaScript that filled it in. React, Vue, Angular and Next.js apps without server-side rendering all behave this way. So do plenty of sites that render most content on the server but inject Open Graph tags client-side.
How to confirm: открыть view-source: instead of the inspector. One shows what the server sent, the other what the browser built. If the title appears in the second and not the first, you have your answer.
The fix: render the page. That means browser mode in a checker, JavaScript rendering enabled in a crawler, or Playwright instead of httpx in your script. Rendering is slow and heavy, so save it for the URLs that need it.
Bot protection answered instead of the page
Cloudflare, Akamai and DataDome sit in front of a large share of the commercial web. When they decide a request looks automated, they return a challenge page. That page has a status code, HTML, and its own perfectly valid <title>, usually something like “Just a moment…”.
This is worse than an empty result, because it does not look like a failure. You get a filled cell with the wrong contents. Unless you read the values, you never notice.
How to confirm: sort your export by title. If unrelated domains share one title, you checked the same challenge page repeatedly.
The fix: a real browser fingerprint and an IP that does not look like a datacenter. If your requests arrive from a cloud provider’s address range, some sites refuse them no matter how the request is shaped. No amount of parsing logic gets around that, it is what резидентские прокси are for.
You hit a rate limit
Fifty requests to one domain inside ten seconds is not a browsing pattern. Large publishers, ecommerce platforms and anything behind a CDN will start returning 429s, timeouts or empty bodies partway through the run.
The tell is positional. Early URLs succeed and later ones fail, which is a pattern no page-side problem produces.
How to confirm: re-run just the failures on their own. If they pass in isolation, the pages were never the problem.
The fix: slow down and spread out. Add a delay between requests, cap concurrency per domain, rotate the source IP for larger jobs. A 50-URL list pointing at one domain needs more care than 50 URLs across 50 domains.
The page serves different metadata by country
International sites vary titles and descriptions by locale, redirect visitors to a country subfolder, or serve a different hreflang set depending on where the request comes from.
Audit example.com/product from Istanbul and you may be reading the Turkish page. Your German colleague runs the same check and gets different values. Neither of you is wrong.
How to confirm: compare the URL you requested against the URL that answered. A redirect to a country subfolder is the visible version of this problem. The invisible version keeps the URL and swaps the content.
The fix: check from the country you care about. This one is easy to underestimate, so the next section takes it apart properly.
A 200 status on the wrong page
Soft 404s return a success code with an error page. Redirect chains land somewhere other than where you aimed. Either way the metadata is real and complete, and it describes a different URL than the one in your spreadsheet.
How to confirm: record the final URL alongside the requested one and compare the two columns.
The fix: nothing technical. Read the columns. It costs nothing and catches the errors that look most like clean data.
The metadata you can’t see from one country
If you audit an international site from a single location, you are auditing one version of it.
Say a page carries a German title, a French title and an English fallback. From one IP you see one of them. The other two stay invisible, and nothing in your export hints that they exist. Nothing looks wrong. The CSV has no blanks, and the titles your German customers see never entered the spreadsheet.
The same applies to hreflang. A page can declare a full set of language alternates from one region and a truncated set from another. Checking return tags from a single country will show you a valid configuration that is broken for everyone else.
Three checks worth running on any site with international traffic:
- Does the requested URL survive? Compare what you asked for against what answered. A redirect to a country subfolder means every subsequent field describes a different page.
- Does the title change without the URL changing? Same address, different content, no signal in the response. This is the one that slips past every audit.
- Is the
hreflangset the same from each region? Missing return tags are a common cause of the wrong language ranking in the wrong market.
All three mean sending the request from the country you care about, which is what прокси для веб-скрейпинга are for.
Be precise about the reason, though. Here a proxy is the measuring instrument, not a way around a blocking system. Auditing German metadata from Turkey is like checking a translation in a language you cannot read.
A practical workflow for a 50-URL audit
- Export the URL list from your sitemap, Search Console or CMS. Keep it under 50 per run.
- Run everything on the default fetch mode first. Most URLs will resolve, and the cheap pass tells you which ones will not.
- Sort by title before you look at the blanks. Identical titles across unrelated domains mean bot protection answered. This is the failure that hides.
- Compare requested URL against final URL. Any mismatch invalidates every other field in that row.
- Re-run the failures with browser rendering. This recovers the JavaScript-built pages and some of the protected ones.
- Re-run anything locale-sensitive from the target country. Only for international sites, and only for pages that matter commercially.
- Export to CSV and sort by issue rather than by URL. Missing titles, missing descriptions and accidental
noindexdirectives are three separate tickets, and sorting by issue groups the work.
