HTTP Header Directive Audit

Most technical review reads the HTML, but several directives live in the response headers instead, where an edge layer can add, strip or override them with no change to the application and no trace in view-source. Headers are also the only way to control indexing for files that have no head element at all: PDFs, images, CSV exports and API responses. Run this whenever the application says one thing and the index shows another.

The skill file

---
name: http-header-directive-audit
description: Use when directives appear to be ignored, when PDFs or other non-HTML files show up in search, or whenever a CDN, edge worker or reverse proxy sits between the application and the crawler.
---

# HTTP Header Directive Audit

Most technical review reads the HTML, but several directives live in the response headers instead, where an edge layer can add, strip or override them with no change to the application and no trace in view-source. Headers are also the only way to control indexing for files that have no head element at all: PDFs, images, CSV exports and API responses. Run this whenever the application says one thing and the index shows another.

## What you need first

- The ability to request URLs and read raw response headers from outside your own network, since an internal request may bypass the CDN altogether
- An inventory of publicly reachable non-HTML URLs: documents, downloads, media, exports, API endpoints
- The CDN or edge configuration including page rules, workers, geo rules and any security product in the path
- A crawl export flagging indexable non-HTML URLs

## Method

1. Request one URL per content type, headers only, from outside your network and without cookies. An office IP, a VPN or a staging bypass returns headers no crawler will ever receive.
2. Look specifically for X-Robots-Tag and a Link rel=canonical header. Both are honoured, both are invisible in the page source, and both are typically set once by a developer and never revisited.
3. Resolve every conflict between header and HTML. Where a header noindex meets an HTML index tag the restrictive one wins, and the header is the one nobody thinks to check.
4. Work through the non-HTML inventory. A PDF duplicating a page, an old export, staging media: all are indexable by default, and the header is the only way to exclude them while keeping them crawlable.
5. Confirm robots.txt is served by the origin you believe it is. A CDN, a hosting platform or a security product can serve its own, and a 5xx on robots.txt suspends crawling of the entire host rather than just that file.
6. Repeat the requests from another region and with a mobile user agent. Country-based redirects and blocks show a crawler a different site from the one you are auditing, because Googlebot crawls predominantly from US addresses.
7. Check Vary and cache headers wherever the response differs by user agent or device, and confirm the edge is not caching one audience's response and serving it to another.

## What this produces

A header map per content type and per edge rule, listing every directive sent, the layer that set it, and how each header-versus-HTML conflict was resolved.

## Where this goes wrong

- Auditing through browser dev tools with your session cookies attached, so you read the logged-in variant instead of the crawler variant
- Applying X-Robots-Tag noindex to a whole directory and catching pages inside it that were meant to rank
- Assuming robots.txt is yours to control when the platform or the security layer serves its own version
- Blocking a PDF in robots.txt and expecting it to leave the index, when the file has to stay crawlable for the noindex header to be read at all

---

From the QuQi skill library - https://www.quqi.io/skills/http-header-directive-audit

What you need first

• The ability to request URLs and read raw response headers from outside your own network, since an internal request may bypass the CDN altogether
• An inventory of publicly reachable non-HTML URLs: documents, downloads, media, exports, API endpoints
• The CDN or edge configuration including page rules, workers, geo rules and any security product in the path
• A crawl export flagging indexable non-HTML URLs

Method

1. 01 Request one URL per content type, headers only, from outside your network and without cookies. An office IP, a VPN or a staging bypass returns headers no crawler will ever receive.
2. 02 Look specifically for X-Robots-Tag and a Link rel=canonical header. Both are honoured, both are invisible in the page source, and both are typically set once by a developer and never revisited.
3. 03 Resolve every conflict between header and HTML. Where a header noindex meets an HTML index tag the restrictive one wins, and the header is the one nobody thinks to check.
4. 04 Work through the non-HTML inventory. A PDF duplicating a page, an old export, staging media: all are indexable by default, and the header is the only way to exclude them while keeping them crawlable.
5. 05 Confirm robots.txt is served by the origin you believe it is. A CDN, a hosting platform or a security product can serve its own, and a 5xx on robots.txt suspends crawling of the entire host rather than just that file.
6. 06 Repeat the requests from another region and with a mobile user agent. Country-based redirects and blocks show a crawler a different site from the one you are auditing, because Googlebot crawls predominantly from US addresses.
7. 07 Check Vary and cache headers wherever the response differs by user agent or device, and confirm the edge is not caching one audience's response and serving it to another.

What this produces

A header map per content type and per edge rule, listing every directive sent, the layer that set it, and how each header-versus-HTML conflict was resolved.

Where this goes wrong

• Auditing through browser dev tools with your session cookies attached, so you read the logged-in variant instead of the crawler variant
• Applying X-Robots-Tag noindex to a whole directory and catching pages inside it that were meant to rank
• Assuming robots.txt is yours to control when the platform or the security layer serves its own version
• Blocking a PDF in robots.txt and expecting it to leave the index, when the file has to stay crawlable for the noindex header to be read at all

Use this skill in your own AI

The download is a plain markdown file with the name and trigger in its frontmatter. Where an assistant supports skills it can load itself, that frontmatter is what it reads to decide this one applies.

Questions about this skill

More in Technical SEO