Skip to content

fix(crawl): a one-shot headless fetch reads the page; a headless crawl still pays #402

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
sh1pt-actions-fleet merged 1 commit into from
Oct 28, 2026
Merged

fix(crawl): a one-shot headless fetch reads the page; a headless crawl still pays #402

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions bun.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

11 changes: 7 additions & 4 deletions lib/crawl-policy.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,13 @@ import { HEADLESS_AGENTS, TRAINING_AGENTS, isTrainingAgent } from "@profullstack
import { parseDevice } from "@/lib/tracker/device";

export const COMMERCIAL_CRAWLERS = ["SemrushBot", "SiteAuditBot", "AhrefsBot", "AhrefsSiteAudit", "MJ12bot", "DotBot", "BLEXBot", "DataForSeoBot"];
// Headless browsers that say so (Lightpanda, HeadlessChrome, PhantomJS): nobody
// reads a page through one; the gateway's default list charges them since 0.8.
export const PAID_CRAWLERS = [...TRAINING_AGENTS, ...COMMERCIAL_CRAWLERS, ...HEADLESS_AGENTS];
const FAMILIES = [...PAID_CRAWLERS, "Amazonbot", "Googlebot", "Bingbot", "Applebot", "OAI-SearchBot", "ChatGPT-User", "Claude-SearchBot", "Claude-User", "PerplexityBot", "Perplexity-User"];
// Headless browsers that say so (Lightpanda, HeadlessChrome, PhantomJS) are not
// on this list since gateway 0.16: the gateway's own `headless` list gives them
// 20 pages an hour per address and charges after, so an agent fetching one page
// reads it (2026-10-28: a one-shot headless fetch of / got 402) and a headless
// crawl still pays.
export const PAID_CRAWLERS = [...TRAINING_AGENTS, ...COMMERCIAL_CRAWLERS];
const FAMILIES = [...PAID_CRAWLERS, ...HEADLESS_AGENTS, "Amazonbot", "Googlebot", "Bingbot", "Applebot", "OAI-SearchBot", "ChatGPT-User", "Claude-SearchBot", "Claude-User", "PerplexityBot", "Perplexity-User"];

/** Labels bound counter cardinality. A user agent identifies policy, not ownership. */
export function crawlerFamily(ua: string | null): string | null {
Expand Down
2 changes: 1 addition & 1 deletion package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@
"@profullstack/stack": "^0.5.0",
"@profullstack/webhooks": "0.2.1",
"@profullstack/x402-client": "^0.3.0",
"@profullstack/x402-gateway": "^0.16.0",
"@profullstack/x402-gateway": "^0.16.1",
"@supabase/ssr": "^0.12.7",
"@supabase/supabase-js": "^2.118.0",
"@types/nodemailer": "^8.0.2",
Expand Down
10 changes: 7 additions & 3 deletions tests/crawl-gateway.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -121,9 +121,13 @@ describe("crawler access through x402-gateway", () => {
});

describe("headless browsers", () => {
it("charges a self-declared headless browser like any paid crawler", async () => {
it("lets a self-declared headless browser read a page, then charges a crawl (gateway 0.16)", async () => {
const { gate } = await import("@/lib/crawl-gateway");
expect((await gate(request("/blog", "Lightpanda/1.0")))?.status).toBe(402);
expect((await gate(request("/blog", "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/155.0.0.0 Safari/537.36")))?.status).toBe(402);
const headless = "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/155.0.0.0 Safari/537.36";
const from = (path: string, ua: string) => request(path, ua, undefined, undefined, "203.0.113.61");
expect(await gate(from("/", headless))).toBeUndefined();
let last: Response | undefined;
for (let i = 0; i < 20; i++) last = await gate(from(`/blog/${i}`, "Lightpanda/1.0"));
expect(last?.status).toBe(402);
});
});