Skip to content

fix(crawl): a one-shot headless fetch reads the page; a headless crawl still pays - #402

Merged
sh1pt-actions-fleet[bot] merged 1 commit into
masterfrom
riot/plain-fetch-1028
Oct 28, 2026
Merged

sh1pt-actions-fleet[bot] merged 1 commit into
masterfrom
riot/plain-fetch-1028

Conversation

@sh1pt-actions-fleet

Copy link
Copy Markdown
Contributor

crawlproof added HEADLESS_AGENTS to its own PAID_CRAWLERS (charged on sight), with a comment saying this matched the gateway default. Since x402-gateway 0.16 (profullstack/x402-gateway#29) the default gives headless strings 20 pages an hour per address, then the 402. Headless agents now leave PAID_CRAWLERS so the allowance applies, and stay in FAMILIES for the counters. Gateway bumped to ^0.16.1. GPTBot and commercial crawlers are unchanged.

Tests: crawl-gateway + tracker + email-tracking + ads-terminal pass, tsc clean.

🤖 Generated with Claude Code

…l still pays

Headless browsers come off PAID_CRAWLERS (charged on sight) and are left to
x402-gateway 0.16.1's headless list: 20 pages an hour per address, then the
402. They stay a labelled family for the counters. Reported 2026-10-28: a
one-shot fetch of the homepage with a headless string got 402.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@socket-security

Copy link
Copy Markdown

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff Package Supply Chain
Security
Vulnerability Quality Maintenance License
Updated@​profullstack/​x402-gateway@​0.16.0 ⏵ 0.16.184 +1100100 +197100

View full report

@github-actions

Copy link
Copy Markdown

ThreatCrush Security Scan

47 finding(s)

HIGH/CRITICAL: 2 | MEDIUM: 29 | LOW: 16

Severity Rule Location
HIGH tls-verification-disabled lib/onion.ts:48
HIGH secret-generic-credential lib/sp/platforms/facebook.ts:32
MEDIUM js-unescaped-html-sink app/(app)/dashboard/admin/email-broadcast/EmailBroadcastForm.tsx:125
MEDIUM js-unescaped-html-sink app/(app)/dashboard/projects/[id]/autoblog/articles/[articleId]/page.tsx:214
MEDIUM js-unescaped-html-sink app/(marketing)/blog/[slug]/page.tsx:71
MEDIUM js-unescaped-html-sink app/(marketing)/blog/[slug]/page.tsx:99
MEDIUM js-unescaped-html-sink app/(marketing)/recent/page.tsx:186
MEDIUM js-unescaped-html-sink app/(marketing)/recent/page.tsx:190
MEDIUM js-unescaped-html-sink app/c/[project]/[slug]/page.tsx:77
MEDIUM js-unescaped-html-sink app/c/[project]/page.tsx:57
MEDIUM js-unescaped-html-sink app/careers.js/route.ts:228
MEDIUM js-unescaped-html-sink app/careers.js/route.ts:285
MEDIUM js-unescaped-html-sink app/layout.tsx:129
MEDIUM js-open-redirect app/login/form.tsx:39
MEDIUM js-unescaped-html-sink app/r/[token]/page.tsx:176
MEDIUM js-open-redirect app/signup/form.tsx:63
MEDIUM js-unescaped-html-sink components/json-ld.tsx:8
MEDIUM js-unescaped-html-sink components/pit-footer.tsx:17
MEDIUM js-unescaped-html-sink components/report/markdown-view.tsx:16
MEDIUM js-unescaped-html-sink lib/careers/page-templates.ts:198
MEDIUM js-dynamic-code-execution lib/crawl-limits.ts:67
MEDIUM redos-nested-quantifier lib/emailMarkdown.ts:41
MEDIUM redos-nested-quantifier lib/emailMarkdown.ts:324
MEDIUM redos-nested-quantifier lib/lx/articleGen.ts:100
MEDIUM redos-nested-quantifier lib/tracker/agent-gate.ts:61
MEDIUM sh-predictable-temp-path ops/selfhost/server/setup-supabase.sh:218
MEDIUM sql-template-interpolation scripts/detect-slot-themes.ts:31
MEDIUM sql-template-interpolation scripts/purge-constructed-keywords.ts:163
MEDIUM sql-template-interpolation scripts/purge-offniche-keywords.ts:124
MEDIUM js-dynamic-code-execution scripts/test-crawl-limits.mjs:14
MEDIUM js-dynamic-code-execution scripts/test-crawl-limits.mjs:24
LOW secret-generic-credential app/(marketing)/docs/autoblog-webhook/page.tsx:156
LOW secret-generic-credential lib/sealed/migrate.ts:38
LOW secret-generic-credential lib/sp/platforms/linkedin.ts:25
LOW js-dynamic-code-execution tests/careers-page-templates.test.ts:21
LOW js-dynamic-code-execution tests/careers-widget-script.test.ts:19
LOW js-dynamic-code-execution tests/careers-widget-script.test.ts:69
LOW js-dynamic-code-execution tests/contract/ad-visitor-id.test.ts:51
LOW js-dynamic-code-execution tests/contract/ad-visitor-id.test.ts:52
LOW js-dynamic-code-execution tests/contract/ads-click-cooldown-redis.test.ts:20
LOW js-dynamic-code-execution tests/contract/ads-click-cooldown-redis.test.ts:24
LOW js-dynamic-code-execution tests/contract/ads-click-cooldown-redis.test.ts:25
LOW js-dynamic-code-execution tests/contract/ads-click-cooldown-redis.test.ts:26
LOW js-dynamic-code-execution tests/contract/ads-click-cooldown-redis.test.ts:31
LOW js-dynamic-code-execution tests/contract/ads-click-cooldown-redis.test.ts:35
LOW secret-generic-credential tests/contract/posthog-integration.test.ts:13
LOW secret-generic-credential tests/lead-campaign.test.ts:16

Snippets are redacted; ThreatCrush never prints matched credential material.

@sh1pt-actions-fleet
sh1pt-actions-fleet Bot merged commit 57bab5e into master Oct 28, 2026
10 checks passed
@sh1pt-actions-fleet
sh1pt-actions-fleet Bot deleted the riot/plain-fetch-1028 branch October 28, 2026 23:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant