Select JSON-LD metadata across scripts and support non-article types - #1030
Draft
vvinayakkk wants to merge 1 commit into
Draft
vvinayakkk wants to merge 1 commit into
vvinayakkk wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Recipe pages can expose useful metadata in JSON-LD that Readability currently skips. A string
@type: "Recipe"is outside the article-type filter, and an array-valued@typethrows when the existing parser calls.match()on it. Simply broadening the filter also allows an earlier video or publisher block to prevent extraction of the main content's metadata.This change accepts non-empty string types, including members of
@typearrays, and separates candidate collection from selection. It collects Schema.org entities across scripts, root arrays, and@grapharrays, then selects:mainEntityOfPagematches the document or canonical URL.A date means a non-empty string
datePublished. Page references support strings,@idobjects,urlobjects when@idis absent, and arrays. Relative URLs resolve against the document base URI; fragments are ignored. Types are not ranked and candidate titles are not compared with the HTML title.The existing extraction of title, author, description, publisher, and publication date from the selected entity is retained, including its existing name-versus-headline check. HTML metadata fallback precedence and content extraction are unchanged. Metadata is not merged across JSON-LD entities.
Examples
@type: ["Recipe", "NewsArticle"]and an@idpage association. The selected publication date is2000-07-11T13:40:38-04:00.VideoObjectbefore a datedRecipein a separate script. The selection returns recipe metadata and2025-03-31T18:10:12+01:00, rather than stopping at the video.@graphcontaining an undatedOrganizationand a datedRecipe, the organization does not prevent selection of the recipe.The checked-in real-page fixtures preserve reduced JSON-LD selection fields and order with synthetic body text. They exercise public
parse()with both jsdom and JSDOMParser and run offline.Behavior changes and limitations
This deliberately changes selection policy beyond array-type support. An undated
NewsArticlewithout a matchingmainEntityOfPageno longer supplies its JSON-LD title, author, or description. HTML fallbacks may provide less complete metadata. Please review whether this tradeoff is appropriate for Readability.First-dated fallback is deterministic, not a guarantee of relevance: a dated video before a dated article can win if neither has a distinguishing page association. A dated candidate also takes precedence over an undated page-associated candidate. Tests document these cases. The implementation is not a general JSON-LD processor: it does not resolve graph references, reverse
mainEntity, remote contexts, or arbitrary nested entities, and does not validate dates.Validation
Reproducible coverage: selection tests, real-page layout tests, and captured selection fields. Run
npm ci,npm test, andnpm run lintfrom the repository root.npm test: 2069 passing on Node v26.5.0.npm run lint: passed.git diff --check: passed.parse()on previously downloaded full HTML for Allrecipes, Good Food, and InStyle; expected publication dates returned with non-empty extracted content. These are saved-page checks, not CI network tests.Related: #1013 already addresses array-valued
@typeindependently. This proposal additionally broadens accepted types and changes cross-entity selection.