Add Hugo Algolia indexing - #968
Conversation
424db81 to
c0a0f9e
Compare
There was a problem hiding this comment.
Pull request overview
This PR migrates Algolia indexing from the prior Jekyll plugin approach to a Hugo-generated JSON export (public/algolia.json) plus a standalone Node.js indexer that validates/transforms the export and (optionally) uploads records to Algolia via GitHub Actions.
Changes:
- Adds a Node.js CLI (
tools/algolia-index.mjs) to validate Hugo’s export, create/split Algolia records, and upload them with deterministic object IDs. - Introduces a Hugo output template (
layouts/index.algolia.json) to emit the Algolia export during the Hugo build. - Adds Node tooling/tests plus a revamped scheduled workflow to build, validate (dry-run), and publish the Algolia index.
Reviewed changes
Copilot reviewed 6 out of 7 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
tools/algolia-index.mjs |
New Node.js indexer: parses Hugo export HTML, builds/splits records, and uploads with Algolia settings. |
test/algolia-index.test.mjs |
Adds Node test-runner coverage for record creation, deterministic IDs, callout exclusion, and splitting behavior. |
package.json |
Defines Node module tooling (scripts/deps) and declares the supported Node engine range. |
package-lock.json |
Locks Algolia/Cheerio (and transitive) dependency versions for reproducible CI installs. |
layouts/index.algolia.json |
Hugo output template that collects searchable pages and emits the JSON export. |
config/_default/hugo.toml |
Updates Algolia index name to the new Hugo-based index. |
.github/workflows/update-algolia.yml |
Reworks the scheduled/manual workflow to build Hugo export, run tests, dry-run validate, and upload records. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
MakisH
left a comment
There was a problem hiding this comment.
I do not have much contact with Javascript to review the .mjs files, so I can only trust what the Copilot review says right now, and that you know what you are doing / the testing you have performed.
On https://muhammadaashiraslam.github.io/precice.github.io/, if I add, precice.github.io to e.g., https://muhammadaashiraslam.github.io/docs/installation/overview/#installing-the-core-library to the resulting URLs (in this case, to get https://muhammadaashiraslam.github.io/precice.github.io/docs/installation/overview/), the search works.
Just a couple of minor comments / questions.
There was a problem hiding this comment.
The test/ directory does not exist at the moment, and it looks like this will be the only test added in the near future. I understand that it makes sense since the script is in-house, but I would keep it in the same folder as the main script (tools/).
| pages: [ | ||
| { | ||
| title: "Example guide", | ||
| url: "/docs/example/", | ||
| html: ` | ||
| <h2 id="install">Install</h2> | ||
| <p>Use <code>precice-config</code> to configure the adapter.</p> | ||
| <table><tbody><tr><td>Version</td><td>3.4</td></tr></tbody></table> | ||
| <script>ignored content</script> | ||
| `, | ||
| plain: "Use precice-config to configure the adapter. Version 3.4", | ||
| section: "docs", | ||
| kind: "page", | ||
| tags: ["adapter"], | ||
| categories: [], | ||
| keywords: "configuration", | ||
| date: 1_784_198_400, | ||
| }, |
There was a problem hiding this comment.
I would keep this example more generic: I would remove mentions to preCICE and concrete versions. This could cause some confusion while searching through the codebase.
Similarly below.
| "license": "MIT", | ||
| "dependencies": { | ||
| "iconv-lite": "^0.6.3", | ||
| "whatwg-encoding": "^3.1.1" |
There was a problem hiding this comment.
I am getting a deprecation warning here:
tools [hugo-algolia]$ npm install
npm warn deprecated whatwg-encoding@3.1.1: Use @exodus/bytes instead for a more spec-conformant and faster implementation
There was a problem hiding this comment.
Yes but this warning comes from the package itself , as it hasnot upgraded to the new dependecy . This is fine in our case
| "node_modules/undici": { | ||
| "version": "7.28.0", | ||
| "resolved": "https://registry.npmjs.org/undici/-/undici-7.28.0.tgz", | ||
| "integrity": "sha512-cRZYrTDwWznlnRiPjggAGxZXanty6M8RV1ff8Wm4LWXBp7/IG8v5DnOm74DtUBp9OONpK75YlPnIjQqX0dBDtA==", | ||
| "license": "MIT", | ||
| "engines": { | ||
| "node": ">=20.18.1" | ||
| } |
There was a problem hiding this comment.
I am getting a vulnerability report here:
tools [hugo-algolia]$ npm audit
# npm audit report
undici 7.0.0 - 7.28.0
Severity: high
undici vulnerable to downstream response desynchronization via retry interceptor - https://github.com/advisories/GHSA-8xcm-r25x-g524
undici vulnerable to cross-user information disclosure and parse-time crash via degenerate private cache directives - https://github.com/advisories/GHSA-4cwx-7wf7-3272
undici vulnerable to CRLF Injection via blob-like body 'type' property - https://github.com/advisories/GHSA-m8rv-5g2x-5cg5
undici vulnerable to cross-user information disclosure via whitespace around equals in Cache-Control directives - https://github.com/advisories/GHSA-jr45-8vmc-qm54
undici vulnerable to cookie attribute injection via unsanitized domain and unparsed setCookie fields - https://github.com/advisories/GHSA-v3r7-h72x-cjcm
fix available via `npm audit fix`
node_modules/undici
1 high severity vulnerability
To address all issues, run:
npm audit fix
There was a problem hiding this comment.
I tried the dry-run from the PR description, but it failed. What am I missing?
tools $ npm run algolia:index -- --dry-run
> algolia:index
> node algolia-index.mjs --dry-run
Algolia indexing failed: Unable to read Algolia export at /home/gc/repos/precice/website/tools/public/algolia.json: ENOENT: no such file or directory, open '/home/gc/repos/precice/website/tools/public/algolia.json'
There was a problem hiding this comment.
Yeah it needs to have the public/algolia.json , which i assume is not working right now due to some legacy code in content/docs/docs-meta/publish-to-pdf.md but if you want simply replace lines 140–141 in content/docs/docs-meta/publish-to-pdf.md (this is solved by #972 )
https://precice.org/titlepage.html
https://precice.org/tocpage.html
Then run hugo or hugo --gc this will create the public/algolia.json
This PR replaces the previous Jekyll Algolia plugin with a Hugo JSON export and a standalone Node.js
indexing tool.
layouts/index.algolia.json
This is a Hugo output template, not the Algolia index itself. During the Hugo build, it collects searchable
pages and writes them to public/algolia.json.
For each page, it stores:
Pages marked with search: exclude are skipped. The template also uses the existing content compatibility
layer so imported documentation is indexed after its legacy markup has been converted to Hugo-compatible
output.
tools/algolia-index.mjs
This is the standalone command-line indexer. It reads the JSON generated by Hugo and converts it into
records that Algolia can search.
The script:
The command supports a dry-run mode:
npm run algolia:index -- --dry-run
A dry run performs all parsing and validation locally without contacting Algolia.
For an actual upload, the script reads the write credential from the environment. The application ID and
index name are supplied by the workflow, while the write key remains a repository secret.
test/algolia-index.test.mjs
These tests use Node.js’s built-in test runner, so no additional test framework is required.
The tests cover the important indexing behavior:
They run with:
npm run test:algolia
package.json
This file defines the Node.js tooling used by the Algolia implementation.
It provides:
The workflow installs these dependencies with npm ci, which uses the lockfile rather than resolving
versions again.
package-lock.json
package-lock.json is generated automatically by npm from package.json. It records the exact versions,
download locations, and integrity checksums of the direct and transitive Node.js dependencies.
It is committed so that local development and GitHub Actions install the same dependency tree every time.
It should not be edited manually; it should be regenerated with npm when package.json changes.
Workflow
The workflow performs the indexing in several controlled stages:
The application ID and index name are configured to match the Hugo configuration. The workflow reads only
ALGOLIA_API_KEY from GitHub repository secrets for the write credential.