feat: pom fetcher

[ulemons]

Summary

Adds a Maven POM fetcher to the packages_worker service that syncs Maven Central package metadata into the packages DB. It pulls candidates from packages_universe, extracts metadata from POM files (with parent-chain resolution), and populates package, version, maintainer, and repository data. This brings the Maven ecosystem to parity with the existing npm pipeline so critical Maven packages get high-quality, enriched metadata for downstream analytics.

Changes

• Two-tier fetch strategy — non-critical packages are DB-only (copy universe stats, no HTTP, ~1000 pkg/sec); critical packages get full POM extraction with parent-chain resolution (max 8 hops) for description, homepage, SCM/repo, licenses, maintainers, and the full version list.
• Two entry points — bin/packages-worker.ts registers the maven-critical Temporal schedule for incremental syncing (skips POM extraction when the version is unchanged), and bin/maven-backfill.ts (pnpm backfill:maven) does a one-shot, resumable full-extraction backfill. The DB state is the cursor, so re-runs pick up where they left off.
• Module-level parent POM cache (extract.ts) — coordinate-keyed LRU with request coalescing, caches only successful fetches (never null, to avoid poisoning), no TTL since Maven coordinates are immutable. This is the main lever against Maven Central rate limiting and works because the rank_in_ecosystem ordering clusters sibling artifacts that share parent POMs. Exposes getPomCacheStats() for hit-rate observability.
• New osspckgs data-access-layer module — query functions for packages, versions, maintainers, and repos (functional, pg-promise via queryExecutor), shared across the worker.
• Delta API support (deltaApi.ts) for incremental upstream change detection, plus benchmark and data-quality validation scripts.
• Adds unit tests for the pure normalization functions.
• Maven-specific config in config.ts (POM_FETCHER_REFRESH_DAYS, POM_CACHE_MAX_ENTRIES, etc.) and .env.dist.local entries.

Type of change

• Bug fix
• New feature
• Refactor / cleanup
• Performance improvement
• Chore / dependency update
• Documentation

---

Note

Medium Risk
Large new external-ingestion path with high write volume to packages-db and aggressive default batch/concurrency in sample env; mitigated by idempotent upserts and rate-limit handling, but operational misconfiguration could throttle Maven Central or overload the DB.

Overview
Adds a Maven POM enrichment pipeline in packages_worker that syncs critical Maven packages from Central into packages-db (packages, versions, maintainers, repos), with a DB-only path for non-critical packages that is implemented but not scheduled (non-critical Temporal registration is commented out).

Runtime: packages-worker now registers a maven-critical Temporal schedule (cron */1 in code) that runs one batch per tick with incremental behavior (skip full POM work when latest_version is unchanged). maven-backfill drains the critical queue in a resumable one-shot with full extraction. A separate maven-worker entrypoint and compose service run only the Maven schedule for local isolation. New required env: POM_FETCHER_* and optional POM_FETCHER_MAVEN_BASE_URL (defaults to Central; sample env points at the GCS mirror).

Implementation highlights: HTTP fetch of maven-metadata.xml and POMs via axios / fast-xml-parser, parent POM resolution (up to 8 hops) with an in-process LRU cache and request coalescing, namespace-ordered batches for cache locality, 403/429 backoff, and transactional upserts with deadlock retry. @crowd/data-access-layer gains osspckgs helpers (listMavenPackagesToSync, upsertPackage, version/maintainer/repo upserts, audit). Maintainer emails are stored as SHA-256 hashes. Unit tests cover prerelease detection, repo URL parsing, and SCM normalization.

Ops / misc: pnpm-lock.yaml picks up axios and fast-xml-parser; local .env.dist.local fills placeholder GCS creds and POM fetcher tuning; packages-worker build metadata references maven-worker.

^{Reviewed by Cursor Bugbot for commit 02a4d95. Bugbot is set up for automated code reviews on this repo. Configure here.}

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
0 out of 2 committers have signed the CLA.

❌ mbani01
❌ ulemons
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[github-actions]

Conventional Commits FTW!

[Copilot]

Pull request overview

Introduces a new Maven “POM fetcher” worker loop to enrich packages data in the osspckgs database by fetching Maven Central metadata/POMs, and adds corresponding data-access-layer helpers for selecting candidates and upserting packages/maintainers.

Changes:

• Added @crowd/data-access-layer osspckgs module with queries for Maven enrichment candidates and upserts into packages, maintainers, and package_maintainers.
• Added a pom-fetcher worker (config + entrypoint + enrichment loop) that resolves latest Maven versions and extracts POM metadata (licenses, SCM, developers/contributors).
• Wired up scripts/deps for running the new worker (package.json scripts, docker-compose service yaml, lockfile updates).

Reviewed changes

Copilot reviewed 11 out of 13 changed files in this pull request and generated 12 comments.

Show a summary per file

File	Description
services/libs/data-access-layer/src/osspckgs/types.ts	Adds DB-facing types for osspckgs package/maintainer upserts and universe rows.
services/libs/data-access-layer/src/osspckgs/packages.ts	Adds query to list Maven universe packages needing enrichment + upsert into packages.
services/libs/data-access-layer/src/osspckgs/maintainers.ts	Adds upserts for maintainers and package_maintainers.
services/libs/data-access-layer/src/osspckgs/index.ts	Re-exports osspckgs DAL surface.
services/libs/data-access-layer/src/index.ts	Exposes osspckgs DAL from the package root.
services/apps/packages_worker/src/pom-fetcher/runPomEnrichmentLoop.ts	Implements batch/concurrent enrichment loop and persistence of extracted metadata.
services/apps/packages_worker/src/pom-fetcher/metadata.ts	Resolves latest version via maven-metadata.xml.
services/apps/packages_worker/src/pom-fetcher/extract.ts	Fetches POMs and extracts fields with limited parent inheritance traversal.
services/apps/packages_worker/src/config.ts	Adds pom-fetcher config loader.
services/apps/packages_worker/src/bin/pom-fetcher.ts	Adds runnable entrypoint with shutdown handling.
services/apps/packages_worker/package.json	Adds scripts and deps (axios, fast-xml-parser) for pom-fetcher.
scripts/services/pom-fetcher.yaml	Adds docker-compose service definition for pom-fetcher.
pnpm-lock.yaml	Updates lockfile for new deps (but includes an unexpected workspace importer).

Files not reviewed (1)

• pnpm-lock.yaml: Language not supported

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 21 out of 24 changed files in this pull request and generated 16 comments.

Files not reviewed (1)

• pnpm-lock.yaml: Language not supported

[Copilot]

Pull request overview

Copilot reviewed 25 out of 28 changed files in this pull request and generated 9 comments.

Files not reviewed (1)

• pnpm-lock.yaml: Language not supported

[Copilot]

Pull request overview

Copilot reviewed 24 out of 27 changed files in this pull request and generated 15 comments.

Files not reviewed (1)

• pnpm-lock.yaml: Language not supported

[github-actions]

⚠️ Jira Issue Key Missing

Your PR title doesn't contain a Jira issue key. Consider adding it for better traceability.

Example:

• feat: add user authentication (CM-123)
• feat: add user authentication (IN-123)

Projects:

• CM: Community Data Platform
• IN: Insights

Please add a Jira issue key to your PR title.

[Copilot]

Pull request overview

Copilot reviewed 24 out of 27 changed files in this pull request and generated 12 comments.

Files not reviewed (1)

• pnpm-lock.yaml: Language not supported

[Copilot]

Pull request overview

Copilot reviewed 24 out of 27 changed files in this pull request and generated 10 comments.

Files not reviewed (1)

• pnpm-lock.yaml: Language not supported

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 3c087e1. Configure here.}

[Copilot]

Pull request overview

Copilot reviewed 23 out of 27 changed files in this pull request and generated 12 comments.

Files not reviewed (1)

• pnpm-lock.yaml: Language not supported