Feat/pom fetcher#4144
Draft
ulemons wants to merge 14 commits into
Draft
Conversation
Signed-off-by: Uroš Marolt <uros@marolt.me>
Signed-off-by: Mouad BANI <mouad-mb@outlook.com>
Signed-off-by: Mouad BANI <mouad-mb@outlook.com>
Signed-off-by: Mouad BANI <mouad-mb@outlook.com>
…dRepos) Signed-off-by: Mouad BANI <mouad-mb@outlook.com>
…plicitly Signed-off-by: Mouad BANI <mouad-mb@outlook.com>
Signed-off-by: Mouad BANI <mouad-mb@outlook.com>
Signed-off-by: Mouad BANI <mouad-mb@outlook.com>
…ved to packages_worker) Signed-off-by: Mouad BANI <mouad-mb@outlook.com>
Signed-off-by: Mouad BANI <mouad-mb@outlook.com>
Signed-off-by: Mouad BANI <mouad-mb@outlook.com>
Signed-off-by: Joana Maia <jmaia@contractor.linuxfoundation.org>
Signed-off-by: Mouad BANI <mouad-mb@outlook.com>
Signed-off-by: Umberto Sgueglia <usgueglia@contractor.linuxfoundation.org>
|
|
![github-license-compliance[bot]](https://ghub.bbstale.com/avatars/u/9919?s=60&v=4){kind=small}
![github-actions[bot]](https://ghub.bbstale.com/avatars/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
PR titles must follow Conventional Commits. Love from, Your reviewers ❤️.
Contributor
There was a problem hiding this comment.
Pull request overview
Introduces a new Maven “POM fetcher” worker loop to enrich packages data in the osspckgs database by fetching Maven Central metadata/POMs, and adds corresponding data-access-layer helpers for selecting candidates and upserting packages/maintainers.
Changes:
- Added
@crowd/data-access-layerosspckgs module with queries for Maven enrichment candidates and upserts intopackages,maintainers, andpackage_maintainers. - Added a
pom-fetcherworker (config + entrypoint + enrichment loop) that resolves latest Maven versions and extracts POM metadata (licenses, SCM, developers/contributors). - Wired up scripts/deps for running the new worker (package.json scripts, docker-compose service yaml, lockfile updates).
Reviewed changes
Copilot reviewed 11 out of 13 changed files in this pull request and generated 12 comments.
Show a summary per file
| File | Description |
|---|---|
| services/libs/data-access-layer/src/osspckgs/types.ts | Adds DB-facing types for osspckgs package/maintainer upserts and universe rows. |
| services/libs/data-access-layer/src/osspckgs/packages.ts | Adds query to list Maven universe packages needing enrichment + upsert into packages. |
| services/libs/data-access-layer/src/osspckgs/maintainers.ts | Adds upserts for maintainers and package_maintainers. |
| services/libs/data-access-layer/src/osspckgs/index.ts | Re-exports osspckgs DAL surface. |
| services/libs/data-access-layer/src/index.ts | Exposes osspckgs DAL from the package root. |
| services/apps/packages_worker/src/pom-fetcher/runPomEnrichmentLoop.ts | Implements batch/concurrent enrichment loop and persistence of extracted metadata. |
| services/apps/packages_worker/src/pom-fetcher/metadata.ts | Resolves latest version via maven-metadata.xml. |
| services/apps/packages_worker/src/pom-fetcher/extract.ts | Fetches POMs and extracts fields with limited parent inheritance traversal. |
| services/apps/packages_worker/src/config.ts | Adds pom-fetcher config loader. |
| services/apps/packages_worker/src/bin/pom-fetcher.ts | Adds runnable entrypoint with shutdown handling. |
| services/apps/packages_worker/package.json | Adds scripts and deps (axios, fast-xml-parser) for pom-fetcher. |
| scripts/services/pom-fetcher.yaml | Adds docker-compose service definition for pom-fetcher. |
| pnpm-lock.yaml | Updates lockfile for new deps (but includes an unexpected workspace importer). |
Files not reviewed (1)
- pnpm-lock.yaml: Language not supported
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+3
to
+10
| export interface IDbPackageUniverse { | ||
| id: number | ||
| purl: string | null | ||
| ecosystem: string | ||
| namespace: string | null | ||
| name: string | ||
| rankInEcosystem: number | null | ||
| } |
Comment on lines
+40
to
+44
| export type IDbPackageMaintainerUpsert = { | ||
| packageId: number | ||
| maintainerId: number | ||
| role: 'author' | 'maintainer' | null | ||
| } |
Comment on lines
+22
to
+35
| SELECT | ||
| pu.id, | ||
| pu.purl, | ||
| pu.namespace, | ||
| pu.name | ||
| FROM packages_universe pu | ||
| LEFT JOIN packages p ON p.purl = pu.purl | ||
| WHERE | ||
| pu.ecosystem = 'maven' | ||
| AND pu.namespace IS NOT NULL | ||
| AND ( | ||
| p.id IS NULL | ||
| OR p.last_synced_at < NOW() - ($(staleDays) || ' days')::interval | ||
| ) |
Comment on lines
+51
to
+52
| export async function upsertPackage(qx: QueryExecutor, item: IDbPackageUpsert): Promise<number> { | ||
| const row = await qx.selectOne( |
Comment on lines
+94
to
+95
| return row.id as number | ||
| } |
Comment on lines
+123
to
+127
| for (const person of allPeople) { | ||
| const username = person.username ?? person.email ?? person.displayName | ||
| if (!username) continue | ||
|
|
||
| const emailHash = person.email |
Comment on lines
+211
to
+215
| totalProcessed += result.processed | ||
| totalSkipped += result.skipped | ||
| totalErrors += result.errors | ||
| offset += config.batchSize | ||
|
|
Comment on lines
+86
to
+95
| } catch (err) { | ||
| if (axios.isAxiosError(err)) { | ||
| const status = err.response?.status | ||
| if (status === 404) { | ||
| log?.(`POM not found (404): ${url}`) | ||
| return null | ||
| } | ||
| log?.(`HTTP ${status ?? 'unknown'} fetching POM: ${url}`) | ||
| return null | ||
| } |
Comment on lines
+33
to
+39
| export function getPomFetcherConfig() { | ||
| return { | ||
| batchSize: parseInt(process.env.POM_FETCHER_BATCH_SIZE ?? '200', 10), | ||
| concurrency: parseInt(process.env.POM_FETCHER_CONCURRENCY ?? '10', 10), | ||
| staleDays: parseInt(process.env.POM_FETCHER_STALE_DAYS ?? '7', 10), | ||
| idleSleepSec: parseInt(process.env.POM_FETCHER_IDLE_SLEEP_SEC ?? '3600', 10), | ||
| } |
Comment on lines
+1375
to
+1379
| services/apps/pom_fetcher_worker: | ||
| dependencies: | ||
| '@crowd/archetype-standard': | ||
| specifier: workspace:* | ||
| version: link:../../archetypes/standard |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Changes
Type of change
JIRA ticket