GitHub getChangeLogJSON regularly fails with HTTP 502 #42137
How are you running Renovate?Self-hosted Renovate CLI Which platform you running Renovate on?GitLab (.com or self-hosted) Which version of Renovate are you using?43.86.0 Please tell us more about your question or problemOn the renovate repository itself, I can show my config file, Gitlab details, etc. Whatever may help to troubleshoot. Thanks in advance, David Logs (if relevant)Logs |
Replies: 4 comments 1 reply
|
Hi all, any news. I'm still getting tons of these a day :( |
|
This is likely to be a general GitHub availability issue you're running into (https://www.githubstatus.com/) Are you caching anything between runs? Either using a persistent volume or Redis? |
|
We hit the same symptom on a self-hosted CI job, and I wanted to add some measurements — in our case the "general GitHub availability" explanation doesn't hold, because it reproduces deterministically. Our trigger is different from yoursOur dependency is
refs(first: $count, after: $cursor,
orderBy: {field: TAG_COMMIT_DATE, direction: DESC},
refPrefix: "refs/tags/")Measured against the public GraphQL API from a developer machine (indicative, not benchmark-grade):
Removing With a ~8.7s baseline against a ~10s budget there is almost no headroom, so GitHub-side load decides each run. We see it fail on some days and pass on others. Worth noting for anyone tracking this: #41498 reported "over 71,000 tags" for this repo in February 2026. It is 81,691 now, so this is drifting further over the limit rather than being a stable condition. However —
|
first: |
100 | 50 | 25 | 10 | 1 |
|---|---|---|---|---|---|
| time | 8.58s | 7.77s | 8.02s | 7.37s | 8.99s |
The cost is O(total refs) from the sort, not O(page size), so the mitigation retries three times and then gives up — adding latency without ever changing the outcome.
On removing the sort, or using REST
The obvious reaction is "then don't sort by commit date", so to save others the round trip: it cannot simply be removed globally. lib/util/github/graphql/readme.md explains that the newest-first ordering is what makes the incremental cache work — pages are fetched until the first already-cached item is reached. Drop the ordering and that early exit goes away.
But that reasoning applies to the datasource path, which needs releaseTimestamp per item. It does not apply to the path that triggers this thread's error. lib/workers/repository/update/pr/changelog/source.ts:
async getAllTags(endpoint: string, repository: string): Promise<string[]> {
const tags = (await getPkgReleases({ /* datasource: github-tags */ }))?.releases;
...
return tags.map(({ version }) => version); // <- everything but the name is discarded
}and the only consumer, findTagOfRelease, does pure string matching on those names — no timestamp is read anywhere in the changelog tag path.
So for changelog purposes Renovate pays for a commit-date sort over the entire ref set, plus per-tag commit resolution, and then throws the timestamps away.
Measured alternatives against aws/aws-sdk-go-v2, with the identical field selection in each case:
| approach | time | commit dates |
|---|---|---|
orderBy: {field: TAG_COMMIT_DATE} (current) |
9.07s | yes |
orderBy: {field: ALPHABETICAL} |
2.93s | yes |
REST /repos/{owner}/{repo}/tags?per_page=100 |
1.84s | no — but not needed here |
Switching only the sort field is a ~3x improvement with everything else unchanged. REST is faster still and its lack of commit dates does not matter for this call path, since they are discarded anyway.
For what it's worth, a REST-based tag lookup was already requested in #41498, where it was reported that git ls-remote --tags and the matching-refs REST API both return the full tag set of a ~58,000-tag monorepo in a couple of seconds. I could not find any existing GraphQL→REST fallback in the codebase to model that on, so I appreciate it is a larger change than the one below.
The exit code looks like the actual bug here
Whatever causes the 502, this is what follows. In lib/workers/repository/update/pr/changelog/index.ts:
} catch (err) /* istanbul ignore next */ {
logger.error({ config, err }, 'getChangeLogJSON error');
return null;
}return nullmeans "recoverable, keep going" — and it genuinely is. Our branches and PRs are created correctly and the run reachesRepository finished. The only user-visible loss is the Release Notes section of the PR body.logger.errormeans the run fails, becauselib/workers/global/index.tsdoes:
const loggerErrors = getProblems().filter((p) => p.level >= ERROR);
if (loggerErrors.length) { /* ... */ return 1; }So a run in which every dependency update succeeded still exits non-zero and fails the CI job. Those two decisions contradict each other, and the /* istanbul ignore next */ suggests the branch was never really considered a designed path.
There is an adjacent precedent that treats the analogous failure differently — in lib/modules/datasource/github-tags/index.ts, the supplementary queryReleases call is logged at debug:
} catch (err) /* istanbul ignore next */ {
logger.debug({ err }, `Error fetching additional info for GitHub tags`);Both are "supplementary metadata could not be fetched", but one is debug and the other fails the whole run.
Two suggestions, independently actionable
-
Log recoverable changelog failures below
error— at minimumExternalHostError— consistent with the neighbouringqueryReleaseshandling. This keeps them visible without failing a run that otherwise did its job, and it touches neither the pagination nor the caching design. It also helps this thread's case, where the 502 really may be transient. -
Don't request a commit-date sort for the changelog tag lookup, since the timestamps are discarded. Switching that call path to
ALPHABETICAL(or to REST/tags) would cut the cost substantially for large-tag repositories without affecting the datasource path that genuinely depends on the chronological ordering.
The first is the one I'd expect to be uncontroversial; the second is what would actually stop the 502s for repositories like aws/aws-sdk-go-v2.
Reproduction
The underlying slowness reproduces without Renovate:
# Renovate's query shape — slow
gh api graphql -f query='
{ repository(owner:"aws", name:"aws-sdk-go-v2") {
refs(first:100, orderBy:{field:TAG_COMMIT_DATE, direction:DESC}, refPrefix:"refs/tags/") {
nodes { version: name } } } }'
# Same query without orderBy — ~4x faster
gh api graphql -f query='
{ repository(owner:"aws", name:"aws-sdk-go-v2") {
refs(first:100, refPrefix:"refs/tags/") {
nodes { version: name } } } }'Workarounds we found, for anyone landing here
- The accepted answer in GitHub GraphQL 502 errors fetching changelogs for aws-sdk-go-v2 #41498 — disable changelog fetching for the offending package. This is what we are applying:
{ "matchPackageNames": ["github.com/aws/aws-sdk-go-v2{/,}**"], "fetchChangeLogs": "off" } fetchChangeLogs: "branch"was suggested in that thread as a lighter alternative that keeps release notes.- Redis caching does not appear to help — per a report in GitHub GraphQL 502 errors fetching changelogs for aws-sdk-go-v2 #41498, GraphQL tag results are not cached.
Environment
- Renovate
43.285.7, self-hosted, GitHub.com - The code quoted above is unchanged on
mainat the time of writing
|
We experience this with |
|
We're improving the changelog fetching with #45503 that hopefully should reduce issues like this 🤞🏼 |
We're improving the changelog fetching with #45503 that hopefully should reduce issues like this 🤞🏼