Skip to content

Separated timeouts + cleaner handling - #485

Open
vipbhardwaj wants to merge 16 commits into
mainfrom
CBG-5727
Open

Separated timeouts + cleaner handling#485
vipbhardwaj wants to merge 16 commits into
mainfrom
CBG-5727

Conversation

@vipbhardwaj

Copy link
Copy Markdown
Contributor

The stages in Jenkinsfile for infra-setup and test-run have been separated so that the confusion between how much time is a total test.sh is taking (which has both infra setup and then uv run) can be cleaner, simpler and granular.
This also helps now in allocating better timeouts for each of the two stages, hence the total time-taken for the test run will be now more precise.
COMPARED TO the previous ballpark measure of 2hours (inclusive of infra+run)

MORE IMPORTANTLY : This is also better in terms of helping out with a bug which might be related to increased INFRA time vs increased TEST-RUN time, since if the infra time is increased, Jenkins would timeout right there, it won't waste the time running the tests as well, leading to harder debugging later on.

@vipbhardwaj
vipbhardwaj requested review from torcolvin and a lite review from Copilot August 17, 2026 13:32

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

This PR improves the SGW QE Jenkins pipeline’s reliability around timeouts by separating infra setup from test execution and ensuring pytest can exit gracefully to preserve artifacts/log collection.

Changes:

  • Split Jenkins pipeline into separate “Infra Setup” and “Run Tests” stages with distinct timeouts.
  • Add pytest session and per-test timeouts, plus a SIGTERM→SIGINT pytest plugin to allow graceful teardown under Jenkins termination.
  • Enhance sgcollect behavior/documentation to collect on session timeout as well as test failures.

Reviewed changes

Copilot reviewed 5 out of 6 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
jenkins/pipelines/QE/sgw/test.sh Adds --skip-setup, guards setup execution, and introduces pytest timeouts for graceful shutdown.
jenkins/pipelines/QE/sgw/Jenkinsfile Splits infra setup vs test execution into nested stages with separate timeout budgets.
client/src/cbltest/plugins/sigterm_handler.py New pytest plugin to convert SIGTERM into SIGINT so pytest runs cleanup/finalizers.
client/src/cbltest/plugins/sgcollect_fixture.py Collect sgcollect artifacts on session timeout as well as on failures; updates help text.
client/pyproject.toml Adds pytest-timeout dependency and registers the new pytest plugin entrypoint.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread jenkins/pipelines/QE/sgw/test.sh Outdated
Comment thread jenkins/pipelines/QE/sgw/test.sh Outdated
Comment thread client/src/cbltest/plugins/sigterm_handler.py Outdated
Comment thread client/src/cbltest/plugins/sgcollect_fixture.py
@vipbhardwaj

Copy link
Copy Markdown
Contributor Author

From some reason this page is unable to load merge status and parent branch-PR for this PR, hence lets hold on this.
Since I plan on merging stacked PRs for this and the parent: #484

Comment thread client/src/cbltest/plugins/sigterm_handler.py Outdated
Comment thread jenkins/pipelines/QE/sgw/Jenkinsfile Outdated
Comment thread jenkins/pipelines/QE/sgw/test.sh
Co-authored-by: Tor Colvin <tor.colvin@couchbase.com>
@vipbhardwaj
vipbhardwaj changed the base branch from main to CBG-5712 August 18, 2026 05:24
@vipbhardwaj
vipbhardwaj requested a review from torcolvin August 18, 2026 10:33
Base automatically changed from CBG-5712 to main August 18, 2026 19:56
@vipbhardwaj
vipbhardwaj requested a review from borrrden August 19, 2026 06:21
@vipbhardwaj

Copy link
Copy Markdown
Contributor Author

I feel this PR advances to a cleaner approach for Jenkins testing, hence I request we keep it alive, why I think so :

  • Right now main has infra-setup, actual test-run, sg_collect running, sgcollect zips + junit + session.log collection, and then their accumulative mv artifacts all in the same section.
  • As we add more tests to increase coverage, the Run Test stage grows in time, which is expected behaviour and its okay
  • But there has been a recent surge of about 50min in TEST RUNTIME, as in, the vaguely cluttered Run Test stage in Jenkins
  • But because bullet-pt-1, its very difficult to tell WHY? since there could have been few tests added that take ~20mins individually, (since there are 2 such candidates I see right now)
  • But that could also be sg_collect running poorly or artifacts moving slowing down due to Jenkins Node slowness or network based issues since we've had VPN-weirdness recently?
  • I feel its the latter but that's just conjecture. And I do not want to move ahead on conjecture.
  • This led me to actually separate the STAGES so that : infra setup, tests running, artifacts being collected are all transparently seen and a bloat is individually spotted and hence fixed easily.
  • I raised this PR (and the connected ticket) solely for SyncGateway TDK pipeline, ie, /sgw/Jenkinsfile+test.sh, hence I am not able to align with Tor's suggestion of doing it for every other platform as well. Because the time-surge is seen in SyncGateway Jenkins-pipeline only
  • And sg_collect (3 times totalling zipped data of ~400Mb) runs on SyncGateway Pipeline ONLY as well, not on other platforms
  • Hence I feel it is okay to move-ahead with this direction, ofcourse I need to clean stuff up and also need to NOT RELATE this PR with the previous one, where Tor is right and hence I've paused on this right now.
  • But I feel we should go ahead with this for SGW pipeline (opt-in for other platform, not necessary)

Please take into account, the tech debt that is being created for SGW pipeline causing the current test-runtime-bloat difficult to be answerable for.
I say we go ahead with this but you have final say. cc @borrrden @torcolvin

@borrrden

Copy link
Copy Markdown
Member

I don't mind the separated steps but it's cleaner and easier to read if you separate the two steps into two scripts rather than having one script do both with differing flags.

@borrrden
borrrden removed their request for review August 28, 2026 21:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants