fix: add retry logic to CI VM status and log upload functions #1012
+146
−17
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cfsmp3
requested review from
canihavesomecoffee and
thealphadollar
as code owners
canihavesomecoffee
approved these changes
Jan 19, 2026
Previously, the CI VM scripts (both Linux and Windows) had no error handling for status POST requests. If the curl request failed due to network issues or server errors, the script would continue without retrying, potentially leaving tests stuck in "testing" state forever. This fix adds: - Retry logic with exponential backoff (3 attempts, 5s -> 10s -> 20s delay) - HTTP status code checking (success = 2xx response) - Curl exit code checking - Detailed logging of retry attempts and failures This addresses the issue where test 7935 completed all 237 tests but the completion status was never reported to the server, leaving the test stuck at 100% progress but "testing" state. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
canihavesomecoffee
force-pushed
the
fix/ci-script-retry-logic
branch
from
7fcc78c to
c2d5db0
Compare
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
postStatusandsendLogFilefunctions now check curl exit codes and HTTP response codessendLogFilefunction in Windows script (previously inline)Root Cause
Test 7935 completed all 237 tests but never reported completion to the server. The test remained stuck in "testing" state with 100% progress. Investigation revealed that the CI VM scripts had no error handling for status POST requests - if the curl request failed due to network issues or server timeout, the script would continue without retrying and then shut down, leaving the test stuck forever.
Changes
Linux (
install/ci-vm/ci-linux/ci/runCI):postStatus(): Now retries up to 3 times with exponential backoffsendLogFile(): Now retries up to 3 times with exponential backoffWindows (
install/ci-vm/ci-windows/ci/runCI.bat)::postStatus: Now retries up to 3 times with exponential backoff:sendLogFile: New function with retry logic (previously inline curl call):haltAndCatchFire: Now usessendLogFilefunctionTest plan
🤖 Generated with Claude Code