Performance was the major CI problem, ranking higher than other problems such as tools, onboarding process, or code review delays, mentioned in my Developer Experience survey. At that moment, almost a year ago, I realized it was no longer a background concern, but a real productivity problem. Our Merge Train pipeline took an hour to complete, and almost 1 in 5 tries were unsuccessful. The delay cost was not noticeable in any individual run, but really enormous in aggregate.
Over the couple of months, we managed to reduce the Merge Train duration from roughly one hour to 22 minutes (a 64% reduction) while also growing our test suite by 90%, improving CI health from around 82% to above 90%, and reducing the fail rate from 18% to the point between 6% and 10%. During this change our monthly infrastructure costs increased only by roughly 10%.
This article is my walkthrough of what changed, what didn't work, and which decisions ended up having the biggest impact.
Methodology
All metrics that I mention in this document have been collected using CI dashboards and internal reports during the mid-2025 – early-2026 timeframe. In all experiments aimed at comparing infrastructure configurations, we ran 10 sequential pipelines with no changes made otherwise, and measured total runtime from the start of the pipeline to its finish.
The starting point for CI infrastructure optimization looked like this:
And this is our CI foundation of the organization of about 400 engineers (~200 developers), processing about 45 MRs and ~340 pipelines per day. The scale of the tasks required that even a small gain in a minute could be considered valuable and important rather than the time spent on it.
Prioritizing the work
Rather than optimizing starting with the most time-consuming job, we first decided to compile a list of all the possible optimizations and then evaluate each of them based on their effect and risk factor:
And then we learned a valuable lesson - order is important. If you change too many things simultaneously, it is difficult to track the actual impact of any change.
Compute instance migration
This one change actually produced the biggest result among all the things we’ve done on this project. We shifted from our C5 fleet to newer generation hardware, and a reduction in time could be seen immediately.
We decided to test 3 configurations with 10 isolated runs each and take into account both average runtime and variance.
At first AMD looked like the obvious configuration, but while being cheaper and working faster, it was unacceptable because some runs on it took just under 22 minutes, and others took more than 30 — the variance was just too high. In a Merge Train infrastructure variance is more important than mean time, so we chose Intel.
Service tests help to validate the behavior of the API as well as the underlying business logic. Service tests are different from end-to-end tests: they do not require a browser. For some reason old infrastructure treated these 2 distinct workloads as the same, wasting resources every time.
All the 30 test nodes would load up Chrome containers, not accounting for whether they were running browser-based tests. It’s not logical, we were wasting memory and startup time to power jobs we never used.
After division into 2 different pools our new structure looked like this:
- 25 nodes for feature tests with Chrome containers
- 5 nodes for services tests without the need for a browser
It was a real breakthrough, which immediately saved us around 2 GB of memory, and helped segregate roles and responsibilities.
Small changes that added up
As the big work was being done, there were three easy optimizations running in parallel.
CloudWatch API collection time was roughly 120 seconds per one pipeline. The reason for it were synchronous calls to the deprecated `get_metric_statistics` API, which were working only with one metric at a time, while also collecting entire metric history each time. Just by switching it to `get_metric_data` API which retrieves many metrics simultaneously, we managed to cut down the time of metric gathering to 20 seconds per pipeline. Only 2 days of coding and nothing to lose here.
The issue of Prettier, ESLint, and RuboCop rebuilding cache on every run was resolved by caching linters results, where time saving was around 2 minutes per pipeline. We did this within one day.
Skip-CI for changes to documentation allowed pipeline skips to occur for changelog entries, README files, and documentation contained in the docs/ directory. Up to 10 Merge Requests per day can now skip the entire pipeline.
None of these optimizations is particularly impressive on its own, sure, but together they demonstrate that small savings, which are low-risk and involve no coordination, are worth taking in parallel with larger changes.
Stability of pipelines
Reducing build time uncovered an actual problem which had been there all along but was difficult to spot as long as running pipelines took one hour: one out of five runs of the Merge Train failed. The thing about failures was that it was not the fault of buggy code but that of flaky tests.
A flaky test in a Merge Train means a failed pipeline will hold up not only its author but all the pipelines behind it.
Developers came to the realization that they could just run such faulty pipelines again, rather than try to debug them; that was very much logical. This, however, led to a lack of trust in CI as an indicator of software quality.
To fix this we decided to start automatically detecting flaky tests. This is how we did it:
- Every one of the failed tests was logged together with its associated merge request. If one and the same test fails several times independently in a short span of time among different merge requests, it was marked as suspicious by the algorithm.
- A second layer was introduced based on LLMs' analysis. It would look at the test log and compare it with git diff associated with each test to check whether the test failure is explainable by the code modification. Now we could identify some flaky tests on their first appearance instead of waiting for them to fail repeatedly.
- We put flagged tests into quarantine, and made the runner check the compiled list of flaky tests and exclude those suspect tests from testing each time before launching new tasks. Every case of placing a test in quarantine was documented and posted into the team's channel along with evidence. The cost of the entire algorithm layer is about $100 per month, which is negligible comparing to the engineer's time wasted because of Merge Train crashes.
However, the system was not flawless as it misclassified some tests during the initial months. Despite that, the deal turned out to be very beneficial. The price of an error in such cases was much cheaper than the repeated reboots of Merge Trains caused by the identified instabilities of the tests.
Fixing more than 30 flaky tests made the success rate of Merge Trains increase from 82% to 90-94%, preventing almost 150 unnecessary retries per month.
The test suite increased by 90%, but pipeline time has risen by only 11%. The optimization effort consumed the biggest part of that growth. The Merge Train pipeline itself, on its own, became 64% faster. This is precisely what we had hoped to achieve: the possibility of adding tests without having to pay for them in terms of waiting time.
This added about $1,680 a month to our infrastructure bill, but I say it was worth it because we no longer wasted 1,300+ engineer hours on such things as waiting, retries, and Merge Train delays. Assuming an hourly rate for one engineer would be $50/hour, that's more than $65,000 worth of engineering productivity being regained each month when applying a utilization factor.
What we'd tell other teams
The improvement of 64% in Merge Train Duration did not result from some magical insight. It was hard work and calculated steps. Some lessons that our team learned from this process were especially valuable.
Intuition cannot beat data. It was a mistake to go for Intel against AMD when we were considering average running time alone. Adding the variance to this formula made us move in a totally different direction from where we started.
Order of operations does count. It is hard to know which of your optimizations help when they all are applied at the same moment.
Speed and stability are 2 sides of the same coin. A 20-minute pipeline that you can’t rely on 24/7 is not as good in reality as a stable and trusted but slow pipeline.
When we were finally successful, the size of the test case had doubled while the amount of time it took to execute the pipeline had gone from an hour to just 22 minutes.
