Compare
Fair question, and honestly you should. Pasting a workflow YAML into ChatGPT costs
nothing and takes ten seconds. So we did it for webpack/webpack, the same repo we'd
already measured cold, and put its advice next to ours line by line. Here's where the advice is
genuinely useful, and where it runs out.
We handed ChatGPT webpack's Github Actions workflow and asked how to make it
faster. It answered fast and it wasn't wrong: cache dependencies, shard the test job, parallelize
where the graph allows, drop to a bigger runner. Those are the right hypotheses to start from,
and it produced them in seconds. The catch is what it was working from. It never saw a single
real run. It can't fetch webpack's Actions history, so it never knew that the median wall-clock
over the last 30 days sat at 45m57s across our ten interleaved live runs
sampled 2026-07-21, or which jobs actually sat on the critical path. It reasoned about the YAML,
not about the repo.
Where the two answers diverge
| ChatGPT | greenmerge | |
|---|---|---|
| Reads your workflow YAML and suggests ideas | Yes, in seconds | Yes |
| Works from your repo's real run history | No: can't fetch it | Yes: modeled from live runs |
| Proposes a specific structural fix | Generic, untested | Yes: a composed, named diff |
| Runs the fix and measures the real result | No | Yes: interleaved A/B on your CI |
| Will guarantee a number | No: any % it gives is invented | Yes: the measured floor |
| Costs nothing to ask | Yes | $49 scan, credited to a sprint |
The first and last rows are real points in ChatGPT's favor: it's instant and it's free. That's exactly why it's a good place to start. The middle rows are where a starting point stops being an answer.
Ask ChatGPT how much its advice will save and it will tell you. When we asked, it offered "you could cut build time by 30 to 50%." That number wasn't measured against anything. It never ran the change, it never saw a baseline, and it can't tell you whether test sharding even touches your critical path or just speeds up a job that was already off it. The estimate reads with total confidence and rests on nothing, which is the one failure mode that costs you a sprint: you refactor the pipeline, the median doesn't move, and you can't say why.
Our engine originally modeled a +45.6% win for webpack from two composed changes. We ran it for real: ten interleaved runs on GitHub-hosted CI, the substrate a client actually uses. It came back +14.1%, statistically significant but a third of the model's number. Read the full case study → If our own modeled estimate can miss by 3x until we measure it, a chat window that never ran your CI at all isn't going to land closer. The difference isn't that we guess better. It's that we don't ship the number until the runs come back.
Keep using ChatGPT for the first pass. It's a great way to generate hypotheses about where your CI might be slow. When you want one of those hypotheses turned into a measured, guaranteed number on your real repo, bring it to us. We'll tell you which ones are worth the sprint.
Book an engagement