How can continuous integration scale for very large projects/teams?
Traditionally CI systems only monitor the quality of the code in an integration branch, signalling when regressions occur. Human intervention is required for repairs.
As the number of developers working on the same branch increases the risk of breakages/blockages increases. Eventually a point is reached where on average by the time a breakage is fixed a new one appears—progress on the branch becoming effectively negligible.
Splitting into multiple teams each working on a separate integration branch which would be merged at a later moment might help, but greatly extends the project duration, since it simply delays the necessary integration for a later moment while adding churn/noise/technical dept related to the partial integration from individual branch merges. Costs also increase due to CI setups for each of the branches, each with their own build/QA resources, etc. And overall quality is not necessarily better.
Scalability is, at best, dubious.
Is there a method for making such large projects actually scale?
Answer
This question is, to me, like quite a few other questions that make me think like "why is this even a question". Not that it is a bad question, but rather something that in the world I'm coming from (mainframes, I bet you know by now ...) is not really a question anymore. That's what over there we're doing for at least a decade or so already, and IMO it works like a charm. How otherwise would it be possible to keep a banking system going. I just don't know if it makes sense to post an answer looking at it with mainframe glasses. What do you think? The scale of the project here means the size and speed of the *changes* to the system being performed, not necessarily the size of the system itself (which would matter, for example, if the project would be re-writing the whole system or building it from scratch). And the context is a single branch, detaching on feature branches for days/weeks/months at a time is not continuous integration. Not saying that it doesn't happen at your place, only saying that the mainframe context itself doesn't imply large scale continuous integration. Think hundreds of commits per day into the same branch. This question leaves me with so many more questions - how did you end up in this situation? Why are so many developers working on the same branch? Why would feature branches need their own QA environment? Why are changes being pushed to the integration branch that break the build, and why isn't that the problem being addressed rather than CI scaling? Are they not testing locally? Not merging from upstream before they merge to it? @Adrian Branches are evil - they bring the Integration Hell with them - risky, slow, expensive. Especially for large teams. Branches can easily diverge too much and make merging impractical. Lookup the CI advantages. Branching means working in isolation - canceling all those advantages and bringing back the complementary disadvantages. Yes, branches have been used to successfully ship sw for decades - there was no alternative. But now we have CI. Yet many still use branches - they're used to them, not even feeling the pain anymore or choosing to ignore it. Or hit scalability issues. CI solves a completely different problem than branches, and used correctly, branches work with CI, not against it. Not using branches doesn't avoid integration hell either, it makes it worse: everyone working in the same branch has to merge every time they push. @Adrian Yes - the perfect merges: done by *exactly* the right person you want to be doing each and every one of them, with no delay and confirmed to be completed successfully or having problems at the very next CI cycle - just a regular change (even if technically it's a branch merge under the hood). This needs to be done anyways, going on a branch won't avoid it, it'll just delay it (while making it worse). **Note:** I should have specified - by branches I meant branches intended to be merged for integration. I'm OK with release branches (as long as they aren't merged with other branches). Is that in contrast to branch merges? If your branch merges are being done by the wrong person at the wrong time and not validated in the next CI cycle, that might be part of the problem. Claiming "branches are evil" implies some bad experiences caused by poor practices and blamed on branches, like a carpenter hitting his thumb and declaring hammers are evil. @Adrian I can't be swayed to ever re-consider integration branches and I know and accept that some can't be bothered with CI (as in trunk based development). Let's agree to disagree and leave it at that - this is not the right place for such debate.
