How can successfully pre-verified changes cause regressions that should have been caught?

In a CI context one of the commonly-used measures of increasing the quality levels of the integration branch is a mandatory set of pre-commit quality verifications (typically including building some artifacts, performing unit tests and even some feature/integration tests).

Yet some regressions (build breakages, various test failures) are detected by the CI system verifications in exactly the areas which were supposed to be covered by these mandatory pre-commit verifications.

During analysis of these regressions an argument often heard is that the developer who committed the change identified as root-cause of the regression has successfully passed all such verifications. And often the claim is supported by hard evidence indicating that:

• after the final version of the change was reached it was ported to a fresh workspace based on the tip of the branch
• the required artifacts were built from scratch (so the build was totally fine, no cache-related issues, etc.)
• all mandatory tests passed, including those covering the area in question and should have detected the regression
• no intermittent false-positives affected the respective verifications
• no file merges were detected when committing the change to the branch
• none of the files being modified was touched by any other change committed in the branch since the fresh workspace was pulled

Is it really possible for a software change to cause such a regression despite correctly following all the prescribed processes and practices? How?

Answer

By pre-commit, you mean before entering the CI system (on the dev workstation) or SCM hooks checks ? I see a case on dev workstaion, but not on pre-commit hooks. I guess it depends on the exact SCM system. Yes if the pre-commit hook executions for 2 changes going into the same branch can overlap in time - as it's the case for git, for example. Sorry I was unclear, I mean does your pre-commit checks run on dev workstation or on the SCM side (central repository) ? All in all, I wonder if you're talking about checks a dev will launch himself before commit or if it is an automated testing system on 'receive' by the central server. Yes, that's where I'm heading. With the note that the regression can happen even with centralized checks - if they are allowed to overlap in time. Ok, rhetorical question then (I mean, one where you already have the answer and know what you're waiting). That doesn't leave much place for real answers at all. Well, yours would still be valuable - it covers a case I obviously didn't think about since I didn't clarified it in the question ;) Your case is also much more common, I'd suspect. I was a lead dev on the version control system developed internally at Oracle and wrote the feature that dealt with exactly those issues in automated fashion, as the pre-commits also often cannot run a full set of tests, as the nightly test suite. If you later post a question on resolving these issues, let me know. But this is exactly what causes one of the hard to solve problems of fully automated integration pipeline, which would accept, reject and rollback changes to keep the daily builds pristine. @JiriKlouda see http://devops.stackexchange.com/questions/268/is-there-a-ci-tool-that-guarantees-no-regressions-in-the-branch-quality-level