How can I unify my distributed build environments using Jenkins and Docker?

HOW TO -️ October 18, 2021

I have recently taken over the CI management for a medium-sized team of developers. Currently, we have one Jenkins server instance as well as six other instances of Jenkins agents to build upon. We have been using labels to determine which builds to run. For example, some instances are setup to run under environments A or B. In the future, we would like to add more instances of A and B, as well as add new environments, e.g., C, D.

My goal now is to unify these build environments and define them in one place to allow for better scale-ability and ease of management of these instances. Docker seems to be a solution to this approach. I know that I can create a Docker image for each of my individual environments. What I don't know is how to distribute the image into containers across my six machine instances. The solution I am looking for involves allowing multiple docker containers to be spun up on a single instance (docker host?), but only if needed (i.e., the scenario when all other hosts are currently hosting a container). Additionally, I need a solution that supports Windows and Linux based hosts.

I know that Docker support in Jenkins seems to be based largely via plugins, so I was hoping for some direction in which plugin to research first.


it sounds like you need to add several docker "clouds" (docker endpoints). Does the jenkins server have access to the Docker API on the agents? Something like what is described here