How do you handle the huge number of possible permutations of Dockerfiles / Teraform files?
We have adopted Docker and Terraform as the foundation forour devOps. We have a basic version of it up and running today.
Dockerfiles live in each projects git repository and Terraform files are more centralized.
The number of projects we manage is increasing as do the number of environments, data seeding To make things more interesting we are also moving toward splitting services up into microservices where that makes sense.
So for each project, we have options like:
- What destination are you targetting (dev local, dev cloud, test (several variations), staging prod)
- What version / tag of the source code should it build from.
- What data should be seeded?
- Who should be notified?
- What infrastructure should it use? (We want to be able to allow "prod infrastructure) to be available for developers to spin up in dev cloud for debugging and testing) (along with several other combination)
There are a few other parameters we are considering and all in all the number of possible permutations become very high.
There is no way we can cleanly keep order of that many Dockerfiles.
So I am looking for a tool that helps us keep track of all the different configurations and makes it easy to find them.
Ideally, inside my head, I am picturing a wizard-based web app, where you enter in the parameters and it executes the necessary scripts automagically. This would also contain logs of what has been done, what environments are running and a few other things.
So far I havent had much luck. I know a lot of companies must have solved it and I must be thinking about it wrong, or there is a big opportunity here.
Is there such a tool? How do large companies solve this?
Answer
I don't get why you would need more dockerfiles than project, the project should build its container, release it as an artifact and then you deploy with needed services (each being an artifact also) in a specific environment, usually with a pipeline to promote from one environment to another. But if you want to keep a file based 'all in one' approach, I assume using templates is the way to go, how to write those templates is highly dependent on your needs and common ground between project and infrastructures Each project, can be deployed to any of lets say 5 different destinations where each destination can have different "hardware" setups, require different data seeding. So the main process is the same and you just need different variables for those environments ? Nice to see you're using containers to deploy your IaC/terraform code. I think it's more about how you organize that IaC that will help you locate it. For example using a combination of repos per AWS account and extensive use of terraform modules. Also, the way you organize your CI/CD pipelines will help visually too.The way we do it is documented here https://docs.cloudposse.com/reference-architectures/introduction/
