Jenkins

The controller is the brain: it serves the web UI, stores configuration, credentials and build history, schedules builds and runs plugins. Agents are workers that connect to the controller and actually execute build steps. An agent has a number of executors, each able to run one build. Builds pick agents by label, so a job that needs Docker asks for docker rather than a hostname. The controller should not run builds itself, because build scripts are code from repositories and could read its secrets or damage its configuration. Modern setups use ephemeral agents, such as a Kubernetes pod or cloud VM per build, which are clean every time and scale with demand.

Declarative pipeline has a fixed structure: pipeline, agent, stages, steps, post, when, options. Jenkins can validate it before running, the Blue Ocean style views understand it, and it is easier for the whole team to read. Scripted pipeline is plain Groovy inside node blocks: maximum flexibility, but easy to turn into an unreadable program with subtle CPS serialisation issues. My default is Declarative for every Jenkinsfile, with a script block only for small unavoidable logic. When real logic is needed, it belongs in a Shared Library with tests, called from a short Declarative Jenkinsfile, rather than inline Groovy in each repository.

Store them in the Jenkins credentials store, scoped to the folder of the team that needs them, or fetch them from an external manager like Vault through a plugin. In pipelines, bind them only where needed with withCredentials or credentials() in an environment block, which also masks them in logs. Use single-quoted sh strings so the shell, not Groovy, expands the variable; Jenkins even warns about insecure interpolation. Never echo secrets or pass them as command arguments where they show in process lists. Restrict who can configure jobs, because anyone who can edit a pipeline can print a credential it has access to, and audit credentials usage periodically.

Treat it like any production service. Run LTS and upgrade on a schedule, rehearsing on a staging controller restored from a recent backup. Keep plugins few and current, removing unused ones. Describe the controller with Configuration as Code in Git, so it can be rebuilt; jobs come from Jenkinsfiles via Multibranch or organization folders, not the UI. Back up jenkins_home, or at least the parts not reproducible from Git. Use ephemeral agents from images built in CI, so agent tooling is versioned too. Monitor disk space, queue length and executor usage, and discard old builds and artifacts with logRotator so the disk does not fill silently.

A Multibranch Pipeline is a job type that points at a repository and automatically creates a sub-job for every branch, and optionally every pull request, that contains a Jenkinsfile. When a branch is created, Jenkins discovers it through a webhook or periodic scan and starts building; when the branch is deleted, the sub-job is removed. Combined with when { branch 'main' } or changeRequest() conditions, one Jenkinsfile can behave differently for feature branches, pull requests and main. An Organization Folder goes one level higher and scans a whole GitHub organization or GitLab group, creating Multibranch jobs for every repository with a Jenkinsfile.

Rarely by default, but there are honest cases. Air-gapped or highly regulated environments where no SaaS control plane is allowed and the organization already runs Jenkins well. Complex hardware workflows, such as embedded or device testing, that depend on specific plugins or agent setups. Very large build farms where an experienced team operates Jenkins with Kubernetes agents, JCasC and Shared Libraries at a lower cost than hosted minutes. Against it: operational load, plugin security risk, Groovy-specific skills and a weaker developer experience than built-in CI. If the code is on GitLab or GitHub Enterprise Server, their CI with self-hosted runners usually gives the same on-premises control with less to maintain, so I would choose Jenkins only with a clear reason and a team to own it.

One giant controller becomes a single point of failure and a performance bottleneck, because Groovy pipeline logic, the UI and plugins all run there. The common pattern is several controllers, split by business unit or trust level, each managed identically with Configuration as Code and a curated plugin image, possibly with CloudBees CI or similar tooling to manage them centrally. Agents are ephemeral pods or VMs that autoscale. Shared Libraries provide standard pipeline steps, versioned and tested. Teams get folders with scoped credentials and permissions. Monitoring covers queue time, executor usage, controller heap and disk. The trade-off is fleet management overhead versus blast radius: many controllers mean more upgrades to orchestrate, but a bad plugin or runaway job affects only one part of the organization.

Several controls must combine. In Multibranch configuration, set trust so that pull requests from forks, or from contributors without write access, build with the Jenkinsfile from the target branch, not their modified one. Scope credentials to folders and give pull request builds none of the production credentials at all; deploy credentials are only bound in stages that run on main. Masking helps but is easily bypassed by encoding, so it is not a control. Keep the Groovy sandbox enabled and do not approve risky script signatures. Require code review for Jenkinsfile and Shared Library changes with branch protection. For the highest-value secrets, use short-lived credentials from Vault so even a leak expires quickly. The underlying principle: anyone who can change pipeline code can use any secret that pipeline can reach.