Jenkins Shared Library

This library allows to have a minimal Jenkinsfile in each repository by providing all language-agnostic build aspects. The goal is to duplicate as little as possible between repositories and have an easy way to ship updates to all projects.

Usage

Load the shared library in your Jenkinsfile like this:

def final projectId = "hugo"
def final componentId = "be-node-express"
def final credentialsId = "${projectId}-cd-cd-user-with-password"
def sharedLibraryRepository
def dockerRegistry
node {
  sharedLibraryRepository = env.SHARED_LIBRARY_REPOSITORY
  dockerRegistry = env.DOCKER_REGISTRY
}

library identifier: 'ods-library@production', retriever: modernSCM(
  [$class: 'GitSCMSource',
   remote: sharedLibraryRepository,
   credentialsId: credentialsId])

odsPipeline(
  image: "${dockerRegistry}/cd/jenkins-slave-maven:latest",
  projectId: projectId,
  componentId: componentId,
  branchToEnvironmentMapping: [
    'master': 'test',
    '*': 'dev'
  ]
) { context ->
  stage('Build') {
      // custom stage
  }
  stageScanForSonarqube(context) // using a provided stage
}

Provided Stages

Following stages are provided (see folder vars for more details):

Workflow

The shared library does not impose which Git workflow you use. Whether you use git-flow, GitHub flow or a custom workflow, it is possible to configure the shared library according to your needs. There are just two settings to control everything: branchToEnvironmentMapping and autoCloneEnvironmentsFromSourceMapping.

branchToEnvironmentMapping

Example:

branchToEnvironmentMapping: [
  "master": "prod",
  "develop": "dev",
  "hotfix/": "hotfix",
  "*": "review"
]

Maps a branch to an environment. There are three ways to reference branches:

  • Fixed name (e.g. master)

  • Prefix (ending with a slash, e.g. hotfix/)

  • Any branch (*)

Matches are made top-to-bottom. For prefixes / any branch, a more specific environment might be selected if:

  • the branch contains a ticket ID and a corresponding env exists in OCP. E.g. for mapping "feature/": "dev" and branch feature/foo-123-bar, the env dev-123 is selected instead of dev if it exists.

  • the branch name corresponds to an existing env in OCP. E.g. for mapping "release/": "rel" and branch release/1.0.0, the env rel-1.0.0 is selected instead of rel if it exists.

autoCloneEnvironmentsFromSourceMapping

Caution! Cloning environments on-the-fly is an advanced feature and should only be used if you understand OCP well, as there are many moving parts and things can go wrong in multiple places.

Example:

autoCloneEnvironmentsFromSourceMapping: [
  "hotfix": "prod",
  "review": "dev"
]

Instead of deploying multiple branches to the same environment, individual environments can be created on-the-fly. For example, the mapping "*": "review" deploys all branches to the review environment. To have one environment per branch / ticket ID, you can add the review environment to autoCloneEnvironmentsFromSourceMapping, e.g. like this: "review": "dev". This will create individual environments (named e.g. review-123 or review-foobar), each cloned from the dev environment.

Examples

If you use git-flow, the following config fits well:

branchToEnvironmentMapping: [
  'master': 'prod',
  'develop': 'dev',
  'release/': 'rel',
  'hotfix/': 'hotfix',
  '*': 'preview'
]
// Optionally, configure environments on-the-fly:
autoCloneEnvironmentsFromSourceMapping: [
  'rel': 'dev',
  'hotfix': 'prod',
  'preview': 'dev'
]

If you use GitHub Flow, the following config fits well:

branchToEnvironmentMapping: [
  'master': 'prod',
  '*': 'preview'
]
// Optionally, configure environments on-the-fly:
autoCloneEnvironmentsFromSourceMapping: [
  'preview': 'prod'
]

If you use a custom workflow, the config could look like this:

branchToEnvironmentMapping: [
  'production': 'prod',
  'master': 'dev',
  'staging': 'uat'
]
// Optionally, configure environments on-the-fly:
autoCloneEnvironmentsFromSourceMapping: [
  'uat': 'prod'
]

Writing stages

Inside the closure passed to odsPipeline, you have full control. Write stages just like you would do in a normal Jenkinsfile. You have access to the context, which is assembled for you on the master node. The context can be influenced by changing the config map passed to odsPipeline. Please see vars/odsPipeline.groovy for possible options.

When you write stages, you have access to both global variables (defined without def in the Jenkinsfile) and the context object. It contains the following properties:

Property Description

jobName

Value of JOB_NAME. It is the name of the project of the build.

buildNumber

Value of BUILD_NUMBER. The current build number, such as 153.

buildUrl

Value of BUILD_URL. The URL where the results of the build can be found (e.g. http://buildserver/jenkins/job/MyJobName/123/)

buildTime

Time of the build, collected when the odsPipeline starts.

image

Container image to use for the Jenkins agent container. This value is not used when podContainers is set.

alwaysPullImage

Determine whether to always pull the container image before each build run. Defaults to true. This value is not used when podContainers is set.

resourceRequestMemory

How much memory the container requests - defaults to 1Gi. This value is not used when podContainers is set.

resourceLimitMemory

Maximum memory the container can use - defaults to 2Gi. This value is not used when podContainers is set.

resourceRequestCpu

How much CPU the container requests - defaults to 10mi. This value is not used when podContainers is set.

resourceLimitCpu

Maximum CPU the container can use - defaults to 300mi. This value is not used when podContainers is set.

podLabel

Pod label, set by default to a random label to avoid caching issues. Set to a stable label if you want to reuse pods across builds.

podContainers

Custom pod containers to use if the default, automatically configured container is not suitable for your use case (e.g. if you need multiple containers such as app and database). Please see https://github.com/jenkinsci/kubernetes-plugin#pod-and-container-template-configuration for more information.

podVolumes

Volumes to make available to the pod.

podServiceAccount

Serviceaccount to use when running the pod.

credentialsId

Credentials identifier (Credentials are created and named automatically by the OpenShift Jenkins plugin).

tagversion

The tagversion is made up of the build number and the first 8 chars of the commit SHA.

notifyNotGreen

Whether to send notifications if the build is not successful.

nexusHost

Nexus host (with scheme).

nexusUsername

Nexus username.

nexusPassword

Nexus password.

nexusHostWithBasicAuth

Nexus host (with scheme), including username and password as BasicAuth.

branchToEnvironmentMapping

Define which branches are deployed to which environments.

autoCloneEnvironmentsFromSourceMapping

Define which environments are cloned from which source environments.

cloneSourceEnv

The environment which was chosen as the clone source.

environment

The environment which was chosen as the deployment target, e.g. dev.

targetProject

Target project, based on the environment. E.g. foo-dev.

cloneProjectScriptBranch

Branch on ods-core used for the clone-project.sh and dependent scripts. Default is production.

groupId

Group ID, defaults to: org.opendevstack..

projectId

Project ID, e.g. foo.

componentId

Component ID, e.g. be-auth-service.

gitUrl

Git URL of repository

gitBranch

Git branch for which the build runs.

gitCommit

Git commit SHA to build.

gitCommitAuthor

Git commit author.

gitCommitMessage

Git commit message.

gitCommitTime

Git commit time in RFC 3399.

sonarQubeBranch

Branch on which to run SonarQube analysis.

failOnSnykScanVulnerabilities

Boolean flag (default true) that disables build failure in case Snyk Scan finds vulnerabilities

dependencyCheckBranch

Branch on which to run dependency checks.

environmentLimit

Number of environments to allow.

openshiftHost

OpenShift host - value taken from OPENSHIFT_API_URL.

odsSharedLibVersion

ODS Jenkins shared library version, taken from reference in Jenkinsfile.

bitbucketUrl

BitBucket URL - value taken from BITBUCKET_URL. If BITBUCKET_URL is not present, it will default to https://<BITBUCKET_HOST>;.

bitbucketHost

BitBucket host - value derived from bitbucketUrl.

environmentCreated

Whether an environment has been created during the build.

openshiftBuildTimeout

Timeout for the OpenShift build of the container image.

ciSkip

Whether the build should be skipped, based on the Git commit message.

dockerDir

The docker directory to use when building the image in openshift. Defaults to docker.

Slave customization

The slave used in the pipeline can be customized by adjusting the image to use. Further, alwaysPullImage (defaulting to true) can be used to determine whether this image should be refreshed on each build.

Resource constraints of the container can be changed via resourceRequestCpu, resourceLimitCpu, resourceRequestMemory and resourceLimitMemory.

The setting podVolumes allows to mount persistent volume claims to the pod (the value is passed to the podTemplate call as volumes).

To completely control the container(s) within the pod, set podContainers (which is passed to the podTemplate call as containers).

Configuring of a customized slave container in a Jenkinsfile:

odsPipeline(
  projectId: projectId,
  podContainers: [
    containerTemplate(
      name: 'jnlp', // do not change, see https://github.com/jenkinsci/kubernetes-plugin#constraints
      image: "${dockerRegistry}/foo-cd/jenkins-slave-custom",
      workingDir: '/tmp',
      resourceRequestCpu: '100m',
      resourceLimitCpu: '500m',
      resourceRequestMemory: '2Gi',
      resourceLimitMemory: '4Gi',
      alwaysPullImage: true,
      args: '${computer.jnlpmac} ${computer.name}'
    )
  ],
  ...
  ) { context ->
  stageBuild(context)
  ...
}

See the kubernetes-plugin documentation for possible configuration.

Versioning

Each Jenkinsfile references a Git revsison of this library, e.g. library identifier: 'ods-library@production'. The Git revsison can be a branch (e.g. production or 0.1.x), a tag (e.g.0.1.1) or a specific commit.

Git LFS (Git Large File Storage extension)

If you are working with large files (e.g.: binary files, media files, files bigger than 5MB…​), you can follow the following steps:

  • Check this HOWTO about Git LFS

  • Track your large files in your local clone, as explained in previous step

  • Enable Git LFS in your repository (if BitBucket: under repository’s settings main page you can enable it)

NOTE: if already having a repository with large files and you want to migrate it to using git LFS:

git lfs migrate

How to add Snyk scanning to your ODS project

  1. Setup organisation in snyk.io

    1. If you donĀ“t have an snyk account just create one at snyk.io

    2. Once you logged into snyk.io, in your snyk group create an organisation for your project with exactly same name as project name.

    3. Create a service account in settings for the created organisation and keep the displayed token. You will need it later.

  2. Add environment variable to jenkins in your cd project

    1. Add the environment variable SNYK_AUTHENTICATION_CODE in jenkins in your openshift cd project with service account token as value.

  3. Edit your project Jenkinsfile

    1. Read auth code from environment by adding:

       node {
           ...
         snykAuthenticationCode = env.SNYK_AUTHENTICATION_CODE
      }
    2. Add stageScanForSnyk:

       ) { context ->
          ...
      
         stageScanForSnyk(context, snykAuthenticationCode, 'build.gradle', context.projectId)
         ...
       }

Stages

Ods Pipeline

The odsPipeline method offers a complete pipeline for any component created with a quickstarter. It takes care of building the code, uploading artifacts to Nexus, analysing the code and starting builds and triggering deployments in Openshift.

Stage Scan for Snyk

The "Snyk Security Scan" stage does 2 tasks:

  1. uploads the list of project 3rd party dependencies including its licenses for monitoring. Snyk monitoring feature notifies developers about new vulnerabilities per email once this vulnerabilities are reported to the Snyk Vulnerability Database

  2. analyses your project 3rd party dependencies including its licenses and break the build if vulnerable versions are found in the project. Build fail can be disable with the property failOnSnykScanVulnerabilities Note: that if this stage only runs if the SNYK_AUTHENTICATION_CODE is found as environment variable. This variable needs to be defined as environment variable in the deployment configuration of your project jenkins.

Stage Start OpenShift Build

stageStartOpenshiftBuild triggers the BuildConfig related to the repository being built.

stageStartOpenshiftBuild takes two optional params (a) the first one, named "buildArgs", which is a map allowing to customise the image build step in OpenShift. For example: stageStartOpenshiftBuild(context, ["myArg":"val"])

(b) the second one, named "imageLabels", which is a map allowing to customise the image label generation. For example: stageStartOpenshiftBuild(context, [ : ], ["myImageLabel":"valLabel"]). This will end up as label prefixed with 'ext.'

Stage Scan For SonarQube

The "SonarQube Analysis" stage scans your source code and reports findings to SonarQube. The configuration of the scan happens via the "sonar-project.properties" file in the repository being built.

In debug mode, the sonar-scanner binary is started with the "-X" flag.

If no "sonar.projectVersion" is specified in "sonar-project.properties", it is set to the shortened Git SHA.

Development

  • Try to write tests.

  • See if you can split things up into classes.

  • Keep in mind that you need to access e.g. sh via script.sh.

Background

The implementation is largely based on https://www.relaxdiego.com/2018/02/jenkins-on-jenkins-shared-libraries.html. The scripted pipeline syntax was chosen because it is a better fit for a shared library. The declarative pipeline syntax is targeted for newcomers and/or simple pipelines (see https://jenkins.io/doc/book/pipeline/syntax/#scripted-pipeline). If you try to use it e.g. within a Groovy class you’ll end up with lots of script blocks.