Update Guide to version 3.x for OpenDevStack users

Learn all about how to update your OpenDevStack based project to ODS 3.x.

It is assumed that you are updating from ODS 2.x.

New central ods namespace

One of the biggest changes in ODS 3 is that there is a new central namespace, named ods. This namespace hosts the complete ODS installation, such as a provisioning app, Nexus, SonarQube, and images for Jenkins (master and agent). This central namespace replaces the previous namespaces cd (which hosted Nexus, SonarQube, etc.) as well as prov-test, prov-dev and prov-cd (which hosted the provisioning app).

As a user, you will be affected by this change insofar as the images you use for the Jenkins master instance and Jenkins agent nodes will need to be retrieved from ods instead of cd (see next section). Further, depending on the migration done by the administrator, the URLs of Nexus, SonarQube and the provisioning app might have changed.

Update image tags of Jenkins deployments

Go into your OpenShift *-cd namespace and edit (via "Actions" > "Edit") the deployments (jenkins and webhook-proxy) to point to the new image tags (3.x), and trigger a deployment. Note that you also need to change the namespace of the images from cd to ods (see above). Since ODS 2, it is also recommended to remove any image triggers from jenkins and webhook-proxy, so that any image changes are only rolled out by yourself. If you choose to remove the image trigger, ensure that the imagePullPolicy is set to Always to always get the latest 3.x image when you deploy.

If the administrator of your ODS cluster chose to use a custom Bitbucket project for the 3.x installation, then you’ll need to set the environment variable ALLOWED_EXTERNAL_PROJECTS=<ods-bitbucket-project> for the webhook-proxy DeploymentConfig. Otherwise you will not be able to provision quickstarters.

Jenkins Shared Library changes

Renamed pipelines and stages

3.x ships with the three pipelines instead of just one. In 2.x, the shared library offered one pipeline to build components, named odsPipeline. This has been renamed to odsComponentPipeline to make its purpose clearer and to distinguish it from the other pipelines, odsQuickstarterPipeline (to provision quickstarters) and odsOrchestrationPipeline (to orchestrate multiple repositories using the release manager).

To provide backwards compatibility, odsPipeline is still available, but deprecated and will be removed in ODS 4.

In line with this change, the stages have been renamed too, which also makes them easier to distinguish from custom stages written by yourself in the Jenkinsfile. As with the pipeline, the old stages are still available but slated for removal in ODS 4. Here’s a list of the changes:

  • stageScanForSonarqube is now odsComponentStageScanWithSonar

  • stageScanForSnyk is now odsComponentStageScanWithSnyk

  • stageStartOpenshiftBuild is now odsComponentStageBuildOpenShiftImage

  • stageDeployToOpenshift is now odsComponentStageRolloutOpenShiftDeployment

  • stageUploadToNexus is now odsComponentStageUploadToNexus

Please note the new casing (e.g. OpenShift instead of Openshift). Further, the new stages all take the same two arguments: IContext context and Map config = [:]. Please see the documentation for details. In particular, some options which would previously have been set on odsPipeline can and should be set on the stage where they are used (however all options have been kept for backwards compatibility).

Reduced boilerplate

A Jenkinsfile in 2.x looks roughly like this:

def final projectId = 'foo'
def final componentId = 'bar'
def final credentialsId = "${projectId}-cd-cd-user-with-password"
def dockerRegistry
node {
  dockerRegistry = env.DOCKER_REGISTRY
}

@Library('ods-jenkins-shared-library@2.x') _

odsPipeline(
  image: "${dockerRegistry}/cd/jenkins-slave-golang:2.x",
  projectId: projectId,
  componentId: componentId,
  branchToEnvironmentMapping: ['master': 'dev']
) { context ->
  stageScanForSonarqube(context)
  stageStartOpenshiftBuild(context)
  stageDeployToOpenshift(context)
}

That is a lot of boilerplate. 3.x reduces this dramatically:

@Library('ods-jenkins-shared-library@3.x') _

odsComponentPipeline(
  imageStreamTag: 'ods/jenkins-agent-golang:3.x',
  branchToEnvironmentMapping: ['master': 'dev']
) { context ->
  odsComponentStageScanWithSonar(context)
  odsComponentStageBuildOpenShiftImage(context)
  odsComponentStageRolloutOpenShiftDeployment(context)
}

Note that the new Jenkinsfile example already uses the new pipeline/stage names as outlined above. Apart from this, it makes use of a new config option, imageStreamTag which removes the need to read the DOCKER_REGISTRY environment variable. If you need it, the image config is still present (e.g. to reference images outside of OpenShift). projectId and componentId are now automatically configured, but can still be set if desired.

Changed agent images

Up to and including 2.x, Jenkins used "slave images" to do its work. They have been renamed to "agent images" in 3.x. All new agent images are located in the new central namespace ods. Therefore, you need to update the references in your Jenkinsfile (as shown above):

  • cd/jenkins-slave-base is now ods/jenkins-agent-base

  • cd/jenkins-slave-golang is now ods/jenkins-agent-golang

  • cd/jenkins-slave-maven is now ods/jenkins-agent-maven

  • cd/jenkins-slave-nodejs10-angular is now ods/jenkins-agent-nodejs10-angular

  • cd/jenkins-slave-python is now ods/jenkins-agent-python

  • cd/jenkins-slave-scala is now ods/jenkins-agent-scala

Changed context properties

The context object available to use in stages has been slimmed down by removing properties unlikely to be useful to author stages (such as resource constraints of the Jenkins agent pod). Please review the documentation in case your authored stages make heavy use of the context object.

One notable change to the context object is that bitbucketHost did not include the scheme previously. Now, bitbucketHost is an alias for bitbucketUrl which does include the scheme. This change was made to align the property with nexusHost and nexusUrl, which both include the scheme as well. If you used context.bitbucketHost in one of your Jenkinsfile`s, you may now use `context.bitbucketHostWithoutScheme.
A notable addition to the context object is a new property issueId, which exposes the Jira issue ID (such as 123 from branch feature/FOO-123-bar-baz).

Different image tags

Previously, images produced by odsComponentStageBuildOpenShiftImage where tagged with context.tagversion, which consisted of the Jenkins build number and the (shortened) Git commit (e.g. 7-cd3e9082). This made it difficult for other processes (unaware of the Jenkins build number) to find those images.

The images are now tagged with just the (shortened) Git commit (e.g. cd3e9082). This change also has the huge benefit that it allows to promote images between environments (avoiding to rebuild them) using the new stage odsComponentStageImportOpenShiftImageOrElse.

Changes to Jenkins agent images

Some agent images (jenkins-agent-maven, jenkins-agent-nodejs10-angular, jenkins-agent-scala) exposed the environment variables NEXUS_HOST and NEXUS_USERNAME/NEXUS_PASSWORD/NEXUS_AUTH. This was not done intentionally, but as a side effect of using those variables during image build time. In 3.x, that has been cleaned up. Now those variables are passed as build arguments to the image build, which means they are no longer accessible as environment variables in the running container. Nexus credentials can and should be accessed from the pipeline context (context.nexusUrl, context.nexusUsername and context.nexusPassword). As a consequence, if a Jenkinsfile relies on the presence of those environment variables without setting them explicitly using withEnv, it will fail to run in 3.x. That said, the Jenkinsfile templates which shipped by default with the quickstarters did not rely on the environment variables - with the exception of fe-ionic which did not set NEXUS_HOST explicitly in the Build stage.

General upgrade to Python 3.8

Jenkins agent jenkins-agent-python as well as all python related quickstarters (be-python-flask, ds-jupyter-notebook and ds-ml-service) have been upgraded to python 3.8. This is not affecting default provisioned Jenkinsfile for ds-jupyter-notebook nor be-python-flask quickstarters that exist before ODS 3.x. But, ds-ml-service provisoned quickstarters before ODS 3.x need to slightly modify its Jenkinsfile before upgrading to ODS 3.x by just avoiding running any pip install <..dependencies..> --user (see new Jenkinsfile.template to get a suggested fix, basically using virutalenv; which is the recommended way of working when requiring building python packages).

Airflow Cluster Jenkins agent no longer supported

If you provisioned airflow-cluster quickstarter, please note that it has been moved out from the officially supported quickstarters (ods-quickstarters) repository and moved to extra-quickstarters. Therefore, the jenkins-slave-airflow:2.x can be shifted to jenkins-agent-python:3.x when upgrading to ODS 3.x.