Update Guide to version 3.x for OpenDevStack users
Learn all about how to update your OpenDevStack based project to ODS 3.x.
It is assumed that you are updating from ODS 2.x. |
New central ods
namespace
One of the biggest changes in ODS 3 is that there is a new central namespace,
named ods
. This namespace hosts the complete ODS installation, such as a
provisioning app, Nexus, SonarQube, and images for Jenkins (master and agent).
This central namespace replaces the previous namespaces cd
(which hosted Nexus,
SonarQube, etc.) as well as prov-test
, prov-dev
and prov-cd
(which hosted
the provisioning app).
As a user, you will be affected by this change insofar as the images you use for
the Jenkins master instance and Jenkins agent nodes will need to be retrieved
from ods
instead of cd
(see next section). Further, depending on the
migration done by the administrator, the URLs of Nexus, SonarQube and the
provisioning app might have changed.
Update image tags of Jenkins deployments
Go into your OpenShift *-cd
namespace and edit (via "Actions" > "Edit") the
deployments (jenkins
and webhook-proxy
) to point to the new image tags
(3.x
), and trigger a deployment. Note that you also need to change the
namespace of the images from cd
to ods
(see above). Since ODS 2, it is also
recommended to remove any image triggers from jenkins
and webhook-proxy
, so
that any image changes are only rolled out by yourself. If you choose to remove
the image trigger, ensure that the imagePullPolicy
is set to Always
to
always get the latest 3.x
image when you deploy.
If the administrator of your ODS cluster chose to use a custom Bitbucket
project for the 3.x installation, then you’ll need to set the environment variable
ALLOWED_EXTERNAL_PROJECTS=<ods-bitbucket-project> for the webhook-proxy
DeploymentConfig. Otherwise you will not be able to provision quickstarters.
|
Jenkins Shared Library changes
Renamed pipelines and stages
3.x ships with the three pipelines instead of just one. In 2.x, the shared
library offered one pipeline to build components, named odsPipeline
. This has
been renamed to odsComponentPipeline
to make its purpose clearer and to
distinguish it from the other pipelines, odsQuickstarterPipeline
(to
provision quickstarters) and odsOrchestrationPipeline
(to orchestrate
multiple repositories using the release manager).
To provide backwards compatibility, odsPipeline
is still available, but
deprecated and will be removed in ODS 4.
In line with this change, the stages have been renamed too, which also makes
them easier to distinguish from custom stages written by yourself in the
Jenkinsfile
. As with the pipeline, the old stages are still available but
slated for removal in ODS 4. Here’s a list of the changes:
-
stageScanForSonarqube
is nowodsComponentStageScanWithSonar
-
stageScanForSnyk
is nowodsComponentStageScanWithSnyk
-
stageStartOpenshiftBuild
is nowodsComponentStageBuildOpenShiftImage
-
stageDeployToOpenshift
is nowodsComponentStageRolloutOpenShiftDeployment
-
stageUploadToNexus
is nowodsComponentStageUploadToNexus
Please note the new casing (e.g. OpenShift
instead of Openshift
). Further,
the new stages all take the same two arguments: IContext context
and
Map config = [:]
. Please see the
documentation for details.
In particular, some options which would previously have been set on
odsPipeline
can and should be set on the stage where they are used (however
all options have been kept for backwards compatibility).
Reduced boilerplate
A Jenkinsfile
in 2.x looks roughly like this:
def final projectId = 'foo'
def final componentId = 'bar'
def final credentialsId = "${projectId}-cd-cd-user-with-password"
def dockerRegistry
node {
dockerRegistry = env.DOCKER_REGISTRY
}
@Library('ods-jenkins-shared-library@2.x') _
odsPipeline(
image: "${dockerRegistry}/cd/jenkins-slave-golang:2.x",
projectId: projectId,
componentId: componentId,
branchToEnvironmentMapping: ['master': 'dev']
) { context ->
stageScanForSonarqube(context)
stageStartOpenshiftBuild(context)
stageDeployToOpenshift(context)
}
That is a lot of boilerplate. 3.x reduces this dramatically:
@Library('ods-jenkins-shared-library@3.x') _
odsComponentPipeline(
imageStreamTag: 'ods/jenkins-agent-golang:3.x',
branchToEnvironmentMapping: ['master': 'dev']
) { context ->
odsComponentStageScanWithSonar(context)
odsComponentStageBuildOpenShiftImage(context)
odsComponentStageRolloutOpenShiftDeployment(context)
}
Note that the new Jenkinsfile
example already uses the new pipeline/stage
names as outlined above. Apart from this, it makes use of a new config option,
imageStreamTag
which removes the need to read the DOCKER_REGISTRY
environment variable. If you need it, the image
config is still present (e.g.
to reference images outside of OpenShift). projectId
and componentId
are
now automatically configured, but can still be set if desired.
Changed agent images
Up to and including 2.x, Jenkins used "slave images" to do its work. They have
been renamed to "agent images" in 3.x. All new agent images are located in the
new central namespace ods
. Therefore, you need to update the references in
your Jenkinsfile
(as shown above):
-
cd/jenkins-slave-base
is nowods/jenkins-agent-base
-
cd/jenkins-slave-golang
is nowods/jenkins-agent-golang
-
cd/jenkins-slave-maven
is nowods/jenkins-agent-maven
-
cd/jenkins-slave-nodejs10-angular
is nowods/jenkins-agent-nodejs10-angular
-
cd/jenkins-slave-python
is nowods/jenkins-agent-python
-
cd/jenkins-slave-scala
is nowods/jenkins-agent-scala
Changed context
properties
The context
object available to use in stages has been slimmed down by
removing properties unlikely to be useful to author stages (such as resource
constraints of the Jenkins agent pod). Please review
the documentation
in case your authored stages make heavy use of the context
object.
One notable change to the context object is that bitbucketHost did
not include the scheme previously. Now, bitbucketHost is an alias for
bitbucketUrl which does include the scheme. This change was made to align the
property with nexusHost and nexusUrl , which both include the scheme as well.
If you used context.bitbucketHost in one of your Jenkinsfile`s, you may now
use `context.bitbucketHostWithoutScheme .
|
A notable addition to the context object is a new property issueId , which
exposes the Jira issue ID (such as 123 from branch feature/FOO-123-bar-baz ).
|
Different image tags
Previously, images produced by odsComponentStageBuildOpenShiftImage
where
tagged with context.tagversion
, which consisted of the Jenkins build number
and the (shortened) Git commit (e.g. 7-cd3e9082
). This made it difficult
for other processes (unaware of the Jenkins build number) to find those images.
The images are now tagged with just the (shortened) Git commit (e.g. cd3e9082
).
This change also has the huge benefit that it allows to promote images between
environments (avoiding to rebuild them) using the new stage odsComponentStageImportOpenShiftImageOrElse.
Changes to Jenkins agent images
Some agent images (jenkins-agent-maven
, jenkins-agent-nodejs10-angular
, jenkins-agent-scala
) exposed the environment variables NEXUS_HOST
and NEXUS_USERNAME
/NEXUS_PASSWORD
/NEXUS_AUTH
. This was not done intentionally, but as a side effect of using those variables during image build time. In 3.x, that has been cleaned up. Now those variables are passed as build arguments to the image build, which means they are no longer accessible as environment variables in the running container. Nexus credentials can and should be accessed from the pipeline context (context.nexusUrl
, context.nexusUsername
and context.nexusPassword
). As a consequence, if a Jenkinsfile
relies on the presence of those environment variables without setting them explicitly using withEnv
, it will fail to run in 3.x. That said, the Jenkinsfile
templates which shipped by default with the quickstarters did not rely on the environment variables - with the exception of fe-ionic
which did not set NEXUS_HOST
explicitly in the Build
stage.
General upgrade to Python 3.8
Jenkins agent jenkins-agent-python
as well as all python related quickstarters (be-python-flask
, ds-jupyter-notebook
and ds-ml-service
) have been upgraded to python 3.8. This is not affecting default provisioned Jenkinsfile
for ds-jupyter-notebook
nor be-python-flask
quickstarters that exist before ODS 3.x. But, ds-ml-service
provisoned quickstarters before ODS 3.x need to slightly modify its Jenkinsfile
before upgrading to ODS 3.x by just avoiding running any pip install <..dependencies..> --user
(see new Jenkinsfile.template
to get a suggested fix, basically using virutalenv
; which is the recommended way of working when requiring building python packages).
Airflow Cluster Jenkins agent no longer supported
If you provisioned airflow-cluster
quickstarter, please note that it has been moved out from the officially supported quickstarters (ods-quickstarters
) repository and moved to extra-quickstarters
. Therefore, the jenkins-slave-airflow:2.x
can be shifted to jenkins-agent-python:3.x
when upgrading to ODS 3.x.