Update Guide

Learn all about how to update your OpenDevStack repositories and the running installation of it.

How to update your OpenDevStack repositories

Updating repositories means that new refs from repositories under github.com/opendevstack are pushed into the repositories in your BitBucket instance. Note that only updates to the production branch in BitBucket will have any effect on the OpenDevStack installation.

First, you need a clone of each repository in BitBucket which should be updated on your local machine.

Once this has been done, you need to fetch new refs from github.com/opendevstack. To do so, add a remote pointing to it like this:

git remote add ods https://github.com/opendevstack/<REPO_NAME>.git

Now you are ready to update the refs. It is recommended to update both the master branch and, unless you want to live off the bleeding edge, a release branch such as 1.0.x. Use the steps shown below:

# Ensure you have the latest refs from ODS locally
git fetch ods
# Update master
git checkout master
git reset --hard ods/master
git push origin master
# Update 1.0.x
git checkout 1.0.x
git reset --hard ods/1.0.x
git push origin 1.0.x

So far, your OpenDevStack installation has not been affected yet because the production branch has not been updated yet. To do that, it is recommended to create a pull request on BitBucket from 1.0.x into production This serves as a gate to control changes coming in, and spread knowledge about what has changed exactly.

Now that changes are "live", you need to update (parts of) the installation.

How to update your OpenDevStack installation

Updating consists of two parts: following the general update procedure (applicable to all version updates) and a version specific update procedure.

General update procedure


Before proceeding, it is advisable to make a backup of the existing OpenShift configuration. This can be done easily with Tailor:

# backup CD project
tailor export -n cd > backup_CD.yml

# backup provision app projects
tailor export -n prov-cd > backup_PROV_CD.yml
tailor export -n prov-dev > backup_PROV_DEV.yml
tailor export -n prov-test > backup_PROV_TEST.yml

Note that the executing user needs to have permissions to access all resources in the cd namespaces for this to work properly.


Next, update Tailor to the version corresponding to your new OpenDevStack version. At the moment, this version is noted at the start of each version specific update procedure.


Then, update/add/remove the configuration parameters (located in ods-configuration) according to what has changed in the ods-configuration-sample repository.


Since 1.2.x release, this sub-chapter only applies for versions below 1.2.x.

Once configuration is up-to-date, the OpenShift templates stored within OpenShift need to be updated:

# Within your local "ods-project-quickstarters" repository
cd ocp-templates/scripts

OCP resources

Next, run tailor update in all ocp-config folders from all the repositories to bring OCP resources (such as DC or BC) into sync. Review the diff produced by Tailor carefully, especially around changes to PVCs. To find each ocp-config folder, you can use e.g. find . -name ocp-config -not -path "./ods-configuration*" on Unix.

Shared images

After all OCP resources have been updated, you need to start a build for all build configs in the cd and shared-services namespaces to create new images.

Provisioning App

Also, the provisioning app should be updated. To do that, run tailor update in each ocp-config folder, and then trigger a build in Jenkins to redeploy the service.

Shared library

Then, push the vanilla branch or tag of the shared library to your BitBucket instance. E.g. if your production branch is based on 1.0.x, push ods-jenkins-shared-library@1.0.x. All quickstarters in the 1.0.x branch point to this Git reference in the Jenkinsfile. It is not recommended to point to production in your quickstarters, otherwise consumers will encounter failing builds when breaking changes are introduced in the production branch. You may also choose to create a custom tag from your production branch and make all `Jenkinsfile`s point to this tag.


Finally, in your Rundeck instance: * Update/Sync all jobs from this release * Update quickstarts/airflow_cluster job to get proper values for -h, -r and -oc flags. * Run verify global rundeck settings job

Now that the general procedure has been completed, you need to apply all the update notes below which apply to your version change.

1.1.x to 1.2.x

1.2.x requires Tailor 0.10.2.

Prepare installation for MRO/DocGen quickstarter

The new functionality to create documents via Jenkins requires the presence of an image for the DocGen service. In an upcoming release, this will be integrated nicely. For 1.2.x, the image needs to be built once during the update procedure. The recommended way to do this is to build the image in the prov namespaces and then move the image tag into the cd namespace. The first step is to create a pipeline oc -n prov-cd process -f pipeline.yml --param REPO_BASE=<YOUR-REPO-BASE-HERE> --param TRIGGER_SECRET=<YOUR-SECERET-HERE> | oc -n prov-cd create -f -, where pipeline.yml looks like this:

apiVersion: v1
kind: Template
- apiVersion: build.openshift.io/v1
  kind: BuildConfig
    name: docgen-production
    nodeSelector: {}
    output: {}
    postCommit: {}
    resources: {}
    runPolicy: Serial
        ref: production
        uri: ${REPO_BASE}/opendevstack/ods-document-generation-svc.git
        name: cd-user-with-password
        type: Git
        jenkinsfilePath: Jenkinsfile
        type: JenkinsPipeline
    - generic:
        secret: ${TRIGGER_SECRET}
        type: Generic
  required: true
- name: REPO_BASE
  required: true
  description: Path to repository, e.g. https://cd_user@bitbucket.domain.com/scm

Then, create the BuildConfig and ImageStream in prov-dev using oc -n prov-dev process -f bc-is.yml | oc -n prov-dev create -f -, where bc-is.yml looks like this:

apiVersion: v1
kind: Template
- apiVersion: v1
  kind: BuildConfig
    creationTimestamp: null
      app: prov-docgen
    name: docgen
    failedBuildsHistoryLimit: 5
    successfulBuildsHistoryLimit: 5
    nodeSelector: null
        kind: ImageStreamTag
        name: docgen:latest
    postCommit: {}
    resources: {}
    runPolicy: Serial
      binary: {}
      type: Binary
      dockerStrategy: {}
      type: Docker
    triggers: []
- apiVersion: v1
  kind: ImageStream
      app: prov-docgen
    name: docgen
    dockerImageRepository: docgen
      local: false

Note that this points to the production branch of ods-document-generation-svc - ensure this branch is present.

After all is setup, start a build in Jenkins, and then move the built image to the cd namespace:

oc tag prov-dev/docgen:latest cd/docgen:latest

1.0.x to 1.1.x

1.1.x requires Tailor 0.9.4.

There are no further mandatory changes apart from the general procedure described above when updating from 1.0.x.

However, it is highly recommended to take a look at the updates done to the boilerplates, especially the Jenkinsfile and Dockerfile. E.g. the Python quickstarter is now building an image containing all dependencies instead of installing them during runtime.

0.1.0 to 1.0.x

1.0.x requires Tailor 0.9.3.

Update xyz-cd projects

There is a new webhook proxy now, which proxies webhooks sent from BitBucket to Jenkins. As well as proxying, this service creates and deletes pipelines on the fly, allowing to have one pipeline per branch. To update:

  • Setup the image in the cd project by running tailor update in ods-core/jenkins/ocp-config.

  • Build the image.

  • Setup the webhook proxy next to each Jenkins instance. E.g., go to ods-project-quickstarters/ocp-templates/templates and run oc process cd//cd-jenkins-webhook-proxy | oc create -f- -n xyz-cd. Repeat for each project.

Update components

For each component, follow the following steps:

In Jenkinsfile:

  1. Set the shared library version to 1.0.x.

  2. Replace stageUpdateOpenshiftBuild with stageStartOpenshiftBuild.

  3. Remove stageCreateOpenshiftEnvironment and stageTriggerAllBuilds.

  4. Adapt the build logic to match the latest state of the quickstarter boilerplates.

  5. Remove verbose: true config (replace with debug: true if you want debug output).

  6. Configure branchToEnvironmentMapping, see README.md. If you used environment cloning, also apply the instructions for that.

In docker/Dockerfile:

  • Adapt the content to match the latest state of the quickstarter boilerplates.

  • No Nexus upload build artifact is required anymore, use a copy in Jenkins shell command to docker folder (see in any boilerplate how it is done now).

  • In BitBucket, remove the existing "Post Webhooks" and create a new "Webhook", pointing to the new webhook proxy. The URL has to be of the form https://webhook-proxy-$PROJECT_ID-cd.$DOMAIN?trigger_secret=$SECRET. As events, select "Repository Push" and "Pull request Merged + Declined".

Update provisioning app

If you want to build the provisioning app automatically when commits are pushed to BitBucket, add a webhook as described in the previous section.

Fix Jenkins master BUILD_URL

1.0.x makes use of the BUILD_URL env variable automatically set by Jenkins. This env variable might be null in your Jenkins master. To fix this, copy https://github.com/opendevstack/ods-core/blob/1.0.x/jenkins/master/configuration/init.groovy.d/url.groovy into each Jenins master to /var/lib/jenkins/init.groovy.d/url.groovy.

Fix JSON patch replace error in Jenkins build

1.0.x sets image labels on the BuildConfig in Jenkins. It does this by issuing a JSON patch replace request to /spec/output/imageLabels. This path was not present in prior versions, which can lead to the following error: Error from server: jsonpatch replace operation does not apply: doc is missing key: /spec/output/imageLabels. For newly provisioned components, this has been fixed with https://github.com/opendevstack/ods-project-quickstarters/pull/188. For existing components, add the path to the BuildConfig manually by editing the YAML in OpenShift.