Update Guide for OpenDevStack administrators

Learn all about how to update your OpenDevStack repositories and the running installation of it.

How to update your OpenDevStack repositories

Updating repositories means that new refs from repositories under github.com/opendevstack are pushed into the repositories in your BitBucket instance.

First, you need a clone of each repository in BitBucket which should be updated on your local machine.

Once this has been done, you need to fetch new refs from github.com/opendevstack. To do so, add a remote pointing to it like this:

git remote add ods https://github.com/opendevstack/<REPO_NAME>.git

Now you are ready to update the refs. It is recommended to update both the master branch and, unless you want to live off the bleeding edge, a release branch such as 2.x. Use the steps shown below:

# Ensure you have the latest refs from ODS locally
git fetch ods
# Update master
git checkout master
git reset --hard ods/master
git push origin master
# Update 2.x
git checkout 2.x
git reset --hard ods/2.x
git push origin 2.x

If your OpenDevStack installation is based on a custom branch (such as 2.acme), then you need to create a pull request on BitBucket from 2.x into that custom branch now.

Now that the repositories are updated, you also need to modify the images and the running instances in OpenShift.

How to update your OpenDevStack installation

Updating consists of two parts: following the general update procedure (applicable to all version updates) and a version specific update procedure.

General update procedure

Backup

Before proceeding, it is advisable to make a backup of the existing OpenShift configuration. This can be done easily with Tailor:

# Backup CD project
tailor export -n cd > backup_CD.yml

# Backup provision app namespaces
tailor export -n prov-cd > backup_PROV_CD.yml
tailor export -n prov-dev > backup_PROV_DEV.yml
tailor export -n prov-test > backup_PROV_TEST.yml

Note that the executing user needs to have permissions to access all resources in the cd namespaces for this to work properly.

Tailor

Next, update Tailor to the version corresponding to your new OpenDevStack version, which is noted at the start of each version specific update procedure.

Configuration

Then, update/add/remove the configuration parameters (located in ods-configuration). To do this, use the ./update script located in ods-core/configuration-sample.

OCP resources

Next, run tailor update in ods-core and ods-quickstarters to bring all OCP resources (such as DC or BC) into sync. Review the diff produced by Tailor carefully, especially around changes to PVCs.

Images

After all OCP resources have been updated, you need to start a build for all build configs in the cd namespace to create new images.

Provisioning App

Also, the provisioning app should be updated. To do that, run tailor update in each ocp-config folder, and then trigger a build in Jenkins to redeploy the service.

Now that the general procedure has been completed, you need to apply all the update notes below which apply to your version change.

1.2.x to 2.x

2.x requires Tailor 0.11.0.

Setup secure route checking

Secure route checking has been removed for version 3.x as this is an optional step. The code now is available at https://github.com/BIX-Digital/ods-contrib.

Go to ods-core/check-ocp-secure-routes/ocp-config and run tailor update to setup a cron job that will check exposed routes once a day (see https://github.com/opendevstack/ods-core/pull/280).

Project specific CD users

As each project may use a specific CD user now, you have to configure the username of the global CD user. To do so, add username: Y2RfdXNlcg== to secret cd/cd-user-token.

Deprecation of `shared-images` namespace

The shared-images namespace is no longer part of OpenDevStack. If you do not have any users that use images from that namespace, you may simply delete it via oc delete project shared-images. Otherwise, you can leave it in place and remote it when you see fit.

Rundeck removal

Rundeck is longer part of OpenDevStack and can simply be removed.

Image puller rights

Images in the cd namespaces should be pullable from all authenticated users. This permission is required for the new project provisioning approach to work:

oc adm policy add-cluster-role-to-group system:image-puller system:authenticated -n cd
oc adm policy add-role-to-group view system:authenticated -n cd

Further, -dev and -test namespaces should be able to pull images from the corresponding *-cd namespaces to make it easy to shared base images within a project (see https://github.com/opendevstack/ods-core/issues/293). It is recommended to grant these rights for every project in your cluster. If you don’t do this, users will have to add the permissions manually if they want to use this flow.

Rollout new webhook proxy instances

2.x allows the webhook proxy to build repositories in external projects if configured (see https://github.com/opendevstack/ods-core/issues/229). This feature is required for the new quickstarter provisioning approach to work. Therefore, it is recommended to tag a webhook proxy images built from the 2.x branch or v2.0 tag with latest so that all webhook proxies in the cluster get updated.

Configure the provisioning app

Review the ConfigMap of the provisioning app in prov-dev and prov-test. Depending on your requirements, you might want to configure additional quickstarters (jenkinspipeline.quickstarter…) and/or change the readable repositories of the project specific users (scm.global.readablerepos.opendevstack[x])

1.1.x to 1.2.x

1.2.x requires Tailor 0.10.2.

Prepare installation for release manager quickstarter

The new functionality to create documents via Jenkins requires the presence of an image for the DocGen service. In an upcoming release, this will be integrated nicely. For 1.2.x, the image needs to be built once during the update procedure. The recommended way to do this is to build the image in the prov namespaces and then move the image tag into the cd namespace. The first step is to create a pipeline oc -n prov-cd process -f pipeline.yml --param REPO_BASE=<YOUR-REPO-BASE-HERE> --param TRIGGER_SECRET=<YOUR-SECERET-HERE> | oc -n prov-cd create -f -, where pipeline.yml looks like this:

apiVersion: v1
kind: Template
objects:
- apiVersion: build.openshift.io/v1
  kind: BuildConfig
  metadata:
    name: docgen-production
  spec:
    nodeSelector: {}
    output: {}
    postCommit: {}
    resources: {}
    runPolicy: Serial
    source:
      git:
        ref: production
        uri: ${REPO_BASE}/opendevstack/ods-document-generation-svc.git
      sourceSecret:
        name: cd-user-with-password
        type: Git
    strategy:
      jenkinsPipelineStrategy:
        jenkinsfilePath: Jenkinsfile
        type: JenkinsPipeline
    triggers:
    - generic:
        secret: ${TRIGGER_SECRET}
        type: Generic
parameters:
- name: TRIGGER_SECRET
  required: true
- name: REPO_BASE
  required: true
  description: Path to repository, e.g. https://cd_user@bitbucket.domain.com/scm

Then, create the BuildConfig and ImageStream in prov-dev using oc -n prov-dev process -f bc-is.yml | oc -n prov-dev create -f -, where bc-is.yml looks like this:

apiVersion: v1
kind: Template
objects:
- apiVersion: v1
  kind: BuildConfig
  metadata:
    creationTimestamp: null
    labels:
      app: prov-docgen
    name: docgen
  spec:
    failedBuildsHistoryLimit: 5
    successfulBuildsHistoryLimit: 5
    nodeSelector: null
    output:
      to:
        kind: ImageStreamTag
        name: docgen:latest
    postCommit: {}
    resources: {}
    runPolicy: Serial
    source:
      binary: {}
      type: Binary
    strategy:
      dockerStrategy: {}
      type: Docker
    triggers: []
- apiVersion: v1
  kind: ImageStream
  metadata:
    labels:
      app: prov-docgen
    name: docgen
  spec:
    dockerImageRepository: docgen
    lookupPolicy:
      local: false

Note that this points to the production branch of ods-document-generation-svc - ensure this branch is present.

After all is setup, start a build in Jenkins, and then move the built image to the cd namespace:

oc tag prov-dev/docgen:latest cd/docgen:latest

1.0.x to 1.1.x

1.1.x requires Tailor 0.9.4.

There are no further mandatory changes apart from the general procedure described above when updating from 1.0.x.

Users are highly recommended to take a look at the updates done to the boilerplates, especially the Jenkinsfile and Dockerfile. E.g. the Python quickstarter is now building an image containing all dependencies instead of installing them during runtime.

0.1.0 to 1.0.x

1.0.x requires Tailor 0.9.3.

Update `xyz-cd` projects

There is a new webhook proxy now, which proxies webhooks sent from BitBucket to Jenkins. As well as proxying, this service creates and deletes pipelines on the fly, allowing to have one pipeline per branch. To update:

Setup the image in the cd project by running tailor update in ods-core/jenkins/ocp-config.
Build the image.
Setup the webhook proxy next to each Jenkins instance. E.g., go to ods-project-quickstarters/ocp-templates/templates and run oc process cd//cd-jenkins-webhook-proxy | oc create -f- -n xyz-cd. Repeat for each project.

Update components (information for ODS users)

For each component, follow the following steps:

In Jenkinsfile:

Set the shared library version to 1.0.x.
Replace stageUpdateOpenshiftBuild with stageStartOpenshiftBuild.
Remove stageCreateOpenshiftEnvironment and stageTriggerAllBuilds.
Adapt the build logic to match the latest state of the quickstarter boilerplates.
Remove verbose: true config (replace with debug: true if you want debug output).
Configure branchToEnvironmentMapping, see README.md. If you used environment cloning, also apply the instructions for that.

In docker/Dockerfile:

Adapt the content to match the latest state of the quickstarter boilerplates.
No Nexus upload build artifact is required anymore, use a copy in Jenkins shell command to docker folder (see in any boilerplate how it is done now).
In BitBucket, remove the existing "Post Webhooks" and create a new "Webhook", pointing to the new webhook proxy. The URL has to be of the form https://webhook-proxy-$PROJECT_ID-cd.$DOMAIN?trigger_secret=$SECRET. As events, select "Repository Push" and "Pull request Merged + Declined".

Update provisioning app

If you want to build the provisioning app automatically when commits are pushed to BitBucket, add a webhook as described in the previous section.

Fix Jenkins master BUILD_URL

1.0.x makes use of the BUILD_URL env variable automatically set by Jenkins. This env variable might be null in your Jenkins master. To fix this, copy https://github.com/opendevstack/ods-core/blob/1.0.x/jenkins/master/configuration/init.groovy.d/url.groovy into each Jenins master to /var/lib/jenkins/init.groovy.d/url.groovy.

Fix JSON patch replace error in Jenkins build

1.0.x sets image labels on the BuildConfig in Jenkins. It does this by issuing a JSON patch replace request to /spec/output/imageLabels. This path was not present in prior versions, which can lead to the following error: Error from server: jsonpatch replace operation does not apply: doc is missing key: /spec/output/imageLabels. For newly provisioned components, this has been fixed with https://github.com/opendevstack/ods-project-quickstarters/pull/188. For existing components, add the path to the BuildConfig manually by editing the YAML in OpenShift.