Orchestration Pipeline

The library automatically resolves dependencies between repositories to be orchestrated so that they can be delivered in the correct order. Currently, repositories that want to be orchestrated need to be added to the repositories list inside a release manager component’s metadata.yml:

If a named repository wants to announce a dependency on another repo, the dependency needs to be listed in that repository’s release-manager.yml, simply by referring to its repo.id as defined in metadata.yml:

The library supports the following repository types: ods, ods-infra, ods-service, ods-saas-service, ods-test and ods-library. Setting a repository type is required so the orchestrator can make correct assumptions based on the nature of the component at hand:

The library will attempt to resolve the repository URL based on the component’s origin remote URL and one of the following:

1) If the name parameter is provided, and not empty, the last path part of the URL is resolved to ${repo-name}.git. 2) If no name parameter is provided, the last path part of the URL is resolved to ${project-id}-${repo-id}.git (which is the repository name pattern used with OpenDevStack). Here ${project-id} refers to the lowercase value of the top-level id attribute in metadata.yml.

The default branch of the repository will be assumed.

If preview-branch parameter is provided for a repository, then it will be assumed as the default branch.

Instead of merely resolving repositories into a strictly sequential execution model, our library automatically understands which repositories form independent groups and can run in parallel for best time-to-feedback and time-to-delivery.

By default the shared library will rebuild all type ods components, no matter which ones changed since the last release. In order to build only the components whose source code changed (partial rebuilding as we will call it from now on), the following needs to be configured in metadata.yml

If one repository should always be rebuilt, even if partial rebuild is configured on root level, forceRebuild : true can be set at repository level, e.g.

It is important to highlight that, despite having configured partial rebuild, the orchestration pipeline will still deploy all the components (both those which changed and which did not) to the target environment.

By default the shared library will always pull the agent image from the internal docker repository. Depending on the cluster node setup, this may decrease execution performance. In order to re-use loaded images, a knob in the Jenkinsfile configuration of the stage odsOrchestrationPipeline can be turned on:

By default the orchestration pipeline will create a pod based on the jenkins-base-agent image to do much of its work. In seldom cases, ususally with a lot of repositories, one may hit an out of memory error on the pod named 'mro-XX'. In this case the below memory limit should be adjusted (defaulting to '1Gi')

The Zephyr for Jira capability currently supports:

Typically, software is running in different environments, such as one environment for development (DEV), one for quality assurance (QA), and one for production (PROD - this is what end-users of the software consume). Developers work on on the software in the development environment, and once they finish one version (a state) of the software, they bring that version to the QA environment, and once this version is deemed production-ready it is brought to the production environment so that users can consume the new version.

The environment promotion feature of the orchestration pipeline automates moving a certain version of the software from one environment to the next. Developers only have to tell the orchestration pipeline if a new version should be built (in DEV) and packaged as an installable "release bundle", or if an existing "release bundle" should be promoted to either the QA or the production environment.

The environment promotion feature is part of the regular orchestration pipeline. Therefore, the promotion is executed from various Jenkins stages. It is not possible to change the process itself, but you can customize how the promotion happens exactly for each of your software components.

The components of your software are defined in the repositories section of the metadata.yml file in the release manager repository. In order for the orchestration pipeline to know which state of each component should be promoted, it needs to have some knowledge about how version control in your repositories is organised. Everything depends on a user-supplied build parameter named version to the Jenkins pipeline. Other input parameters do not have any impact on source code lookup.

A specific "release bundle" is identified by four data points: a version (as outlined above), a changeId, a build number and an environment. The version, changeId and environment are user-supplied input parameters to the release manager pipeline, the build number is calculated automatically. The changeId can be any string meaningful to the user, its value does not have any effect on the operation of the orchestration pipeline. The environment input variable (such as DEV) will be shortened to a single-letter token (e.g. D).

Technically speaking, a release bundle is a certain state of the release manager repository and the state of each linked repository at that time. This state is identified by a Git tag. For example, a release bundle with version=1, changeId=1234, buildNumber=0 and environment=DEV is identified by the Git tag v1-1234-0-D. This tag is set on the release manager repository, and all repositories the metadata.yml refers to at this time.

The orchestration pipeline assumes three "conceptual" environments: DEV, QA and PROD (with short token forms D, Q and P). Those environments are strictly ordered - a state should go from DEV to QA, and then from QA to PROD.

To ensure that software progresses along the DEV → QA → PROD path, release bundles from environment DEV can only be installed into QA, and only a release bundle from QA can be installed into PROD. Installing a release bundle from DEV into PROD is not allowed.

Each "conceptual" environment is mapped to an OpenShift namespace:

Keep in mind that when you create a new project with OpenDevStack, you get three OpenShift namespaces:

So while there is a corresponding namespace for DEV and QA, there is no namespace corresponding to the PROD environment out-of-the-box. This is because it is assumed that your PROD environment is likely on another cluster altogether. To create foo-prod on another cluster, you (or someone with appropriate rights) can run the script located at https://github.com/opendevstack/ods-core/blob/master/ocp-scripts/create-target-project.sh. Then you need to tell orchestration pipeline two things: where the API of the external cluster is, and the credentials with which to access it. A typical configuration is:

This assumes you have the API token credentials stored in a secret of type kubernetes.io/basic-auth named foo-prod in the foo-cd namespace. This secret needs to be synced with Jenkins (which is achieved by labeling it with credential.sync.jenkins.openshift.io=true). The stored credentials need to belong to a serviceaccount with rights to admin the foo-prod namespace. The easiest way to setup all of this is by running the script located at https://github.com/opendevstack/ods-core/blob/master/ocp-scripts/create-target-sa-secret.sh, which makes use of the output of the create-target-project.sh ran earlier.

As mentioned in the "Source Code Organisation" section, the orchestration pipeline allows to enable separate development environments to isolate different versions. When this mode is enabled, pipeline runs with version=WIP will deploy into the $PROJECT-dev as usual, but pipeline runs with version=X will deploy into $PROJECT-dev-X. The $PROJECT-dev-X environment has to be created beforehand (e.g. by cloning $PROJECT-dev with its serviceaccounts and rolebindings). To enable this feature, set versionedDevEnvs to true in the config of your Jenkinsfile, like this:

Let’s start by assuming you have a project FOO with two components, X and Y. These components are defined under the repositories section in the metadata.yml file of the release manager repository. When you want to create a new release, you start the orchestration pipeline with input parameters - we will use version 1 and change ID 1234 in this example. The environment should be DEV. At the end of the pipeline run, you’ll have a release bundle identified by the tag v1-1234-0-D. This release can later be promoted as-is to QA. Once it is installed there, the same release bundle will be tagged with v1-1234-0-Q which can then be promoted to PROD (where it will be tagged with v1-1234-0-P).

To create a release bundle, the orchestration pipeline will first trigger the build of each component. Then, it will export all resources in your OpenShift namespace ($PROJECT-$ENVIRONMENT, here foo-dev) belonging to the component. By convention, this means all resources labeled with app=$PROJECT-$COMPONENT (e.g. app=foo-x). Any resources without such a label will NOT be part of the release bundle. The exported resources are stored in a template.yml file (an OpenShift template) located in the openshift-exported folder within each component repository. Further, the container image SHA of the running pod is retrieved and stored in the file image-sha in the same folder. Once done, the orchestration pipeline will commit the two files, tag the commit with v1-1234-0-D and push to the remote. After this process has been done for all repositories, the same tag is also applied to the release manager repository. At this stage, the "dev release bundle" is complete and can be installed into QA.

To trigger the installation of an existing release bundle, the user needs to supply a version and changeId which has previously been used to create a release bundle. In our example, supplying version=1, changeId=1234 and environment=QA will promote the release bundle identified by v1-1234-0-D to the QA environment and tag it with v1-1234-0-Q. Now that we have a "QA release bundle", we can promote it to PROD by supplying version=1, changeId=1234 and environment=PROD.

As outlined above, a release bundle is essentially a state of all involved Git repositories. Each component repository contains two artifacts:

You cannot modify the image SHA (it is the result of what the component pipeline builds), but you can influence the OpenShift template. One reason to do so is that e.g. routes or ConfigMap values will need to differ between environments, and you need to tell the orchestration pipeline to parametrize the templates, and to supply the right values when the templates are applied in the target environment.

When the orchestration pipeline exports configuration, it has no way to tell which values should actually be parameters. For example, you might have a route x.foo-dev.dev-cluster.com in DEV, and want this to be x.foo-test.dev-cluster.com in QA and x.foo-prod.prod-cluster.com in PROD. In the exported template, the value x.foo-dev.dev-cluster.com will be hardcoded. To fix this, you can create three files in the release manager repository, dev.env, qa.env and prod.env. These files may contain PARAM=value lines, like this:

dev.env

qa.env

prod.env

All three files need to list the exact same parameters - otherwise applying the templates will fail. Once those param files are present, the orchestration pipeline will pick them up automatically. When you create a release bundle (in DEV), the param file is applied "in reverse", meaning that any concrete param value (on the right) will be substituted with the param key (on the left) in the template. Later when the template is applied in e.g. QA, the param keys are replaced with the concrete values from qa.env.

Next to parametrizing templates, you can also adjust how the export is done. As the export is using Tailor, the best way to customize is to supply a Tailorfile in the openshift-exported folder, in which you can define the options you want to set, such as excluding certain labels or resource types, or preserving specific fields in the live configuration. Please see Tailor’s documentation for more information. It is also possible to have different configuration files per environment if you suffix with the $PROJECT, e.g. Tailorfile.foo-dev.

In the process described above, the OpenShift configuration is exported and stored in the repositories in openshift-exported. This approach is easy to get started with, but it does have limitations:

To overcome these issues, it is possible to author the OpenShift templates yourself instead of exporting them. The fastest way to start with this is by renaming the folder openshift-exported (containing the exported template) to openshift. From this point on, the orchestration pipeline will skip the export, and apply whatever is defined in the openshift folder.

When you author templates, you can also store the secrets in the param files GPG encrypted (.env.enc files). To achieve this, you need to create a private/public keypair for Jenkins, store the private key in a secret called tailor-private-key in your foo-cd namespace, and sync it as a Jenkins credentials item. Once the .env.enc files are encrypted against the public key, the orchestration pipeline will automatically use the private key to decrypt the params on-the-fly. Please see Working with Secrets for more information.