Liquibase Enterprise can be integrated with Azure DevOps to deploy your database code. Visit the Azure DevOps website to learn more about https://learn.microsoft.com/en-us/azure/devops/pipelines/?view=azure-devops.
This document explains how to setup an Azure pipeline with an agent to execute Liquibase Enterprise operations. An Azure DevOps agent can be shared among multiple projects, so one agent can serve several teams. You can learn more about Azure agents here: https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/agents?view=azure-devops&tabs=browser.
Instructions
Prerequisites
Before following the steps in this document, setup your databases and create the Liquibase project.
Liquibase Enterprise requires two repositories which are referred to as Repos in Azure DevOps. Azure Pipelines can be built from code stored in Azure Repos, Bitbucket, GitHub, and other Git or Subversion sources. For this example we will be using Azure Repos.
In Azure Repos create the following two projects:
The Liquibase Project repository.
The SQL code repository.
Push the Liquibase project configuration files to the project repository in Azure Repos.
git init --initial-branch=main git remote add origin git@ssh.dev.azure.com:v3/asmith0101/Liquibase/ABC_DDB git push -u origin --all git add . git commit -m "Initial commit" git push -u origin main
Step 1: Setup an Azure Agent Pool and Azure Agent
Instructions: https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/agents?view=azure-devops
An Azure Agent Pool can be created under Project Settings. Agent pools are shared across an organization.
After setting up an Agent Pool you can create a New agent. Instructions will be provided for the type of agent (Windows, macOS, or Linux) that you wish to use.
Step 2: Setup an Azure Key Vault with the database credentials
Instructions: https://learn.microsoft.com/en-us/azure/devops/pipelines/release/key-vault-in-own-project?view=azure-devops&tabs=portal
The Azure Key Vault is a secure method for storing database credentials. It is recommended that you add Key Vault secrets for all databases in your pipeline including the DMC database.
In this example, Secrets added in the Key Vault will be referenced by the pipeline using Variable Groups.
Step 3: Setup an Azure Library Variable Group
Instructions: https://learn.microsoft.com/en-us/azure/devops/pipelines/library/?view=azure-devops and https://learn.microsoft.com/en-us/azure/devops/pipelines/library/variable-groups?view=azure-devops&tabs=yaml
Use an Azure Library to group credentials from your Key Vault into a Variable Group. This grouping makes it easy to associate credentials to Azure Pipelines and Releases.
Step 1: Install Liquibase Enterprise on an Agent
The agent can be a VM on Azure or other cloud platform. Instructions to setup an Azure VM can be found at https://learn.microsoft.com/en-us/azure/devops/pipelines/process/environments-virtual-machines?view=azure-devops. Make sure to use a supported version of Windows or Linux. Hardware, Software, and Database Requirements
Follow these instructions to install Liquibase Enterprise:
Windows: Installing Liquibase Enterprise Clients on Windows Systems
Linux: Theme: Using the CLI and the Composite Repository to Install Liquibase Enterprise on Linux Systems
Make sure that git is installed on the agent as it will be needed by the Liquibase Enterprise Deploy Packager. SCM Requirements and Features
Step 2: Install GitLab Runner on the Linux Server
https://docs.gitlab.com/runner/install/linux-repository.html
Login to the Linux VM and run the following commands.
curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.rpm.sh" | sudo bash sudo yum install gitlab-runner
This example uses SSH keys for git authentication. You can find information on using SSH keys with GitLab here: https://docs.gitlab.com/ee/user/ssh.html . Copy the SSH private key to /home/gitlab-runner/.ssh.
Step 3: Register the Runner
https://docs.gitlab.com/runner/register/index.html
Login to GitLab.
Navigate to the SQL repo.
Go to Settings > CI/CD
Expand the Runners section.
Copy the registration token.Start a terminal session on the GitLab runner. Run the “gitlab-runner register” command.
You will be prompted for the required information. We recommend creating a tag for the runner so that GitLab CI/CD jobs for Liquibase can be associated with this runner. The executor for the runner should be “shell”.sudo gitlab-runner register Runtime platform arch=amd64 os=linux pid=11051 revision=5316d4ac version=14.6.0 Running in system-mode. Enter the GitLab instance URL (for example, https://gitlab.com/): https://gitlab.com Enter the registration token: <the token that was copied above> Enter a description for the runner: [ip-172-30-3-12.ec2.internal]: myrunner.mydomain.com Enter tags for the runner (comma-separated): liquibase Registering runner... succeeded runner=q8McuSyX Enter an executor: docker, parallels, virtualbox, docker+machine, kubernetes, custom, shell, ssh, docker-ssh+machine, docker-ssh: shell Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!
Step 4: Create an Access Token
The Access Token (PAT) is needed to download artifacts.
To create a personal access token:
Login to GitLab.
On the top-right corner, select your avatar.
Select Edit Profile
On the left sidebar, select Access Tokens.
Enter a name.
Leave expiry date blank (never expires).
Select access token scope: api
Select create personal access token.
Save token. It cannot be retrieved later.
If you would prefer to use a Project Access Token, the instructions are here: https://docs.gitlab.com/ee/user/project/settings/project_access_tokens.html .
Step 5: Configure the .gitlab-ci.yml file
Deploy Packager jobs will be triggered automatically by commits to the packaging branch of the SQL repo. Forecast and Deploy jobs will be triggered manually from the GitLab web interface. The Logging job will always run to generate a .zip of files to be used for debugging errors.
The example .gitlab-ci.yml file should be placed in the top level of the SQL repo. Update the GitLab URLs for your Repos.
# Liquibase Sample gitlab-ci.yml stages: - package - forecast - deploy - logging packager-job: stage: package tags: - liquibase only: refs: - develop variables: - $jobtype == "PACKAGER" except: variables: # Don't execute when Liquibase commits back to the Repo. - $CI_COMMIT_AUTHOR == "Liquibase <support@liquibase.com>" script: - | echo "Packaging Stage" echo "Checking Environment" echo Project directory: $CI_PROJECT_DIR echo Project ID: $CI_PROJECT_ID echo Commit Author: $CI_COMMIT_AUTHOR whoami # Variables needed for Oracle projects. export PATH="$PATH:/opt/datical/DaticalDB/repl:/opt/datical/DaticalDB/instantclient" export LD_LIBRARY_PATH=/opt/datical/DaticalDB/instantclient hammer show version - | echo "Getting SQL repo" cd .. rm src -r -f mkdir -p src cd src git config --global user.name "Liquibase" git config --global user.email "support@liquibase.com" git config --global init.defaultBranch main git init git remote add origin git@gitlab.com:mbohls/$sql_repo.git git fetch origin git checkout -b $branch --track origin/$branch git branch --set-upstream-to=origin/$branch $branch - | echo "Getting DDB repo" cd .. rm ddb -r -f mkdir -p ddb cd ddb git config --global user.name "Liquibase" git config --global user.email "support@liquibase.com" git config --global init.defaultBranch main git init git remote add origin git@gitlab.com:mbohls/$ddb_repo.git git fetch origin git checkout -b main --track origin/main git branch --set-upstream-to=origin/main main - | echo "Packaging" hammer groovy deployPackager.groovy pipeline=$pipeline scm=true labels=$CI_JOB_ID zip -r $appname-$CI_JOB_ID.zip * -x "Reports/*" -x "Logs/*" -x "Snapshots/*" -x "Profiles/*" -x "daticaldb.log" -x "deployPackager.properties"; mv $appname-$CI_JOB_ID.zip $CI_PROJECT_DIR artifacts: paths: - $appname-$CI_JOB_ID.zip forecast-job: stage: forecast tags: - liquibase only: refs: - web variables: - $jobtype == "FORECAST" script: - | echo "Forecast Stage" echo Project directory: $CI_PROJECT_DIR echo Artifact ID: $packagenumber export PATH="$PATH:/opt/datical/DaticalDB/repl" export LD_LIBRARY_PATH=/opt/datical/DaticalDB/instantclient cd ../ddb mkdir -p forecast cd forecast # Get GitLab Artifact curl --output $appname-$packagenumber.zip --header "PRIVATE-TOKEN: <your PAT>" "https://gitlab.com/api/v4/projects/$CI_PROJECT_ID/jobs/$packagenumber/artifacts/$appname-$packagenumber.zip" # Unzip and forecast unzip $appname-$packagenumber.zip if [ -n "${labels}" ] then hammer forecast $environment --labels="${labels}" else hammer forecast $environment fi deploy-job: stage: deploy tags: - liquibase only: refs: - web variables: - $jobtype == "DEPLOY" script: - | echo "Deploy Stage" echo Project directory: $CI_PROJECT_DIR echo Artifact ID: $packagenumber export PATH="$PATH:/opt/datical/DaticalDB/repl" export LD_LIBRARY_PATH=/opt/datical/DaticalDB/instantclient cd ../ddb mkdir -p deploy cd deploy # Get GitLab Artifact curl --output $appname-$packagenumber.zip --header "PRIVATE-TOKEN: yourtoken" "https://gitlab.com/api/v4/projects/$CI_PROJECT_ID/jobs/$packagenumber/artifacts/$appname-$packagenumber.zip" # Unzip and deploy unzip $appname-$packagenumber.zip if [ -n "${labels}" ] then hammer deploy $environment --labels="${labels}" else hammer deploy $environment fi logging-job: stage: logging tags: - daticaldb except: variables: # Don't execute when Liquibase commits to the repo. - $CI_COMMIT_AUTHOR == "Liquibase <support@liquibase.com>" && $jobtype == "PACKAGER" script: - | # Generate log files for debugging echo "Post Stage" echo Project directory: $CI_PROJECT_DIR export PATH="$PATH:/opt/datical/DaticalDB/repl:/opt/datical/DaticalDB/instantclient" export LD_LIBRARY_PATH=/opt/datical/DaticalDB/instantclient # Switch to the DDB directory cd ../ddb hammer debug export --include="datical.project,changelog.xml,daticaldb*.log,*.html,deployPackager.properties,packager.log" --report=scrubbed_debug_files-$CI_JOB_ID.zip mv scrubbed_debug_files-$CI_JOB_ID.zip $CI_PROJECT_DIR # Run even if there was a job failure earlier so that we capture the log files. when: always artifacts: paths: - scrubbed_debug_files-$CI_JOB_ID.zip variables: #These are runtime variables. Global variables are set at the project level. environment: value: "ref" description: "DBDEF name. Valid values are: ref, test, prod" jobtype: value: "PACKAGER" description: "valid values are: PACKAGER, FORECAST, DEPLOY" packagenumber: value: "changeme" description: "FORECAST and DEPLOY JOBS ONLY. Unique number of artifact" labels: description: "FORECAST and DEPLOY JOBS ONLY. Label expression"
Note: You may need to manually initialize the git SSH key for both of the Repos.
git config --global user.name "Martha Bohls" git config --global user.email "mbohls@liquibase.com" git config --global init.defaultBranch main git init git remote add origin git@gitlab.com:mbohls/$ddb_repo.git
Step 6: Configure variables for the pipeline.
Variables are a good way to store database credentials. They also allow the same script to be configured to work for different projects.
Login to GitLab.
Navigate to the SQL repo for the project.
Go to Settings > CI/CD
Expand the Variables section.
The sample script requires the variables shown below.
appname: Name of your Liquibase project
pipeline: Name of the pipeline in the Liquibase project
branch: Branch in the SQL repo that is used for packaging
ddb_repo: Name of the Liquibase project repository
sql_repo: Name of the SQL repository
DDB_USER: Database user name
DDB_PASS: Database password
DDB_DMCDB_USER: Database user for DMCDB
DDB_DMCDB_PASS: Database password for DMCDB
Step 7: Test your pipeline.
Commit a change to the packaging branch in your SQL repo. GitLab should run the pipeline Packaging job using your runner.