Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 10 Current »

Liquibase Enterprise can be integrated with GitLab CI/CD to deploy your database code. Visit the GitLab website to learn more about https://docs.gitlab.com/ee/ci/ .

This document explains how to setup a GitLab CI/CD pipeline with a dedicated runner to execute Liquibase Enterprise operations. A GitLab runner can be shared among multiple Liquibase projects, so one runner can serve several teams. This example uses a Linux server for the runner. You can learn more about GitLab runners here: https://docs.gitlab.com/runner/ .

Prerequisites

Before following the steps in this document, setup your databases and create the Liquibase project.

Liquibase Enterprise requires two repositories which are referred to as projects in GitLab. In GitLab create the following two projects:

  • The Liquibase Project repository.

  • The SQL code repository.

Push the Liquibase project configuration files to the project repository in GitLab.

git init --initial-branch=main
git remote add origin git@gitlab.com:lbe/demo_project.git
git add .
git commit -m "Initial commit"
git push -u origin main

Step 1: Install Liquibase Enterprise on a Linux Server

The Linux server can be a VM on a cloud platform such as AWS. Make sure to use a supported version of Linux. Hardware, Software, and Database Requirements

Follow these instructions to install Liquibase Enterprise: Theme: Using the CLI and the Composite Repository to Install Liquibase Enterprise on Linux Systems

Make sure that git is installed on the Linux server as it will be needed by the Liquibase Enterprise Deploy Packager. SCM Requirements and Features

Step 2: Install GitLab Runner on the Linux Server

https://docs.gitlab.com/runner/install/linux-repository.html

Login to the Linux VM and run the following commands.

curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.rpm.sh" | sudo bash

sudo yum install gitlab-runner

This example uses SSH keys for git authentication. You can find information on using SSH keys with GitLab here: https://docs.gitlab.com/ee/user/ssh.html . Copy the SSH private key to /home/gitlab-runner/.ssh.

Step 3: Register the Runner

https://docs.gitlab.com/runner/register/index.html

  1. Login to GitLab.

  2. Navigate to the SQL repo.

  3. Go to Settings > CI/CD

  4. Expand the Runners section.
    Copy the registration token.

  5. Start a terminal session on the GitLab runner. Run the “gitlab-runner register” command.
    You will be prompted for the required information. We recommend creating a tag for the runner so that GitLab CI/CD jobs for Liquibase can be associated with this runner. The executor for the runner should be “shell”.

    sudo gitlab-runner register
    
    Runtime platform     arch=amd64 os=linux pid=11051 revision=5316d4ac version=14.6.0
    Running in system-mode.
    
    Enter the GitLab instance URL (for example, https://gitlab.com/):
    https://gitlab.com
    
    Enter the registration token:
    <the token that was copied above>
    
    Enter a description for the runner:
    [ip-172-30-3-12.ec2.internal]: myrunner.mydomain.com
    
    Enter tags for the runner (comma-separated):
    liquibase
    
    Registering runner... succeeded                     runner=q8McuSyX
    
    Enter an executor: docker, parallels, virtualbox, docker+machine, kubernetes, custom, shell, ssh, docker-ssh+machine, docker-ssh:
    shell
    
    Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!

Step 4: Create an Access Token

  • The Access Token (PAT) is needed to download artifacts.

  • To create a personal access token:

    • Login to GitLab.

    • On the top-right corner, select your avatar.

    • Select Edit Profile

    • On the left sidebar, select Access Tokens.

    • Enter a name.

    • Leave expiry date blank (never expires).

    • Select access token scope: api

    • Select create personal access token.

    • Save token. It cannot be retrieved later.

  • If you would prefer to use a Project Access Token, the instructions are here: https://docs.gitlab.com/ee/user/project/settings/project_access_tokens.html .

Step 5: Configure the .gitlab-ci.yml file

Deploy Packager jobs will be triggered automatically by commits to the packaging branch of the SQL repo. Forecast and Deploy jobs will be triggered manually from the GitLab web interface. The Logging job will always run to generate a .zip of files to be used for debugging errors.

The example .gitlab-ci.yml file should be placed in the top level of the SQL repo. Update the GitLab URLs for your Repos.

# Liquibase Sample gitlab-ci.yml

stages:
  - package
  - forecast
  - deploy
  - logging

packager-job:
  stage: package
  tags:
    - liquibase
  only:
    refs:
      - develop
    variables:
      - $jobtype == "PACKAGER"
  except:
    variables:
      # Don't execute when Liquibase commits back to the Repo.
      - $CI_COMMIT_AUTHOR == "Liquibase <support@liquibase.com>"
  script:
     - |
      echo "Packaging Stage"
      echo "Checking Environment"
      echo Project directory:  $CI_PROJECT_DIR
      echo Project ID:  $CI_PROJECT_ID
      echo Commit Author:  $CI_COMMIT_AUTHOR
      whoami
      # Variables needed for Oracle projects.
      export PATH="$PATH:/opt/datical/DaticalDB/repl:/opt/datical/DaticalDB/instantclient"
      export LD_LIBRARY_PATH=/opt/datical/DaticalDB/instantclient
      hammer show version
     - |
      echo "Getting SQL repo"
      cd ..
      rm src -r -f
      mkdir -p src
      cd src
      git config --global user.name "Liquibase"
      git config --global user.email "support@liquibase.com"
      git config --global init.defaultBranch main
      git init
      git remote add origin git@gitlab.com:mbohls/$sql_repo.git
      git fetch origin
      git checkout -b $branch --track origin/$branch
      git branch --set-upstream-to=origin/$branch $branch
     - |
      echo "Getting DDB repo"
      cd ..
      rm ddb -r -f
      mkdir -p ddb
      cd ddb
      git config --global user.name "Liquibase"
      git config --global user.email "support@liquibase.com"
      git config --global init.defaultBranch main
      git init
      git remote add origin git@gitlab.com:mbohls/$ddb_repo.git
      git fetch origin      
      git checkout -b main --track origin/main
      git branch --set-upstream-to=origin/main main
     - |
      echo "Packaging"

      hammer groovy deployPackager.groovy pipeline=$pipeline scm=true labels=$CI_JOB_ID
      zip -r $appname-$CI_JOB_ID.zip * -x "Reports/*" -x "Logs/*" -x "Snapshots/*" -x "Profiles/*" -x "daticaldb.log" -x "deployPackager.properties";
      mv $appname-$CI_JOB_ID.zip $CI_PROJECT_DIR

  artifacts:
    paths:
      - $appname-$CI_JOB_ID.zip

forecast-job:
  stage: forecast
  tags:
    - liquibase
  only:
    refs:
      - web
    variables:
      - $jobtype == "FORECAST"
  script:
    - |
      echo "Forecast Stage"
      echo Project directory: $CI_PROJECT_DIR
      echo Artifact ID: $packagenumber
      export PATH="$PATH:/opt/datical/DaticalDB/repl"
      export LD_LIBRARY_PATH=/opt/datical/DaticalDB/instantclient
      cd ../ddb
      mkdir -p forecast
      cd forecast
      
      # Get GitLab Artifact
      curl --output $appname-$packagenumber.zip --header "PRIVATE-TOKEN: <your PAT>" "https://gitlab.com/api/v4/projects/$CI_PROJECT_ID/jobs/$packagenumber/artifacts/$appname-$packagenumber.zip"

      # Unzip and forecast
      unzip $appname-$packagenumber.zip

      if [ -n "${labels}" ]
      then
      hammer forecast $environment --labels="${labels}"
      else
      hammer forecast $environment
      fi

deploy-job:
  stage: deploy
  tags:
    - liquibase
  only:
    refs:
      - web
    variables:
      - $jobtype == "DEPLOY"
  script:
    - |
      echo "Deploy Stage"
      echo Project directory: $CI_PROJECT_DIR
      echo Artifact ID:  $packagenumber
      export PATH="$PATH:/opt/datical/DaticalDB/repl"
      export LD_LIBRARY_PATH=/opt/datical/DaticalDB/instantclient
      cd ../ddb
      mkdir -p deploy
      cd deploy
      
      # Get GitLab Artifact
      curl --output $appname-$packagenumber.zip --header "PRIVATE-TOKEN: yourtoken" "https://gitlab.com/api/v4/projects/$CI_PROJECT_ID/jobs/$packagenumber/artifacts/$appname-$packagenumber.zip"

      # Unzip and deploy
      unzip $appname-$packagenumber.zip
      if [ -n "${labels}" ]
      then
      hammer deploy $environment --labels="${labels}"
      else
      hammer deploy $environment
      fi

logging-job:
  stage: logging
  tags:
    - daticaldb
  except:
    variables:
      # Don't execute when Liquibase commits to the repo.
      - $CI_COMMIT_AUTHOR == "Liquibase <support@liquibase.com>" && $jobtype == "PACKAGER"
  script:
    - |
      # Generate log files for debugging
      echo "Post Stage"
      echo Project directory: $CI_PROJECT_DIR
      export PATH="$PATH:/opt/datical/DaticalDB/repl:/opt/datical/DaticalDB/instantclient"
      export LD_LIBRARY_PATH=/opt/datical/DaticalDB/instantclient
      # Switch to the DDB directory
      cd ../ddb
      hammer debug export --include="datical.project,changelog.xml,daticaldb*.log,*.html,deployPackager.properties,packager.log" --report=scrubbed_debug_files-$CI_JOB_ID.zip
      mv scrubbed_debug_files-$CI_JOB_ID.zip $CI_PROJECT_DIR
  # Run even if there was a job failure earlier so that we capture the log files.
  when: always
  
  artifacts:
    paths:
      - scrubbed_debug_files-$CI_JOB_ID.zip

variables:
#These are runtime variables.  Global variables are set at the project level.
  environment: 
    value: "ref"
    description: "DBDEF name. Valid values are: ref, test, prod"
  jobtype: 
    value: "PACKAGER"
    description: "valid values are: PACKAGER, FORECAST, DEPLOY" 
  packagenumber: 
    value: "changeme"
    description: "FORECAST and DEPLOY JOBS ONLY. Unique number of artifact"
  labels: 
    description: "FORECAST and DEPLOY JOBS ONLY. Label expression"

Note: You may need to manually initialize the git SSH key for both of the Repos.

git config --global user.name "Martha Bohls"
git config --global user.email "mbohls@liquibase.com"
git config --global init.defaultBranch main
git init
git remote add origin git@gitlab.com:mbohls/$ddb_repo.git

Step 6: Configure variables for the pipeline.

Variables are a good way to store database credentials. They also allow the same script to be configured to work for different projects.

  1. Login to GitLab.

  2. Navigate to the SQL repo for the project.

  3. Go to Settings > CI/CD

  4. Expand the Variables section.

The sample script requires the variables shown below.

appname: Name of your Liquibase project

pipeline: Name of the pipeline in the Liquibase project

branch: Branch in the SQL repo that is used for packaging

ddb_repo: Name of the Liquibase project repository

sql_repo: Name of the SQL repository

DDB_USER: Database user name

DDB_PASS: Database password

DDB_DMCDB_USER: Database user for DMCDB

DDB_DMCDB_PASS: Database password for DMCDB

Step 7: Test your pipeline.

Commit a change to the packaging branch in your SQL repo. GitLab should run the pipeline Packaging job using your runner.

  • No labels