Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 4 Next »

Liquibase Enterprise can be integrated with GitLab CI/CD to deploy your database code. Visit the GitLab website to learn more about https://docs.gitlab.com/ee/ci/ .

These instructions explain how to setup a GitLab CI/CD pipeline with a dedicated runner to execute Liquibase Enterprise operations. A GitLab runner can be shared among multiple projects, so one runner can serve several teams. This example uses a Linux server for the runner. You can learn more about GitLab runners here: https://docs.gitlab.com/runner/ .

A Liquibase Enterprise project requires two repositories in GitLab.

  • The Liquibase Project repository.

  • The SQL code repository.

Before following these steps setup your databases and create the Liquibase project. Push the Liquibase project configuration files to the project repository in GitLab.

Step 1: Install Liquibase Enterprise on a Linux Server

Follow these instructions to install Liquibase Enterprise: Theme: Using the CLI and the Composite Repository to Install Datical DB on Linux Systems

Make sure that git is installed on the Linux server as it will be needed by the Liquibase Enterprise Deploy Packager.

Step 2: Install GitLab Runner on the Linux VM

https://docs.gitlab.com/runner/install/linux-repository.html

Login to the Linux VM and run the following commands.

curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.rpm.sh" | sudo bash

sudo yum install gitlab-runner

An SSH key is needed for git authentication. Copy the SSH private key to /home/gitlab-runner/.ssh.

Step 3: Register the Runner

https://docs.gitlab.com/runner/register/index.html

  1. Login to GitLab.

  2. Navigate to the SQL repo for the project.

  3. Go to Settings > CI/CD

  4. Expand the Runners section.
    Copy the registration token.

  5. Start a terminal session on the GitLab runner. Run the “gitlab-runner register” command.
    You will be prompted for the required information. We recommend creating a tag for the runner so that GitLab CI/CD jobs for Liquibase can be associated with this runner. The executor for the runner should be “shell”.

    sudo gitlab-runner register

Runtime platform     arch=amd64 os=linux pid=11051 revision=5316d4ac version=14.6.0
Running in system-mode.

Enter the GitLab instance URL (for example, https://gitlab.com/):
https://gitlab.com

Enter the registration token:
<the token that was copied above>

Enter a description for the runner:
[ip-172-30-3-12.ec2.internal]: myrunner.mydomain.com

Enter tags for the runner (comma-separated):
liquibase

Registering runner... succeeded                     runner=q8McuSyX

Enter an executor: docker, parallels, virtualbox, docker+machine, kubernetes, custom, shell, ssh, docker-ssh+machine, docker-ssh:
shell

Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!

C

Create Datical Project: HEB_DEMO

Databases: heb_ref, heb_test, heb_prod

Schemas: DATICAL_USER, DATICAL_TRACKING, MFG, SALES

Repo: heb_demo_sql, heb_demo_ddb

Add .gitignore to the project folder.

Logs/
Reports/
Snapshots/
Profiles/
Resources/sql/*.sql.tmp
.auditdb/
/.classpath
/.metadata
/.reporttemplates
/daticaldb.properties
/daticaldb*.log
*.zip

Configure SSH Key in GitLab


https://docs.gitlab.com/ee/ssh/index.html

  1. Login: http://gitlab.com

  2. Select: Avatar (upper right) > Preferences

  3. Select: SSH Keys (left side bar)

  4. Paste public key.
    Title: martha-win
    No expiration date.

Configure DDB Repo in GitLab

  1. Login: http://gitlab.com

  2. Menu>Project>Create New Project

  3. Select: Create Blank Project
    Project name: HEB_DEMO_DDB
    Project URL: https://gitlab.com/mbohls/heb_demo_ddb
    Visibility Level: Private
    Project Configuration: Do NOT initialize repository with a README

  4. Push Datical Project to the Repo

    git config --global user.name "Martha Bohls"
git config --global user.email "mbohls@liquibase.com"

cd existing_folder
git init --initial-branch=main
git remote add origin git@gitlab.com:mbohls/heb_demo_ddb.git
git add .
git commit -m "Initial commit"
git push -u origin main

Configure SQL Repo in GitLab

  1. Login: http://gitlab.com

  2. Menu>Project>Create New Project

  3. Select: Create Blank Project
    Project name: HEB_DEMO_SQL
    Project URL: https://gitlab.com/mbohls/heb_demo_sql
    Visibility Level: Private
    Project Configuration: Do NOT initialize repository with a README

  4. Push Datical Project to the Repo

    git config --global user.name "Martha Bohls"
git config --global user.email "mbohls@liquibase.com"

cd existing_folder
git init --initial-branch=main
git remote add origin git@gitlab.com:mbohls/heb_demo_sql.git
git add .
git commit -m "Initial commit"
git push -u origin main

git branch develop
git checkout develop
git push -u origin develop

Configure Packager

#deployPackager.properties
databaseBackupRestoreLocation=DATICAL_DATA_PUMP_DIR
databaseBackupRestoreMethod=BackupRestoreOracleDatapump
databaseBackupMode=always
packageOrderStrategy = default

ddbScmSystem=git
ddbScmBranch=main
ddbScmPath=.

# Default values
sqlScmSystem=git
# Don't use the SCM name from gitlab for the Path.  The SCM is automatically cloned for the gitlab-ci.yml file and is write protected.
sqlScmPath=../src
sqlScmSQLBaseDir=.

# Pipeline specific values

current.sqlScmBranch=develop
current.sqlScmLastImportID=2f75039f51d34156a1582e754db60a887a6a0c12
validateRollback = none
disableTraceFile = false

Install GitLab Runner for heb.datical.net

https://docs.gitlab.com/runner/install/linux-repository.html 

curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.rpm.sh" | sudo bash

sudo yum install gitlab-runner

# An SSH key is needed for git authentication.
# Copy the SSH private key to /home/gitlab-runner/.ssh

Register the Runner

https://docs.gitlab.com/runner/register/index.html

  1. From the HEB_DEMO_SQL project go to Settings > CI/CD

  2. Expand the Runners section.
    Copy the registration token.

  3. Start a terminal session heb.datical.net. Run the following command.

    sudo gitlab-runner register

Runtime platform                                    arch=amd64 os=linux pid=11051 revision=5316d4ac version=14.6.0
Running in system-mode.

Enter the GitLab instance URL (for example, https://gitlab.com/):
https://gitlab.com

Enter the registration token:
q8McuSyXwSyxC-NVxbU3

Enter a description for the runner:
[ip-172-30-3-12.ec2.internal]: heb.datical.net

Enter tags for the runner (comma-separated):
heb

Registering runner... succeeded                     runner=q8McuSyX

Enter an executor: docker, parallels, virtualbox, docker+machine, kubernetes, custom, shell, ssh, docker-ssh+machine, docker-ssh:
shell

Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!

Create Personal Access Token

  • A Project Access Token might be better, but I don’t think they are support for our level of SaaS.

  • The PAT is needed to download artifacts.

  • To create a personal access token:

    • On the top-right corner, select your avatar.

    • Select Edit Profile

    • On the left sidebar, select Access Tokens.

    • Enter a name.

    • Leave expiry date blank (never expires).

    • Select access token scope: api

    • Select create personal access token.

    • Save token. It cannot be retrieved later.

glpat-J-3kayfVBXvy_YszUsaD

Create the Pipeline

  • Configure variables for the pipeline.

    • Settings >> CI/CD

  • .gitlab-ci.yml file is located at the top level of the SQL Repo on the main branch.

    # Datical DB Sample .gitlab-ci.yml

stages:
  - prep
  - package
  - forecast
  - deploy
  - post
  
prep-phase-job:
  stage: prep
  tags:
    - heb
  only:
    - web
  script:
    - |
      echo "Checking Environment"
      echo  Project directory:  $CI_PROJECT_DIR
      echo Project ID:  $CI_PROJECT_ID
      whoami
      export PATH="$PATH:/opt/datical/DaticalDB/repl"
      hammer show version
    - |
      echo "Getting SRC repo"
      cd ..
      rm src -r -f
      mkdir -p src
      cd src
      git config --global user.name "Martha Bohls"
      git config --global user.email "mbohls@liquibase.com"
      git config --global init.defaultBranch main
      git init
      git remote add origin git@gitlab.com:mbohls/$sql_repo.git
      git fetch origin
      git checkout -b $branch --track origin/$branch
      git branch --set-upstream-to=origin/$branch $branch
    - |
      echo "Getting DDB repo"
      cd ..
      rm ddb -r -f
      mkdir -p ddb
      cd ddb
      git config --global user.name "Martha Bohls"
      git config --global user.email "mbohls@liquibase.com"
      git config --global init.defaultBranch main
      git init
      git remote add origin git@gitlab.com:mbohls/$ddb_repo.git
      git fetch origin      
      git checkout -b main --track origin/main
      git branch --set-upstream-to=origin/main main

packager-job:
  stage: package
  tags:
    - heb
  only:
    refs:
      - web
    variables:
      - $jobtype == "PACKAGER"
  script:
    - |
      echo "Packager Stage"
      echo Project directory: $CI_PROJECT_DIR
      export PATH="$PATH:/opt/datical/DaticalDB/repl:/opt/datical/DaticalDB/instantclient"
      export LD_LIBRARY_PATH=/opt/datical/DaticalDB/instantclient
      # Switch to the DDB directory
      cd ../ddb
      hammer groovy deployPackager.groovy pipeline=$pipeline scm=true labels=$CI_JOB_ID
      zip -r $appname-$CI_JOB_ID.zip * -x "Reports/*" -x "Logs/*" -x "Snapshots/*" -x "Profiles/*" -x "daticaldb.log" -x "deployPackager.properties";
      mv $appname-$CI_JOB_ID.zip $CI_PROJECT_DIR

  artifacts:
    paths:
      - $appname-$CI_JOB_ID.zip

forecast-job:
  stage: forecast
  tags:
    - heb
  only:
    refs:
      - web
    variables:
      - $jobtype == "FORECAST"
  script:
    - |
      echo "Forecast Stage"
      echo Project directory: $CI_PROJECT_DIR
      echo Artifact ID: $packagenumber
      export PATH="$PATH:/opt/datical/DaticalDB/repl"
      export LD_LIBRARY_PATH=/opt/datical/DaticalDB/instantclient
      cd ../ddb
      mkdir -p forecast
      cd forecast
      
      # Get GitLab Artifact
      curl --output $appname-$packagenumber.zip --header "PRIVATE-TOKEN: glpat-J-3kayfVBXvy_YszUsaD" "https://gitlab.com/api/v4/projects/$CI_PROJECT_ID/jobs/$packagenumber/artifacts/$appname-$packagenumber.zip"

      # Unzip and forecast
      unzip $appname-$packagenumber.zip

      if [ -n "${labels}" ]
      then
      hammer forecast $environment --labels="${labels}"
      else
      hammer forecast $environment
      fi

deploy-job:
  stage: deploy
  tags:
    - heb
  only:
    refs:
      - web
    variables:
      - $jobtype == "DEPLOY"
  script:
    - |
      echo "Deploy Stage"
      echo Project directory: $CI_PROJECT_DIR
      echo Artifact ID:  $packagenumber
      export PATH="$PATH:/opt/datical/DaticalDB/repl"
      export LD_LIBRARY_PATH=/opt/datical/DaticalDB/instantclient
      cd ../ddb
      mkdir -p deploy
      cd deploy
      
      # Get GitLab Artifact
      curl --output $appname-$packagenumber.zip --header "PRIVATE-TOKEN: glpat-J-3kayfVBXvy_YszUsaD" "https://gitlab.com/api/v4/projects/$CI_PROJECT_ID/jobs/$packagenumber/artifacts/$appname-$packagenumber.zip"

      # Unzip and deploy
      unzip $appname-$packagenumber.zip
      if [ -n "${labels}" ]
      then
      hammer deploy $environment --labels="${labels}"
      else
      hammer deploy $environment
      fi

post-job:
  stage: post
  tags:
    - heb
  only:
    refs:
      - web

  script:
    - |
      # Generate log files for debugging
      echo "Post Stage"
      echo Project directory: $CI_PROJECT_DIR
      export PATH="$PATH:/opt/datical/DaticalDB/repl:/opt/datical/DaticalDB/instantclient"
      export LD_LIBRARY_PATH=/opt/datical/DaticalDB/instantclient
      # Switch to the DDB directory
      cd ../ddb
      hammer debug export --include="datical.project,changelog.xml,daticaldb*.log,*.html,deployPackager.properties,packager.log" --report=scrubbed_debug_files-$CI_JOB_ID.zip
      mv scrubbed_debug_files-$CI_JOB_ID.zip $CI_PROJECT_DIR

  when: always
  
  artifacts:
    paths:
      - scrubbed_debug_files-$CI_JOB_ID.zip

variables:
#These are runtime variables.  Global variables are set at the project level.
  environment: 
    value: "heb_ref"
    description: "DBDEF name. Valid values are: heb_ref, heb_test, heb_prod"
  jobtype: 
    value: "PACKAGER"
    description: "valid values are: PACKAGER, FORECAST, DEPLOY" 
  packagenumber: 
    value: "changeme"
    description: "FORECAST and DEPLOY JOBS ONLY. Unique number of artifact"
  labels: 
    description: "FORECAST and DEPLOY JOBS ONLY. Label expression"

  • The runner has the “heb” tag, so the jobs in the script must have the “heb” tag. This tells GitLab that the jobs can only run on the Runner heb.datical.net.

  • Manually initialize the git SSH key for both of the Repos. Otherwise, the Prep Job fails.

    git config --global user.name "Martha Bohls"
git config --global user.email "mbohls@liquibase.com"
git config --global init.defaultBranch main
git init
git remote add origin git@gitlab.com:mbohls/$ddb_repo.git

  • Artifacts are associated with the Packager Job ID. To view artifacts

    • CI/CD > Jobs

    • Select the job

    • Select Browse under Job Artifacts in the right-hand panel.

  • l

kl

  • No labels

Copyright © Datical 2012-2020 - Proprietary and Confidential