Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 3 Next »

Liquibase Enterprise can be integrated with GitLab CI/CD to deploy your database code. Visit the GitLab website to learn more about https://docs.gitlab.com/ee/ci/ .

These instructions explain how to setup a GitLab CI/CD pipeline with a dedicated runner to execute Liquibase Enterprise operations. A GitLab runner can be shared among multiple projects, so one runner can serve several teams. This example uses a Linux VM for the runner. You can learn more about GitLab runners here: https://docs.gitlab.com/runner/ .

A Liquibase Enterprise project requires two repositories in GitLab.

  • The Liquibase Project repository.

  • The SQL code repository.

Before following these steps make sure you

Step 1: Install Liquibase Enterprise on a Linux VM

Follow these instructions to install Liquibase Enterprise: Theme: Using the CLI and the Composite Repository to Install Datical DB on Linux Systems

Step 2: Install GitLab Runner on the Linux VM

https://docs.gitlab.com/runner/install/linux-repository.html

Login to the Linux VM and run the following commands.

curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.rpm.sh" | sudo bash

sudo yum install gitlab-runner

An SSH key is needed for git authentication. Copy the SSH private key to /home/gitlab-runner/.ssh.

Step 3: Register the Runner

https://docs.gitlab.com/runner/register/index.html

  1. From the HEB_DEMO_SQL project go to Settings > CI/CD

  2. Expand the Runners section.
    Copy the registration token.

  3. Start a terminal session heb.datical.net. Run the following command.

    sudo gitlab-runner register

Runtime platform                                    arch=amd64 os=linux pid=11051 revision=5316d4ac version=14.6.0
Running in system-mode.

Enter the GitLab instance URL (for example, https://gitlab.com/):
https://gitlab.com

Enter the registration token:
q8McuSyXwSyxC-NVxbU3

Enter a description for the runner:
[ip-172-30-3-12.ec2.internal]: heb.datical.net

Enter tags for the runner (comma-separated):
heb

Registering runner... succeeded                     runner=q8McuSyX

Enter an executor: docker, parallels, virtualbox, docker+machine, kubernetes, custom, shell, ssh, docker-ssh+machine, docker-ssh:
shell

Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!

C

Create Datical Project: HEB_DEMO

Databases: heb_ref, heb_test, heb_prod

Schemas: DATICAL_USER, DATICAL_TRACKING, MFG, SALES

Repo: heb_demo_sql, heb_demo_ddb

Add .gitignore to the project folder.

Logs/
Reports/
Snapshots/
Profiles/
Resources/sql/*.sql.tmp
.auditdb/
/.classpath
/.metadata
/.reporttemplates
/daticaldb.properties
/daticaldb*.log
*.zip

Configure SSH Key in GitLab


https://docs.gitlab.com/ee/ssh/index.html

  1. Login: http://gitlab.com

  2. Select: Avatar (upper right) > Preferences

  3. Select: SSH Keys (left side bar)

  4. Paste public key.
    Title: martha-win
    No expiration date.

Configure DDB Repo in GitLab

  1. Login: http://gitlab.com

  2. Menu>Project>Create New Project

  3. Select: Create Blank Project
    Project name: HEB_DEMO_DDB
    Project URL: https://gitlab.com/mbohls/heb_demo_ddb
    Visibility Level: Private
    Project Configuration: Do NOT initialize repository with a README

  4. Push Datical Project to the Repo

    git config --global user.name "Martha Bohls"
git config --global user.email "mbohls@liquibase.com"

cd existing_folder
git init --initial-branch=main
git remote add origin git@gitlab.com:mbohls/heb_demo_ddb.git
git add .
git commit -m "Initial commit"
git push -u origin main

Configure SQL Repo in GitLab

  1. Login: http://gitlab.com

  2. Menu>Project>Create New Project

  3. Select: Create Blank Project
    Project name: HEB_DEMO_SQL
    Project URL: https://gitlab.com/mbohls/heb_demo_sql
    Visibility Level: Private
    Project Configuration: Do NOT initialize repository with a README

  4. Push Datical Project to the Repo

    git config --global user.name "Martha Bohls"
git config --global user.email "mbohls@liquibase.com"

cd existing_folder
git init --initial-branch=main
git remote add origin git@gitlab.com:mbohls/heb_demo_sql.git
git add .
git commit -m "Initial commit"
git push -u origin main

git branch develop
git checkout develop
git push -u origin develop

Configure Packager

#deployPackager.properties
databaseBackupRestoreLocation=DATICAL_DATA_PUMP_DIR
databaseBackupRestoreMethod=BackupRestoreOracleDatapump
databaseBackupMode=always
packageOrderStrategy = default

ddbScmSystem=git
ddbScmBranch=main
ddbScmPath=.

# Default values
sqlScmSystem=git
# Don't use the SCM name from gitlab for the Path.  The SCM is automatically cloned for the gitlab-ci.yml file and is write protected.
sqlScmPath=../src
sqlScmSQLBaseDir=.

# Pipeline specific values

current.sqlScmBranch=develop
current.sqlScmLastImportID=2f75039f51d34156a1582e754db60a887a6a0c12
validateRollback = none
disableTraceFile = false

Install GitLab Runner for heb.datical.net

https://docs.gitlab.com/runner/install/linux-repository.html 

curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.rpm.sh" | sudo bash

sudo yum install gitlab-runner

# An SSH key is needed for git authentication.
# Copy the SSH private key to /home/gitlab-runner/.ssh

Register the Runner

https://docs.gitlab.com/runner/register/index.html

  1. From the HEB_DEMO_SQL project go to Settings > CI/CD

  2. Expand the Runners section.
    Copy the registration token.

  3. Start a terminal session heb.datical.net. Run the following command.

    sudo gitlab-runner register

Runtime platform                                    arch=amd64 os=linux pid=11051 revision=5316d4ac version=14.6.0
Running in system-mode.

Enter the GitLab instance URL (for example, https://gitlab.com/):
https://gitlab.com

Enter the registration token:
q8McuSyXwSyxC-NVxbU3

Enter a description for the runner:
[ip-172-30-3-12.ec2.internal]: heb.datical.net

Enter tags for the runner (comma-separated):
heb

Registering runner... succeeded                     runner=q8McuSyX

Enter an executor: docker, parallels, virtualbox, docker+machine, kubernetes, custom, shell, ssh, docker-ssh+machine, docker-ssh:
shell

Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!

Create Personal Access Token

  • A Project Access Token might be better, but I don’t think they are support for our level of SaaS.

  • The PAT is needed to download artifacts.

  • To create a personal access token:

    • On the top-right corner, select your avatar.

    • Select Edit Profile

    • On the left sidebar, select Access Tokens.

    • Enter a name.

    • Leave expiry date blank (never expires).

    • Select access token scope: api

    • Select create personal access token.

    • Save token. It cannot be retrieved later.

glpat-J-3kayfVBXvy_YszUsaD

Create the Pipeline

  • Configure variables for the pipeline.

    • Settings >> CI/CD

  • .gitlab-ci.yml file is located at the top level of the SQL Repo on the main branch.

    # Datical DB Sample .gitlab-ci.yml

stages:
  - prep
  - package
  - forecast
  - deploy
  - post
  
prep-phase-job:
  stage: prep
  tags:
    - heb
  only:
    - web
  script:
    - |
      echo "Checking Environment"
      echo  Project directory:  $CI_PROJECT_DIR
      echo Project ID:  $CI_PROJECT_ID
      whoami
      export PATH="$PATH:/opt/datical/DaticalDB/repl"
      hammer show version
    - |
      echo "Getting SRC repo"
      cd ..
      rm src -r -f
      mkdir -p src
      cd src
      git config --global user.name "Martha Bohls"
      git config --global user.email "mbohls@liquibase.com"
      git config --global init.defaultBranch main
      git init
      git remote add origin git@gitlab.com:mbohls/$sql_repo.git
      git fetch origin
      git checkout -b $branch --track origin/$branch
      git branch --set-upstream-to=origin/$branch $branch
    - |
      echo "Getting DDB repo"
      cd ..
      rm ddb -r -f
      mkdir -p ddb
      cd ddb
      git config --global user.name "Martha Bohls"
      git config --global user.email "mbohls@liquibase.com"
      git config --global init.defaultBranch main
      git init
      git remote add origin git@gitlab.com:mbohls/$ddb_repo.git
      git fetch origin      
      git checkout -b main --track origin/main
      git branch --set-upstream-to=origin/main main

packager-job:
  stage: package
  tags:
    - heb
  only:
    refs:
      - web
    variables:
      - $jobtype == "PACKAGER"
  script:
    - |
      echo "Packager Stage"
      echo Project directory: $CI_PROJECT_DIR
      export PATH="$PATH:/opt/datical/DaticalDB/repl:/opt/datical/DaticalDB/instantclient"
      export LD_LIBRARY_PATH=/opt/datical/DaticalDB/instantclient
      # Switch to the DDB directory
      cd ../ddb
      hammer groovy deployPackager.groovy pipeline=$pipeline scm=true labels=$CI_JOB_ID
      zip -r $appname-$CI_JOB_ID.zip * -x "Reports/*" -x "Logs/*" -x "Snapshots/*" -x "Profiles/*" -x "daticaldb.log" -x "deployPackager.properties";
      mv $appname-$CI_JOB_ID.zip $CI_PROJECT_DIR

  artifacts:
    paths:
      - $appname-$CI_JOB_ID.zip

forecast-job:
  stage: forecast
  tags:
    - heb
  only:
    refs:
      - web
    variables:
      - $jobtype == "FORECAST"
  script:
    - |
      echo "Forecast Stage"
      echo Project directory: $CI_PROJECT_DIR
      echo Artifact ID: $packagenumber
      export PATH="$PATH:/opt/datical/DaticalDB/repl"
      export LD_LIBRARY_PATH=/opt/datical/DaticalDB/instantclient
      cd ../ddb
      mkdir -p forecast
      cd forecast
      
      # Get GitLab Artifact
      curl --output $appname-$packagenumber.zip --header "PRIVATE-TOKEN: glpat-J-3kayfVBXvy_YszUsaD" "https://gitlab.com/api/v4/projects/$CI_PROJECT_ID/jobs/$packagenumber/artifacts/$appname-$packagenumber.zip"

      # Unzip and forecast
      unzip $appname-$packagenumber.zip

      if [ -n "${labels}" ]
      then
      hammer forecast $environment --labels="${labels}"
      else
      hammer forecast $environment
      fi

deploy-job:
  stage: deploy
  tags:
    - heb
  only:
    refs:
      - web
    variables:
      - $jobtype == "DEPLOY"
  script:
    - |
      echo "Deploy Stage"
      echo Project directory: $CI_PROJECT_DIR
      echo Artifact ID:  $packagenumber
      export PATH="$PATH:/opt/datical/DaticalDB/repl"
      export LD_LIBRARY_PATH=/opt/datical/DaticalDB/instantclient
      cd ../ddb
      mkdir -p deploy
      cd deploy
      
      # Get GitLab Artifact
      curl --output $appname-$packagenumber.zip --header "PRIVATE-TOKEN: glpat-J-3kayfVBXvy_YszUsaD" "https://gitlab.com/api/v4/projects/$CI_PROJECT_ID/jobs/$packagenumber/artifacts/$appname-$packagenumber.zip"

      # Unzip and deploy
      unzip $appname-$packagenumber.zip
      if [ -n "${labels}" ]
      then
      hammer deploy $environment --labels="${labels}"
      else
      hammer deploy $environment
      fi

post-job:
  stage: post
  tags:
    - heb
  only:
    refs:
      - web

  script:
    - |
      # Generate log files for debugging
      echo "Post Stage"
      echo Project directory: $CI_PROJECT_DIR
      export PATH="$PATH:/opt/datical/DaticalDB/repl:/opt/datical/DaticalDB/instantclient"
      export LD_LIBRARY_PATH=/opt/datical/DaticalDB/instantclient
      # Switch to the DDB directory
      cd ../ddb
      hammer debug export --include="datical.project,changelog.xml,daticaldb*.log,*.html,deployPackager.properties,packager.log" --report=scrubbed_debug_files-$CI_JOB_ID.zip
      mv scrubbed_debug_files-$CI_JOB_ID.zip $CI_PROJECT_DIR

  when: always
  
  artifacts:
    paths:
      - scrubbed_debug_files-$CI_JOB_ID.zip

variables:
#These are runtime variables.  Global variables are set at the project level.
  environment: 
    value: "heb_ref"
    description: "DBDEF name. Valid values are: heb_ref, heb_test, heb_prod"
  jobtype: 
    value: "PACKAGER"
    description: "valid values are: PACKAGER, FORECAST, DEPLOY" 
  packagenumber: 
    value: "changeme"
    description: "FORECAST and DEPLOY JOBS ONLY. Unique number of artifact"
  labels: 
    description: "FORECAST and DEPLOY JOBS ONLY. Label expression"

  • The runner has the “heb” tag, so the jobs in the script must have the “heb” tag. This tells GitLab that the jobs can only run on the Runner heb.datical.net.

  • Manually initialize the git SSH key for both of the Repos. Otherwise, the Prep Job fails.

    git config --global user.name "Martha Bohls"
git config --global user.email "mbohls@liquibase.com"
git config --global init.defaultBranch main
git init
git remote add origin git@gitlab.com:mbohls/$ddb_repo.git

  • Artifacts are associated with the Packager Job ID. To view artifacts

    • CI/CD > Jobs

    • Select the job

    • Select Browse under Job Artifacts in the right-hand panel.

  • l

kl

  • No labels

Copyright © Datical 2012-2020 - Proprietary and Confidential