How To: Backfill your pipeline

This guide is applicable when

• When you have multiple pipelines in your Datical project
• You have deployed on one pipeline (for example Oct release pipeline) and ready to backfill your deployments in the other pipeline (for example Nov release pipeline)
• You bypassed your deployments to rush something to production and now you need to deploy your changes to lower environments to keep them in sync

Overview

A backfill is really two operations in one:

1. First: You add the label to your changesets to indicate that your changes are eligible for deployment in the new pipeline. For example, if you used "oct_release" label to deploy October changes and now you want to backfill these changes to your November pipeline, you would add "nov_release" label to your October changesets
2. Second: You perform a deploy operation on November pipeline using the "nov_release" label. This way, all your October changes will be deployed into November pipeline.

We will assume that you have a Datical project setup with two pipelines that look like the diagram below. You used "Pipeline1" to deploy October changes using the label "oct_release". Your October changes are now in UAT. You are now ready to start working on November code and in preparation for deploying November changes into "Pipeline2" you now need to prepare REF2, DEV2 and QA2 databases with October changes. 

Step-by-step guide (Using automation via "hammer" CLI)

Using Datical DB's command line tool "hammer", follow these steps to implement backfill operation, i.e., deploy October changes into "Pipeline 2":

1. CLI - Add Labels: You will first add November label "nov_release" to all October changes
   1. use "hammer addLabels" command to add labels to October changes

      1. Here is the help for "hammer addLabels" command

         addLabels help Expand source

         USAGE:
           addLabels --lookupChangesets=<label expression> --labels=<comma separated list of labels to add> --scriptChecksum=<SQL script checksum value>
                     --matchAll=<true/false>Options:
           lookupChangesets    - Expression used to match the change sets (required)
           labels              - Comma-separated list of labels (required)
           scriptChecksum      - SQL script checksum value
           matchAll            - SQL Script checksums must match for change set lookup. Default value is true

   2. For our example, use this command to add "nov_release" label to October changes:
      1. hammer addLabels --lookupChangesets="oct_release" --labels="nov_release"
      2. Successful completion of this command generates the following output:
      3. addLabels Expand source

         C:\Users\Datical\datical\DDBProject>hammer addLabels --lookupChangesets="oct_release" --labels="nov_release"
         Updating labels for change set j6fm-1 (createTable tableName=AAA_TESTA1)
         Updating labels for change set jvrp-1 (createTable tableName=AAA_TESTA2)
         Updating labels for change set 20170714131141429_Insert_dml_AAA_TESTA1
         Updating labels for change set 20170714131141461_Insert_dml_AAA_TESTA2
         Updating labels for change set AutoPermissions_PPADM_Tables
         
         
         C:\Users\Datical\datical\DDBProject>

   3. You have now successfully added "nov_release" label to all October changes.

2. CLI - Deploy: Now deploy October changes into Pipeline 2 using "hammer" command
   1. You can now Forecast on REF2. 
      
      1. Use "hammer forecast" command. Here is the help for "hammer forecast" command
         1. forecast help Expand source

            C:\Users\Datical\datical\DDBProject>hammer help forecast
            forecast - Simulates all change sets that have not been applied to the indicated database reference.
            
            USAGE:
              forecast <dbref> [--log=/path/to/log] [--report=/path/to/report] [--context=context1,context2]  [--labels=<label expression>]
            
            EXAMPLE:
              forecast myAppDevDb --log=/home/user/logs/ --report=/home/user/reports --context=dev,qa --labels="JUN AND (poolApp or beachApp)"
            OPTIONAL PARAMETERS:
              --log - Location to write the daticaldb.log file. Log is written to 'Logs' directory of project by default.
              --report - Location to write the forecast report. Report is written to the 'Reports' directory by default.
              --context - Only change sets marked with the contexts specified will be executed.  To run all contexts, specify $all. 
              --labels - When set, only change sets marked with the label expression will be executed.  To run all labels, specify $all.
            
            C:\Users\Datical\datical\DDBProject> 

      2. For our example, use this command to forecast "nov_release" label to October changes:

         1. hammer forecast REF2 --labels="nov_release"
         2. Successful completion of this command generates the following output:
            1. forecast Expand source

               C:\Users\Datical\datical\DDBProject>hammer forecast REF2 --labels="nov_release"
               There were no rules files found.
               Defaulting contexts to REF2 from REF2
               Defaulting to database definition contexts of 'REF2'
               There are 5 change sets forecast to be deployed using labels 'nov_release' and contexts 'ref2'
               
               Report ready at C:\Users\Datical\datical\DDBProject\Reports\2017\07-Jul\REF2\forecast_REF2_20170724_152507\forecastReport.html
               
               C:\Users\Datical\datical\DDBProject>

         3. Review the forecast report. 
   2. Since backfill using CLI is typically performed by an automation tool, it is a good practice to perform "statusDetails" both before and after the deploy operation in order to track pre-deploy and post-deploy status of your database.
      1. Use "hammer statusDetails REF2" command to obtain status of the REF2 database prior to the deploy operation
      2. Here is a sample output of "hammer statusDetails" command:

      3. 

   3. Use "hammer deploy" command. Here is the help for "hammer forecast" command
      1. deploy help Expand source

         C:\Users\Datical\datical\DDBProject>hammer help deploy
         The deploy command deploys all change sets that have not been applied to the indicated database reference in the project.
         USAGE:
           deploy <dbref> [--pipeline=<pipeline name>] [--log=/path/to/log] [--report=/path/to/report] [--context=context1,context2] [--labels=<label expression>]
         
         EXAMPLE:
           deploy myAppDevDb --pipeline=DEV --log=/home/user/daticallogs/ --report=/home/user/daticalreports --context=dev,integration --labels="JUN AND (poolApp or beachApp)"
         OPTIONAL PARAMETERS:
           --log - When set, the daticaldb.log file will be written to the directory specified.  Log is written to 'Logs' directory of project by default.
           --pipeline - When set, this is the pipeline which will be associated with the deployment for auditing purposes.
                        If not set, and the dbDef is a member of one and only one pipeline, then that pipeline will be used.
                        If not set, and the dbDef is a member of multiple pipelines, then an error will be displayed.
           --report - When set, deploy report will be written to the directory specified.  Report is written to the 'Reports' directory by default.
           --context - When set, only change sets marked with the contexts specified will be executed.  To run all contexts, specify $all.
           --labels - When set, only change sets marked with the label expression will be executed.  To run all labels, specify $all.
         
         C:\Users\Datical\datical\DDBProject> 

      2. For our example, use this command to forecast "nov_release" label to October changes:

         1. hammer deploy REF2 --labels="nov_release"

      3. Review the deploy report for any errors during deployment
   4. Perform "hammer statusDetails" again after successful deployment:
      1. Use "hammer statusDetails REF2" command to obtain status of the REF2 database after the deploy operation
   5. You have now successfully backfilled October changes into REF2 database. 
   6. Repeat steps 2a-2d to backfill DEV2 and QA2 databases.

Step-by-step guide (Using Datical DB GUI)

Using Datical GUI follow these steps to implement backfill operation, i.e., deploy October changes into "Pipeline 2":

GUI - Add Label: You will first add November label "nov_release" to all October changes

Click on the "Status" button on your REF1. 

Click on "Fully Deployed" link (or "Undeploy Changes" depending on your database status)

Your list of changes may look something like this where you have "oct_release" label tied to several change sets:

Use the search box at the top. Type "oct_release" and click Filter by Label button to get a list of only Octobers:

Now click "Select All (on this tab) button at the bottom to select all October changes, and then click Add Labels button:

In the "Ad Hoc Labels to Add" field, type "nov_release". 

1. Alternately, you may have the nov_release label listed in the "Available" column if you setup your project settings with these labels.
2. 

Click Next.

Review your "Labels Before" and "Labels After" screen and click Finish. 

You have now successfully added "nov_release" label to all October changes. 

GUI - Deploy: Go back to the Plan tab and switch over to the Pipeline 2.

You can now Forecast on REF2. Click the Forecast button (or "Forecast with Options" link) to bring up Forecast Options window. 

1. Check the box for "Use Labels"
2. Specify "nov_release" label 
3. Click Finish.
4. 
5. 

If the Forecast report shows no errors, then click on the Deploy button for REF2 (or "Deploy with Options" link) to bring up Deploy Options window. 

1. Check the box for "Use Labels"
2. Specify "nov_release" label and click Finish.
3. 

You have now successfully backfilled October changes into REF2 database. 

Repeat steps 2a-2b to backfill DEV2 and QA2 databases.

Related articles