"Scrub" Sensitive Data from your Files before sending them to Datical/Liquibase Support

Liquibase Business or Liquibase Enterprise/Datical DB customers:  Providing your log files, project files, changelogs, and html reports can help Datical Support our support team diagnose and resolve an issue more quickly.  However, some of your Datical files may contain sensitive information. 

For Liquibase Business or Liquibase Enterprise/Datical DB versions 5.9 and higher, you can use the hammer debug export command to "scrub" sensitive information such as hosts, ports, and credentials from your files before sending them to Datical Supportour support team.  The hammer debug export command will create a new zip file in your <ProjectName>/Reports/debug directory that contains the files you specify to include. The files in the zip will have the actual values for hosts, ports, and credentials replaced usernames and password hashes (password hashes are only relevant if you use the less secure Stored Credentials option) replaced by the scrubber with more generic strings (tokens).  This is to avoid emailing files that contain the actual values which as they may be sensitive information.

The hammer debug export command provides a simple way to archive the needed files into a zip file that you then send to Datical Supportour support team:

• You can specify which files to include in the zip. 
• The command must be run from the command line (not from the Datical desktop client/GUI).
• The "scrubber" will only handle files that are located somewhere within your <ProjectName> directory structure for that project.  If there are relevant files located outside of the project's directory structure (for example the daticaldb.log file for operations performed in the Datical desktop client/Eclipse GUI), you can copy that log into the project directory structure prior to running the hammer debug export command.
• You can add a new step near the end of your automation jobs so that the hammer debug export command creates the zip of "scrubbed" files automatically for each job.

Command: hammer debug export

The command creates a zip file of files in a project directory to send to Datical/Liquibase Support or for your teams to see more details about a particular operation. 

Options provide control over what files are included in the zip.  The files are "scrubbed" to remove certain sensitive data by default.

Project Directory

The project directory is the source of all files needed. 

Output Location

By default the command zips the files and places the zip archive file in the project directory:

<project><ProjectName>/Reports/debug/<project>.zip

Use the --report option to specify a different file name and/or location. 

...

Which sensitive data is Scrubbed/Replaced/Redacted

By default, the hammer debug export  command ensures that sensitive information is redacted. Values are replaced with tokens. Each token is a more generic string that uniquely replaces the actual values for a particular item of information.

...

• hostname or IP address
• port
• database names
  • Oracle: service name, TNS name, SID
  • Other database:  database name
• user names
• passwords (the encoded "password hashes" in the datical.project file, IF you use the less secure Stored Credentials option with Liquibase Business or Liquibase Enterprise/Datical DB)

Including and Excluding Files

Include (–include) or exclude (–exclude) files. The value for either option is a search string. All files with names and/or extensions that match the string search are included.  

All files must be in the project file hierarchy. If you want to add a file from outside the project, copy it to the file hierarchy. Place it in the root <project-name> directory<ProjectName> directory, or in the <ProjectName>/Log subdirectory, or other subdirectory.  

Including Files by Last Modified Date

You can include only files modified within a specified number of hours.    This is optional.

Use the --lastmodified option to specify a number of hours.  All files are included that have a last modified date within the number of hours measured from the time the command is run. 

Including Non-Project Files

If Datical Support asks you to include files that are not already part of the Datical project directory, you must place them in the project directory structure before running the hammer debug export command:

• If you were running an operation in the Datical desktop client/Eclipse GUI and got an error, that error would be logged in the daticaldb.log file in your Datical installation directory (such as "DaticalDB" or "DaticalDB_6.12").  In that scenario you would need to copy the log file from your installation directory to your project directory structure prior to running the hammer debug export "scrubber" command. 

...

• If you use automation software like Jenkins to run packager or deploys, Datical/Liquibase Support may ask you for the packager.log or daticaldb.log from the automation job. You must copy them into

...

• your local project directory before running the hammer debug export command.  To avoid needing to do this, you can add a step near the end of your automation jobs to run the hammer debug export command so the zip of scrubbed files will automatically be created for each job.

Other command options

All of the hammer debug export command options are shown here: CLI Commands#debugexport

Common Use Case Scenarios

Datical has found patterns in customer needs, customer security policies, and requirements for files to fully diagnose a problem.  Here are four some example scenarios.: 

• Minimum Standard for Problem Diagnosis - provide the minimum number of files to start diagnosis.standard files most commonly used for troubleshooting
• Default - provide all data in the project directoryExclude All SQL Files - provide all files except SQL files. Security policy may determine whether you are allowed to share SQL files outside of your company.  if no command options are specified, the command will copy all of the files in the project directory into the debug zip file
• Exclude DML Files - provide all files except DML files. Security policy may determine whether you are allowed to share DML files. DML files may contain sensitive information about people and accounts. 

Scenarios

...

Standard for Problem Diagnosis

Scrub and provide the minimum standard information required for Datical/Liquibase Support to start diagnosing a problem. 

• datical.project - definition of the project, contains connection information (includes encoded "password hashes" IF you use the less secure Stored Credentials option with Liquibase Business or Liquibase Enterprise/Datical DB)
• changelog.xml - all changesets (changes applied to the database)the changelog lists all of the change sets 
• daticaldb.log - main Datical DB log (use an asterisk where a date may be within the filename, daticaldb*.log) 
• packager.log - log from running the Deployment Packager, which refactors SQL files into changesets and deploys them to the REF database

...

• html reports - forecastReport.html, deployReport.html, packagerReport.html (use an asterisk to wildcard all html reports, *.html)

Example of Standard

hammer debug export --include=datical.project,changelog.xml,daticaldb*.log,packager.log,*.html

Using --report

You can use the --report option to specify the alternative path and/or filename for the zip archive produced by this command.  This is optional.  

Example of --report to go to a folder called tmp

hammer debug export --include="datical.project,changelog.xml,daticaldb*.log,*.html,deployPackager.properties,packager.log" --report=/tmp

If you already use a BUILD_NUMBER variable in your automation configuration, you could include the build number as part of the filename of the zip file created by the hammer debug export command.

Example of --report with BUILD_NUMBER

hammer debug export --include="datical.project,changelog.xml,daticaldb*.log,*.html,deployPackager.properties,packager.log" --report=scrubbed_debug_files_${BUILD_NUMBER}.zip

Default (no command options specified)

Include and scrub all files in the project directory.  The zip created may be quite large because it would include all of the sql scripts and all other files.

Example of Default

hammer debug export

Exclude DML

Scrub and include almost all of the files in from the project directory structure, except html reports and DML files and generated SQL in the html reports.in the Resources folders.  (Note that this example is from a single schema project.  The Resources paths may be different for multi-schema projects.)

Example of Excluding DML

hammer debug export --exclude=*.html,Resources/sql,Resources/sql_scripts

PATH

If you are running the hammer debug export command from within your project directory and there is an error that hammer is not found, you may need to add your repl folder to your path.  For example:

C:\Users\John\DaticalDB_6.15\repl