The premise of this document is to provide reader with ideas on how to manage a growing number of reference databases as teams adopt Datical.
This document makes following assumptions:
- Each Datical project will have multiple pipelines. As such, each pipeline will require its own database (SQL Server) or schema(s) (Oracle)
- Others?
Reference Databases for SQL Server
During early phase of Datical adoption, teams typically setup Reference Databases on their own, i.e., they may not be using centralized infrastructure from day 1. This allows teams flexibility to manage and configure their Reference Databases as well as underlying servers.
However, as Datical usage grows, the approach above would create a sprawl of Reference Databases.
To better manage demand, it would be ideal to move Reference Databases to centralized/shared infrastructure. Reference Databases are typically structure-only databases without any significant amount of data (you may still need lookup data or test data to validate objects), one could put large number of Reference Databases on single SQL Server instance. The number of Reference Database per instance would be dictated by the size of each Reference Database (size depends on number of objects in the database and the amount of data needed in the database).
How many Reference Databases can you put on a single SQL Server instance?