I have objects created in dev DB in snowflake. I have 1 pipe & multiple tasks in dev. how can I deploy the tasks to test/prod DB? is there any way to deploy all tasks in a single deployment?
Related
I have multiple Snowflake accounts, having the DB creations and Stored procedures available on GitLab repo. Is there a right approach available to deploy the same using GitLab-CI where DB deploy versions can also be tracked and DB-RollBack also will be feasible.
As of now I am trying with Python on pipeline to connect snowflake and to execute SQL-Script files, and to rollback as well specific SQL are needed for clean-ups and rollback where on-demand rollback becomes a challenge.
Looking for a right approach or sample to refer.
We have 2 RDS databases on aws, qa and production. What I'm trying to do is copy all the records from our production db to qa. I created a new instance of production from a snapshot and pointed the connection string in my app from the qa instance to the new production instance. However the new production instance does not have the migrations needed by the app to run as we have not deployed those changes to yet.
So basically I have migrations in the qa db I'm looking to copy into the new instance of production so I can test the latest changes with current records.
Is there a way to add the migrations from the qa db into prod snapshot or is there a way to copy records from prod into qa which already has those migrations?
What I tried:
small code change to see if redeploy would run migrations
added empty migration and redeployed
-- the prod db sees the new migration but none of the other ones
A Snapshot of an Amazon RDS database contains all data that was in the source database at the time the snapshot was made.
The Snapshot can be used to create a new Amazon RDS database, which will contain all the data from the Snapshot.
It is not possible to modify a Snapshot after it has been created.
As for "a way to copy records from prod into qa", this is not a native capability of Amazon RDS. You would need to use an SQL client to copy data between databases.
We are currently implementing snowflake and dbt and want to split snowflake databases between dev and prod, so that we have a database to test on before releasing new data models. We are planning to use dbt to create all of our data models going forward. I have a couple questions on the logistics of the workflow:
How do we keep dev and prod in sync? (Or should they be?) I know in snowflake theres a clone feature you can recreate metadata without copying data over. Should we clone our prod database to dev? On a daily basis? What about users that have materialized resources in dev -- they would lose that data.
Should we make it so that deployment to prod was part of the CICD process, and only a fully merged pull request (tested on snowflake dev) can be deployed to the snowflake prod? Would that present too much of a bottle neck?
Curious to understand how people have architected their workflows maintaining both a dev and prod snowflake environment.
A common implementation is to have user-specific dev schemas (e.g., dbt_lfolsom) that are written to and overwritten whenever a user executes any kind of dbt run; and then a single prod schema (e.g., analytics) that's written to and overwritten when jobs are executed "in production."
Running dbt "in development" means a dbt command is executed by an individual user (using dbt Cloud or dbt CLI) so that data is written to a dedicated schema that is used specifically for development.
Running dbt "in production" means running dbt commands that are configured as jobs in dbt Cloud (or using another orchestration tool) write data to a specific "prod" target schema.
Check out this article on running dbt in production.
With dbt Cloud, you can also write to PR-specific schemas that are created automatically when you create or update a GitHub PR, which may be what you'd like to use for CI/CD. Check out this article on options.
You can (should) definitely configure prod jobs to run only on fully merged code.
If for some reason you really need prod and dev to be separate databases instead of separate schemas, I think you would create separate dbt projects that use code from the same git remote repo but are configured (in dbt Cloud or using your profiles.yml) to write to different Snowflake databases. But I think that's an unconventional approach that would require more work.
To change the database where you run your code go to
top right corner -> your profile -> in credentials on the left choose the project, fill in the dev section (here you select only schema) and above it you can overwrite the default for this project.
This default will be changed only for you and will be indicated by a strikethrough over the default one and your selection next to it
I'm trying to build a CI/CD for my Microsoft SQL Server database projects. It will work with Microsoft DevOps pipelines.
I have all databases in Visual Studio databases projects with the GIT as source control. My objective is to have something that I can release databases with the use of DevOps pipelines to the diferents enviroments:
DEV
UAT
PROD
I was thinking of using DBGhost: http://www.innovartis.co.uk/ but I can't find updated information about this tool (only very old info) and there is very little information about it on the internet and how to use it (is it still in use?).
I would like to use a mix of DBGhost and DevOps. DBGhost to Source Scripting, Building, Comparing, Synchronizing, Creating Delta Scripts, Upgrading and DevOps to make releases (that would call the builds created by DBGhost)
If you have any ideas using this or other methods I am grateful because currently all releases are manual and it is not very advisable to do.
We have this configured in our environment using just DevOps. Our database is in a Visual Studio database project. The MSBuild task builds the project and generates a DACPAC file as an artifact, and the Release uses the "SQL Server Database Deploy" task to deploy this to the database. The deploy task needs to use an account with enough privileges to create the database, logins, etc., but takes care of performing the schema compare, generating the delta scripts, and executing them. If your deploy is going to make changes that could result in data loss such as removing columns, you will need to include the additional argument /p:BlockOnPossibleDataLoss=false in the deploy task. This flag is not recommended unless you know there will be changes that will cause data loss; without the flag any deploy which would result in data lost will fail.
In our company we have database solution that contains three SQL Server instances each with different databases. Each instance has some jobs and replication.
As for now we are maintaining creation and update scripts manually and execute them with bat files.
Our deployment package contains scripts for all objects including jobs and replication.
We want to automate our process to make and test deployment packages after every svn commit - continuous integration. Also we have branches for every release. Release correspond to a database version. Different clients have different releases/versions installed. We need to create deployment package for any branch.
Can we use SQL Server Data Tools 2012 for our needs? I have only seen tutorials for single database and I don't know how to use it in more complex environment.
Optionally we could use Data Tools for maintaining schema scripts and write manually scripts for jobs/replication. But can we use the build process to combine it all into one package?
You should be able to use SSDT for this, by way of Publish Profiles. Create a publish profile for each instance and set up your CI jobs accordingly.
Standardizing your database names across instances (especially if they're all for the same product) would help.