I'm creating a build script for a database and building the views in alphabetical order. An issue I've run into is that a view is created before a view that is referenced in the definition. The referenced view is created later in the script. Is there a command I can use that would create a view without validating, i.e. ignore the dependencies?
Note: Stored Procedures have the concept of "deferred name resolution" but I don't see a way of using this for views.
To add to #Joe Stefanelli's comment, there are many ways to script SQL objects in the correct order of dependency.
Listing the dependency order from the SSMS UI.
Use sp_depends to find dependency order, or using built-in dependency info tables.
Generate the script using the SSMS Tasks -> Generate Scripts... wizard, which can actually be automated using tools like Scriptio or built-in .NET SMO Library.
Another poor man's solution is just to run the script as many times as you have layers of dependencies. Existing objects would be ignored, and each execution would create more objects that depend on earlier-created objects. I would not recommend this solution if at all possible to find something more direct, as mentioned in the list above.
Related
I'm trying to migrate specific objects from one database to another using sqlpackage.exe /action:Extract and sqlpackage.exe /action:Script. Currently I'm creating the script and filtering the unneeded objects manually, I would like to be able to exclude them all together and to automate the process. So far I didn't find in the documentation any option that does it.Thanks.
There is no way to remove single objects with native functionality. Natively you can remove only specific object types.
You can write your own deployment contributor and then skip whatever objects you need. Here is an example here.
Check Ed Elliot's ready to use contributor with bunch of configuration options (I haven't used it for a while and do not know how does it work with the new versions of SQL Server).
Additionally, in Ed Elliot's blog you can find a lot of useful information.
I am attempting to gain some knowledge and use cases on SSDT Database development and deployments and strugling with some deployment issues.
Specifically when using nested views. For some unknown reason when attempting to deploy / publish the files in the project to a local / live db it seems to mess up the references in the views.
In this project i have the following views (example):
View1
View2
View3
View1 references View2 and View3 is referencing View1.
Building the project works fine. hoewever when i try to publish the database either by generating a dacpac by snapshot and publish it to the database or let Visual Studio generate an update script after (or not) comparing schemas i end up with an update script which tries to create the views in what seems to be the logical order in which they are stored in the project.
In this case View1 -> View2 -> View3. This means the publish fails because of reference issues. It can't create a view if the referenced view does not exist.
I have tried several options by adding the dacpac as reference in the project in various ways (same database, Same Server different Database w/ w/o database parameter) but in many cases i end up with a sql71561 / SQL71508 error which was another PITA to solve.
Online i can't find any good sources which explains how to work around this issue or which explains how this works properly.
Hopefully i can get some help here. If you need extra input from my side or want me to try something let me know.
Issue has been resolved by new insights. When trying to build the demo code to share with SO community i accidentally found the solution because i needed to clean up sensitive data(model) parts. Please let me elaborate on what was the issue.
The solution can be divided into two solutions:
Configuration of Database Project / Solution
the way references work
I'll share some insights on the both matters.
Configuration of Database Project / Solution
The Visual Studio solution contained a single project in which all views were placed. The actual tables and other database items were separated in different Solutions / Project.
Solution1
Project1
View1
View2
View3
Solution2
Project1
Tables
Security
Schemas
Etc...
The views itself contained three-part identifiers [Database].[Schema].[Table/View]. This was both on the items inside the project (views) and on the items outside the project (tables etc.).
By just using that one separate Project with just the views led to missing references. It was not able to find the other views nor the tables (further see references).
One solution to this issue was making sure both the views and the tables refenced are in the same Solution / Project. Even with using three-part identifiers Visual Studio ignores these because of the existence of all items in the same project / solution. It will detect the dependencies this way.
the way references work
The other way to solve it was using references the right way in visual studio. which is the second possible solution.
Considering the earlier example where the views were in a different solution as the other elements led to missing references. However adding a dacpac as a database reference with the setting Same Database led to conflicting references and SQL71508 element already exists in the model. This is true because it exists in the references dacpac and we try to create a new view with the same name referencing itself in the dacpac. This is because it sees the three-part reference as a variable for the dacpac.
When using the dacpac setting for same Server, Different Database it resolves the mixed up references because it sees the three-part identifiers as an external reference and thinks that you creating a local copy of a view which is looking at the external dacpac. in other words it will not detect the nested view because it thinks you referencing a separate database not inside the project.
When building the project this will not lead to errors and deployment will work. however since it thinks you are referencing an external data source (in the form of a dacpac) it does not see the reference to the other local views.
The solution to this (atleast this worked for us) is to use two-part identifiers in our views when we need a local reference to the other views. This way it will look at other files inside the project instead of the referenced dacpac.
Since it will detect the reference to the other local views it will build correctly and detect the dependancies in the views inside the local project. It will then create a good build order for all views.
I guess you could also assign a different Variable name to the referenced Dacpac, use three-part-identifiers all the way but change the ones in the external dacpac to use the newly assigned variable name. We have not tested this (but i will when i get back home tonight).
So in all this was a good learning experience in how Database references work inside Database projects when using partial projects or when you have split up the database into several projects / solutions. Now to understand this Pandora's black box and convert them into a future-proof solution :)
Working with VS2013 and doing my first database project solution. I have three existing databases on the same server. Quotes, Quoting and QuotingUi. Each have a few stored procedures that reference one of the other databases. This is done with database.schema.table (Quotes.dbo.Mytable) syntax and works well in SSMS and all production situations. QuotingUi has a number of views that pull data from Quoting using the same syntax. SPs and EF models have no problem with this.
I have created an empty solution and then added each database to the solution as a project by right clicking on the database in the SQL Object Explorer and Create New Database. Once all three projects are created the solution builds. I do however have warnings on Quotes and Quoting (sps only) that references cannot be resolved to the other database. In QuotingUi I have not only warnings about the sp references but errors (red squigglies) on all the views.
I have tried adding the other projects to each project both as projects (seems it only wants dlls) or databases or both, then rebuild, then close and reopen solution etc. I have set a build dependency for QuoutingUi for the other two projects and a build order that builds them first. No joy.
Have begun Deborah Kurata Pluralsight Course but I do not believe she covers this scenario.
Suggestions welcome.
It sounds like you're still using the existing three part name (e.g. [SameServerDb].[dbo].[Table2]) in your stored procedures and views. You should update it to use the SQLCMD variable name for the reference instead. SQLCMD variables are used so that you could change the referenced database name at deployment time.
This is covered in the help documentation, but here's an example for you. In this case I added a reference to "SameServerDb" as shown below:
Note the "Database Variable" name is $(SameServerDb). Now I just change any reference from [SameServerDb] to [$(SameServerDb)]:
CREATE VIEW [dbo].[View2]
AS SELECT * FROM [$(SameServerDb)].[dbo].[Table2]
For all database projects in your solution, add all their referenced databases: In Solution Explorer window, right click References -> Add database reference, and select the other project. Repeat for all referenced projects. Also, make sure you rebuild all projects.
You can also suppress certain warnings:
http://social.msdn.microsoft.com/Forums/sqlserver/en-US/9b698de1-9f6d-4e51-8c73-93c57355e768/treat-specific-warning-as-error?forum=ssdt
My database has had several successive maintainers over the years and any naming guidelines that may have once been in place have been ignored.
I'd like to rename the stored procedures to a consistent format. Obviously I can rename them from within SQL Server Management Studio, but this will not then update the calls made in the website code behind (C#/ASP.NET).
Is there anything I can do to ensure all calls get updated to the new names, short of searching for every single old procedure name in the code? Does Visual Studio have the ability to refactor such stored procedure names?
NB I do not believe my question to be a duplicate of this question as the latter is solely about renaming within the database.
You could make the change in stages:
Copy of the stored procedures to the new stored procedures under their new name.
Alter the old stored procedures to call the new ones.
Add logging to the old stored procedures when you've changed all the code in the website.
After a while when you're not seeing any calls to the old stored procedures and you're happy you've found all the calls in the web site, you can remove the old stored procedures and logging.
You can move the 'guts' of the SPROC to a new SPROC meeting your new naming conventions, and then leave the original sproc as a shell / wrapper which delegates to the new SPROC.
You can also add an 'audit' table to track when the old wrapper SPROC is called - this way you will know that there are no dependencies on the old SPROC, and the old SPROC can be safely dropped (also, make sure that it isn't just 'your app' using the DB - e.g. cross database joins or other apps)
This has a small performance penalty, and won't really buy you that much (other than being able to 'find' your new SPROCs easier)
You will need to handle this in at least two areas, the application and the database. There could be other areas as well, and you have to be careful not to overlook them.
The Application
A Nice Practice for Future Projects
It helps to abstract your sprocs out. In our apps, we wrap all of our sprocs in a giant class, I can make calls like this:
Dim SomeData as DataTable = Sprocs.sproc_GetSomeData(5)
That way, the code end is nice and encapsulated. I can go into Sprocs.sproc_GetSomeData and tweak the sproc name in just one place, and of course I can right click on the method and do a symbolic rename to fix the method call solution-wide.
Without the Abstraction
Without that abstraction, you can just do Find In Files (Cntl+Shift+F) for the sproc name and then if the results looks right, open the files up and Find/Replace all the occurances.
The Sql Server
Don't Trust View Dependencies
On the SQL server end, theoretically in MSSMS 2008 you can right click on a sproc and select View Dependencies.
That should show you a list of all the places where the sproc is used in the database, however my confidence in this feature is very low. It might be better in SQL 2008, but in previous versions it definitely had problems.
View Dependencies hurt me, and it will take time for that to heal. :)
Wrap It!
You end up having to keep the old sproc around for awhile. This is the major reason why renaming sprocs is a such a project - it can take a month to finally be done with it.
First replace its contents with some simple TSQL that calls the the new sproc with the same parameters, and write some logging so that once some time goes by, you can tell if the old sproc is actually unused.
Finally, when you're sure the old sproc is unused, delete it.
Other Areas?
There could be a lot of other areas as well. Reporting Services springs to mind. SSIS packages. Using the technique of keeping the old sproc around and re-routing to the new one (mentioned above) will help you know if you missed anything, however it won't tell you what you missed. This can lead to much pain!
Good luck!
Short of testing every path in your application to ensure that any calls to the database and the relevant stored procedures have been updated... no.
Use global search and replace (but review each suggested replacement) to try to avoid missing any instances. If you app is well structured then there really should only be 1 place each stored proc is called.
As far as changing your application, I have all my stored procs as settings in the web.config file, so all the names are in one place and can be changed at any time to match changes to the database.
When the application needs to call a stored proc, the name is determined from web.config.
This makes it easier to manage all the potential calls which the application could make to the database services layer.
It will be a bit of a tedious search through your source code and other database objects I'm afraid.
Don't forget SSIS Packages, SQL Agent Jobs, Reporting Services rdl as well as your main application code.
You could use a regular expression like spProc1|spProc2 to search in the source code for all object names at the same time if you have a tool that supports searching through files using regular expressions (I have used RegexBuddy for this in the past)
If you want to just cover the possibility you might have missed the odd one you could leave all the previous stored procedures behind for a month and just have them log a custom SQL trace event with APP_NAME(), SUSER_NAME() and any other info you find helpful then have it call the renamed version. Then set up a trace monitoring this event.
If you use a connection to DB, stored procedures etc, you should create a service class to delegate these methods.
This way when something in your database, SP etc changes, you only have to update your service class, and everything is protected from breaking.
There are tools for VS that can manage changing a name, like refactor, and resharper
I did this and I relied heavily on global search in my source code for stored procedure names and SQL digger to find sql procs that called sql proces.
http://www.sqldigger.com/
SQL Server (as of SQL 2000) poorly understands it own dependencies, so one is left searching the text of the scripts to find dependencies, which could be other stored procs or substrings of dynamic sql.
I would obtain a list of references to a procedure by using the following, because SSMS dependencies doesn't pickup dynamic SQL references or references outside the database.
SELECT OBJECT_NAME(m.object_id), m.*
FROM SYS.SQL_MODULES m
WHERE m.definition LIKE N'%my_sproc_name%'
The SQL needs to be run in every database where there could be references.
syscomments and INFORMATION_SCHEMA.routines have nvarchar(4000) columns. So if "mySprocName" is used at position 3998, it won't be found. syscomments does have multiple lines but ROUTINES truncates. Should you disagree, take it up with gbn.
Based on that list of dependencies, I'd create new stored procedures starting the foundation stored procedures - those with the least dependencies. But I'd mind not to create stored procedures, prefixing the name with "sp_"
Verify the foundation procedures work identically to existing ones
Move to the next level of stored procedures - repeat steps 1-3 as needed till the highest level procedure has been processed.
Test the switch over the application uses to the new procedure - don't wait until the all the procedures are updated to test interaction with the application code. This doesn't need to be done for every stored procedure, but waiting to do this wholesale isn't a great approach either.
Developing in parallel has it's risks too:
Any changes to existing code needs to also be applied to the new code. If possible, work in areas where development is frozen or use a bug fix as an opportunity to migrate to new code rather than apply the patch in two places (while also minimizing downtime for transition).
Use a utility like FileSeek to search the contents inside each and every file in your project folder. Don't trust the windows search - it's slow and user-unfriendly.
So if you had a Stored Procedure named OldSprocOne and want to rename it to SP_NewONe, search all occurrences Of OldSprocOne then search all occurrences of OldSprocOne to see if that name isn't already being used somewhere else and won't cause problems. Then rename each and every occurrence in the code.
This can be very time consuming and repetitive for larger systems.
I would be more concerned about ignoring the names of the procedures and replacing your legacy DAL with Enterprise Library Data Access Block 5
Database Accessors in Enterprise Library 5 DAAB - Database.ExecuteSprocAccessor
Having code that is like
public Contact FetchById(int id)
{
return _database.ExecuteSprocAccessor<Contact>
("FetchContactById", id).SingleOrDefault();
}
Will have atleast a billion times more value than having stored procs with consistent names, especially if the current code passes around DataTables or DataSets ::shudders::
I'me all in favor of refactoring any sort of code.
What you really need here is a method slowly and incrementally renaming your stored procs.
I certainly would not do a global find and replace.
Rather, as you identify small pieces of functionality and understand the relationships between the procs, you can re-factor in small pieces.
Fundamental to this process, though, is source-code control of your database.
If you do not manage changes to your database the same as normal code, you will be in serious trouble.
Have a look at DBSourceTools. http://dbsourcetools.codeplex.com
It's specifically designed to help developers get their databases under source code control.
You need a repeatable method of restoring your database to a specific state - prior to refactoring.
Then re-apply your refactored changes in a controlled way.
Once you have embraced this mindset, this mammoth and error-prone task will become simple.
This is assuming that you use SQL Server 2005 or above. An option that I have used before is to rename the old database object and create a SQL Server Synonym with the old name. This will allow for you to update your objects to whatever convention you choose and replace the refrences in code, SSIS packages, etc... as you come along them. Then you can concentrate updating the references in your code gradually over however maintenance releases you choose (as opposed to breaking them all at once). As you feel that you've found all references you can remove the synonym as the code goes to QA.
We are currently reviewing how we store our database scripts (tables, procs, functions, views, data fixes) in subversion and I was wondering if there is any consensus as to what is the best approach?
Some of the factors we'd need to consider include:
Should we checkin 'Create' scripts or checkin incremental changes with 'Alter' scripts
How do we keep track of the state of the database for a given release
It should be easy to build a database from scratch for any given release version
Should a table exist in the database listing the scripts that have run against it, or the version of the database etc.
Obviously it's a pretty open ended question, so I'm keen to hear what people's experience has taught them.
After a few iterations, the approach we took was roughly like this:
One file per table and per stored procedure. Also separate files for other things like setting up database users, populating look-up tables with their data.
The file for a table starts with the CREATE command and a succession of ALTER commands added as the schema evolves. Each of these commands is bracketed in tests for whether the table or column already exists. This means each script can be run in an up-to-date database and won't change anything. It also means that for any old database, the script updates it to the latest schema. And for an empty database the CREATE script creates the table and the ALTER scripts are all skipped.
We also have a program (written in Python) that scans the directory full of scripts and assembles them in to one big script. It parses the SQL just enough to deduce dependencies between tables (based on foreign-key references) and order them appropriately. The result is a monster SQL script that gets the database up to spec in one go. The script-assembling program also calculates the MD5 hash of the input files, and uses that to update a version number that is written in to a special table in the last script in the list.
Barring accidents, the result is that the database script for a give version of the source code creates the schema this code was designed to interoperate with. It also means that there is a single (somewhat large) SQL script to give to the customer to build new databases or update existing ones. (This was important in this case because there would be many instances of the database, one for each of their customers.)
There is an interesting article at this link:
https://blog.codinghorror.com/get-your-database-under-version-control/
It advocates a baseline 'create' script followed by checking in 'alter' scripts and keeping a version table in the database.
The upgrade script option
Store each change in the database as a separate sql script. Store each group of changes in a numbered folder. Use a script to apply changes a folder at a time and record in the database which folders have been applied.
Pros:
Fully automated, testable upgrade path
Cons:
Hard to see full history of each individual element
Have to build a new database from scratch, going through all the versions
I tend to check in the initial create script. I then have a DbVersion table in my database and my code uses that to upgrade the database on initial connection if necessary. For example, if my database is at version 1 and my code is at version 3, my code will apply the ALTER statements to bring it to version 2, then to version 3. I use a simple fallthrough switch statement for this.
This has the advantage that when you deploy a new version of your application, it will automatically upgrade old databases and you never have to worry about the database being out of sync with the software. It also maintains a very visible change history.
This isn't a good idea for all software, but variations can be applied.
You could get some hints by reading how this is done with Ruby On Rails' migrations.
The best way to understand this is probably to just try it out yourself, and then inspecting the database manually.
Answers to each of your factors:
Store CREATE scripts. If you want to checkout version x.y.z then it'd be nice to simply run your create script to setup the database immediately. You could add ALTER scripts as well to go from the previous version to the next (e.g., you commit version 3 which contains a version 3 CREATE script and a version 2 → 3 alter script).
See the Rails migration solution. Basically they keep the table version number in the database, so you always know.
Use CREATE scripts.
Using version numbers would probably be the most generic solution — script names and paths can change over time.
My two cents!
We create a branch in Subversion and all of the database changes for the next release are scripted out and checked in. All scripts are repeatable so you can run them multiple times without error.
We also link the change scripts to issue items or bug ids so we can hold back a change set if needed. We then have an automated build process that looks at the issue items we are releasing and pulls the change scripts from Subversion and creates a single SQL script file with all of the changes sorted appropriately.
This single file is then used to promote the changes to the Test, QA and Production environments. The automated build process also creates database entries documenting the version (branch plus build id.) We think this is the best approach with enterprise developers. More details on how we do this can be found HERE
The create script option:
Use create scripts that will build you the latest version of the database from scratch, which is empty except the default lookup data.
Use standard version control techniques to store,branch,tag versions and view histories of your objects.
When upgrading a live database (where you don't want to loose data), create a blank second copy of the database at the new version and use a tool like red-gate's link text
Pros:
Changes to files are tracked in a standard source-code like manner
Cons:
Reliance on manual use of a 3rd party tool to do actual upgrades (no/little automation)
Our company checks them in simply because someone decided to put it in some SOX document that we do. It makes no sense to me at all, except possible as a reference document. I can't see a time we'd pull them out and try and use them again, and if we did we'd have to know which one ran first and which one to run after which. Backing up the database is much more important then keeping the Alter scripts.
for every release we need to give one update.sql file which contains all the new table scripts, alter statements, new/modified packages,roles,etc. This file is used to upgrade the database from 1 version to 2.
What ever we include in update.sql file above one all this statements need to go to individual respective files. like alter statement has to go to table as a new column (table script has to be modifed not Alter statement is added after create table script in the file) in the same way new tables, roles etc.
So whenever if user wants to upgrade he will use the first update.sql file to upgrade.
If he want to build from scrach then he will use the build.sql which already having all the above statements, it makes the database in sync.
sriRamulu
Sriramis4u#yahoo.com
In my case, I build a SH script for this work: https://github.com/reduardo7/db-version-updater
How is an open question
In my case I am trying to create something simple that is easy to use for developers and I do it under the following scheme
Things I tested:
File-based script handling in git using GitlabCI
It does not work, collisions are created and the Administration part has to be done by hand in case of disaster and the development part is too complicated
Use of permissions and access via mysql clients
There is no traceability on changes to the database and the transition to production is manual
Use of programs mentioned here
They require uploading the structures and many adaptations and usually you end up with change control just like the word
Repository usage
Could not control the DRP part
I could not properly control the backups
I don't think it is a good idea to have the backups on the same server and you generate high lasgs for the process
This was what worked best
Manage permissions per user and generate traceability of everything that is sent to the database
Multi platform
Use of development-Production-QA database
Always support before each modification
Manage an open repository for change control
Multi-server
Deactivate / Activate access to the web page or App through Endpoints
the initial project is in:
In case the comment manager reads this part, I understand the self-promotion but please just remove this part and leave the rest since I think it complies with the answer to the question reacted in the post ...
https://hub.docker.com/r/arelis/gitdb
I hope this reaches you since I see that several
There is an interesting article with new URL at: https://blog.codinghorror.com/get-your-database-under-version-control/
It a bit old but the concepts are still there. Good Read!