I have a fairly complex setup with eight databases on a server each referencing each other (about every database referencing each other), giving way to quite a complex web. The design is far from ideal, but unfortunately this is something we have to work with.
We need to create a SSDT solution to facilitate CI/CD
The whole project needs to be deployed from scratch on a new instance and I am trying to get my head around this, as I have limited SSDT knowledge for a project this scale.
The approaches I consider are as follows:
1) Split objects into shared objects, and reference the shared objects. This seems to be a nightmare to implement, as we would require different layers because of the complex web of references. (shared object referencing other shared objects). Also how do we deploy such a project on a blank server?
2) Create stubs for each object in a project being referenced by other objects, and make a database reference to these. This seems to be the easiest option, although it seems that if the object the stub is based on gets changed, the stubs also needs to be maintained otherwise the project will break. Is this the right assumption?
3) Only create stubs for projects required to compile (eg. tables referenced by views in other databases), and ignore warning references. I am leaning towards this route as the stubs will be much smaller and project easier to maintain, but I hate to ignore referencewarnings..
If we deploy using the stubs option, do we need to deploy the stubs first and then delete them after successful deployment?
Another (more straightforward question). What is the best way to deploy logins, users and object permissions ?
Thanks for replying.
The question is too broad but these are few suggestions:
You can't do anything with circular reference. There are some ways to workaround it but all of them are "hacky" and most probably will introduce more problems than to solve your problem. So try to move objects in so manner that there is only one way dependency;
Use synonyms for ALL cross database objects, so there supposed to be no straight reference outside database;
I agree with Peter Schott that it is better to ignore logins and users for now as handling them in SSDT is a bit of pain and you need to have good expertise on SSDT to make it working properly.
I am attempting to gain some knowledge and use cases on SSDT Database development and deployments and strugling with some deployment issues.
Specifically when using nested views. For some unknown reason when attempting to deploy / publish the files in the project to a local / live db it seems to mess up the references in the views.
In this project i have the following views (example):
View1
View2
View3
View1 references View2 and View3 is referencing View1.
Building the project works fine. hoewever when i try to publish the database either by generating a dacpac by snapshot and publish it to the database or let Visual Studio generate an update script after (or not) comparing schemas i end up with an update script which tries to create the views in what seems to be the logical order in which they are stored in the project.
In this case View1 -> View2 -> View3. This means the publish fails because of reference issues. It can't create a view if the referenced view does not exist.
I have tried several options by adding the dacpac as reference in the project in various ways (same database, Same Server different Database w/ w/o database parameter) but in many cases i end up with a sql71561 / SQL71508 error which was another PITA to solve.
Online i can't find any good sources which explains how to work around this issue or which explains how this works properly.
Hopefully i can get some help here. If you need extra input from my side or want me to try something let me know.
Issue has been resolved by new insights. When trying to build the demo code to share with SO community i accidentally found the solution because i needed to clean up sensitive data(model) parts. Please let me elaborate on what was the issue.
The solution can be divided into two solutions:
Configuration of Database Project / Solution
the way references work
I'll share some insights on the both matters.
Configuration of Database Project / Solution
The Visual Studio solution contained a single project in which all views were placed. The actual tables and other database items were separated in different Solutions / Project.
Solution1
Project1
View1
View2
View3
Solution2
Project1
Tables
Security
Schemas
Etc...
The views itself contained three-part identifiers [Database].[Schema].[Table/View]. This was both on the items inside the project (views) and on the items outside the project (tables etc.).
By just using that one separate Project with just the views led to missing references. It was not able to find the other views nor the tables (further see references).
One solution to this issue was making sure both the views and the tables refenced are in the same Solution / Project. Even with using three-part identifiers Visual Studio ignores these because of the existence of all items in the same project / solution. It will detect the dependencies this way.
the way references work
The other way to solve it was using references the right way in visual studio. which is the second possible solution.
Considering the earlier example where the views were in a different solution as the other elements led to missing references. However adding a dacpac as a database reference with the setting Same Database led to conflicting references and SQL71508 element already exists in the model. This is true because it exists in the references dacpac and we try to create a new view with the same name referencing itself in the dacpac. This is because it sees the three-part reference as a variable for the dacpac.
When using the dacpac setting for same Server, Different Database it resolves the mixed up references because it sees the three-part identifiers as an external reference and thinks that you creating a local copy of a view which is looking at the external dacpac. in other words it will not detect the nested view because it thinks you referencing a separate database not inside the project.
When building the project this will not lead to errors and deployment will work. however since it thinks you are referencing an external data source (in the form of a dacpac) it does not see the reference to the other local views.
The solution to this (atleast this worked for us) is to use two-part identifiers in our views when we need a local reference to the other views. This way it will look at other files inside the project instead of the referenced dacpac.
Since it will detect the reference to the other local views it will build correctly and detect the dependancies in the views inside the local project. It will then create a good build order for all views.
I guess you could also assign a different Variable name to the referenced Dacpac, use three-part-identifiers all the way but change the ones in the external dacpac to use the newly assigned variable name. We have not tested this (but i will when i get back home tonight).
So in all this was a good learning experience in how Database references work inside Database projects when using partial projects or when you have split up the database into several projects / solutions. Now to understand this Pandora's black box and convert them into a future-proof solution :)
I have a database called CommonDB. I have created shared data sets from this database in one of my report projects. I now have a need to include the same shared data sets in another report project. Ideally it would be nice if I could just point it to the testing site in BIDS and develop my report based on a reference.
I was wondering if there is a way to do this without adding existing data set (as I was hoping to keep the code base the same so I wouldn't have to update it in different projects).I am aware you can also add existing data sets from url but that defeats the purpose as it just downloads a copy to my report solution and it's not synced.
Any ideas? Thanks :)
This scenario is not supported by BIDS/Visual Studio.
It gets worse: If you deploy multiple projects to the same server, using the same Shared Datasets folder, then at runtime they will try to reuse each-others Shared Dataset definitions. The latest deployed Shared Dataset definition will win.
Data Sources have similar issues - although they tend to be less volatile.
To avoid this, I prefer to keep all reports for a single SSRS server instance in a single BIDS/Visual Studio Project. I use Project Configurations to manage deployment of reports to disparate folders.
We have two databases, A and B. A contains tables that should also be deployed to B; however, A should always be considered as the master of those tables. We don't want to duplicate the schema object scripts. We do not want to simply reference A's table from B - they need to be separate, duplicated tables.
As far as I can see, there are two ways to achieve this:
Partial projects: export the shared schema objects to a partial project (.files) file, and import it into the B's database project
Adding shared schema object files to the B's database project as links.
These both have the disadvantage that you need to explicitly specify files - you cannot specify a folder, meaning that any time a schema object that needs sharing is added to A's database project, then either the partial project export would need to be run again, or the new file added as a link to B's project.
What are the advantages and disadvantages of these techniques? Are there any better ways of achieving this that I may have missed? Thanks.
Partial Projects are not supported in the VS2012 RC release, which makes me think that I shouldn't use them. Furthermore, it appears that Composite Projects within SQL Server Data Tools (SSDT) may be the long term solution.
I've discovered that it is possible to link to entire folders by editing the dbproj file manually. For example:
<Build Include="..\SourceDatabase\Schema Objects\Tables\*.sql">
<SubType>Code</SubType>
<Link>Schema Objects\Tables\%(FileName).sql</Link>
</Build>
This works quite nicely, so it will be our preferred solution until we evaluate SSDT. The main drawback that I've found with it so far is that it will include all files in the source database's folders, including those that are not included in the source database project.
I'm considering using Django for a project I'm starting (fyi, a browser-based game) and one of the features I'm liking the most is using syncdb to automatically create the database tables based on the Django models I define (a feature that I can't seem to find in any other framework).
I was already thinking this was too good to be true when I saw this in the documentation:
Syncdb will not alter existing tables
syncdb will only create tables for models which have not yet been installed. It will never issue ALTER TABLE statements to match changes made to a model class after installation. Changes to model classes and database schemas often involve some form of ambiguity and, in those cases, Django would have to guess at the correct changes to make. There is a risk that critical data would be lost in the process.
If you have made changes to a model and wish to alter the database tables to match, use the sql command to display the new SQL structure and compare that to your existing table schema to work out the changes.
It seems that altering existing tables will have to be done "by hand".
What I would like to know is the best way to do this. Two solutions come to mind:
As the documentation suggests, make the changes manually in the DB;
Do a backup of the database, wipe it, create the tables again (with syncdb, since now it's creating the tables from scratch) and import the backed-up data (this might take too long if the database is big)
Any ideas?
Manually doing the SQL changes and dump/reload are both options, but you may also want to check out some of the schema-evolution packages for Django. The most mature options are django-evolution and South.
EDIT: And hey, here comes dmigrations.
UPDATE: Since this answer was originally written, django-evolution and dmigrations have both ceased active development and South has become the de-facto standard for schema migration in Django. Parts of South may even be integrated into Django within the next release or two.
UPDATE: A schema-migrations framework based on South (and authored by Andrew Godwin, author of South) is included in Django 1.7+.
As noted in other answers to the same topic, be sure to watch the DjangoCon 2008 Schema Evolution Panel on YouTube.
Also, two new projects on the map: Simplemigrations and Migratory.
One good way to do this is via fixtures, particularly the initial_data fixtures.
A fixture is a collection of files that contain the serialized contents of the database. So it's like having a backup of the database but as it's something Django is aware of it's easier to use and will have additional benefits when you come to do things like unit testing.
You can create a fixture from the data currently in your DB using django-admin.py dumpdata. By default the data is in JSON format, but other options such as XML are available. A good place to store fixtures is a fixtures sub-directory of your application directories.
You can load a fixure using django-admin.py loaddata but more significantly, if your fixture has a name like initial_data.json it will be automatically loaded when you do a syncdb, saving the trouble of importing it yourself.
Another benefit is that when you run manage.py test to run your Unit Tests the temporary test database will also have the Initial Data Fixture loaded.
Of course, this will work when when you're adding attributes to models and columns to the DB. If you drop a column from the Database you'll need to update your fixture to remove the data for that column which might not be straightforward.
This works best when doing lots of little database changes during development. For updating production DBs a manually generated SQL script can often work best.
I've been using django-evolution. Caveats include:
Its automatic suggestions have been uniformly rotten; and
Its fingerprint function returns different values for the same database on different platforms.
That said, I find the custom schema_evolution.py approach handy. To work around the fingerprint problem, I suggest code like:
BEFORE = 'fv1:-436177719' # first fingerprint
BEFORE64 = 'fv1:-108578349625146375' # same, but on 64-bit Linux
AFTER = 'fv1:-2132605944'
AFTER64 = 'fv1:-3559032165562222486'
fingerprints = [
BEFORE, AFTER,
BEFORE64, AFTER64,
]
CHANGESQL = """
/* put your SQL code to make the changes here */
"""
evolutions = [
((BEFORE, AFTER), CHANGESQL),
((BEFORE64, AFTER64), CHANGESQL)
]
If I had more fingerprints and changes, I'd re-factor it. Until then, making it cleaner would be stealing development time from something else.
EDIT: Given that I'm manually constructing my changes anyway, I'll try dmigrations next time.
django-command-extensions is a django library that gives some extra commands to manage.py. One of them is sqldiff, which should give you the sql needed to update to your new model. It is, however, listed as 'very experimental'.
So far in my company we have used the manual approach. What works best for you depends very much on your development style.
We generally have not so many schema changes in production systems and somewhat formalized rollouts from development to production servers. Whenever we roll out (10-20 times a year) we do a fill diff of the current and the upcoming production branch reviewing all the code and noting what has to be changed on the production server. The required changes might be additional dependencies, changes to the settings file and changes to the database.
This works very well for us. Having it all automated is a niche vision but to difficult for us - maybe we could manage migrations but we still would need to handle additional library, server, whatever dependencies.
Django 1.7 (currently in development) is adding native support for schema migration with manage.py migrate and manage.py makemigrations (migrate deprecates syncdb).