I'm trying to migrate specific objects from one database to another using sqlpackage.exe /action:Extract and sqlpackage.exe /action:Script. Currently I'm creating the script and filtering the unneeded objects manually, I would like to be able to exclude them all together and to automate the process. So far I didn't find in the documentation any option that does it.Thanks.
There is no way to remove single objects with native functionality. Natively you can remove only specific object types.
You can write your own deployment contributor and then skip whatever objects you need. Here is an example here.
Check Ed Elliot's ready to use contributor with bunch of configuration options (I haven't used it for a while and do not know how does it work with the new versions of SQL Server).
Additionally, in Ed Elliot's blog you can find a lot of useful information.
Related
During an upgrade process from 2009 to 2016 I'm trying to remove objects relating to an old discontinued product. The objects are not within the range of or license and consists of both Forms, Tables and Reports. When deleting I'm faced with the well known error:
"You do not have permission to delete the '[object name]' Table."
I've tried with my developers license and the customers license with no luck. Since the product is no longer existing there is no use keeping these objects around and I need them gone for the upgrade process.
What is the best approach or technique when deleting objects that's not in the license?
UPDATE: How this issue was resolved?
I got in contact with the product owner and explained my problem. They sent me a neat PowerShell script to run. This worked like a charm. Reading through the script I can see that it's using the SQL cmdlets to select and delete relevant data from the following SQL tables:
Objects, Object Metadata, Object Metadata Snapshot, Object Tracking, Object Translation, Permission.
This was the preferred method of the product owner who used to develop this product. It should be applicable to all NAV objects. I have not yet successfully tried one of the answers below (more tries to come). Hopefully this new information will provide someone with enough knowledge to provide a good answer.
The way which was successfully used by several people but for sure cannot be recommended for production system is to simply delete these objects via SQL from Object and supplemental tables. In case of tables, you would need to manually delete the SQL table itself as well as its VSIFT views.
A bit more better (probably) way is to change the number of the object via SQL and then delete the object via NAV.
The best way is to use the functionality of "killer objects" - which allow to delete objects via FOB import:
http://navisionary.com/2011/11/how-to-delete-bsolete-dynamics-nav-objects/
If you find the partner who can provide you with such killer objects (they need to have a license to create objects in needed range), it solves you problem in a "clean" way.
If not, you may want to consider creating empty objects in 50000 range in some test DB, changing their number to obsolete range via SQL, exporting them as FOB, and then importing them to your target DB with "Delete" option.
Create new empty database, export only needed objects from old database, import them to new database.
In Nav 2016 application database can be separated from data containing database so (I assume) you could just unmount it from database with old objects and mount it to new application database. Not sure tbh.
It is due to the range of the license, for example your development license has a range of tables 7.000.000 - 7.000.200. If you want to delete a table with ID 20.000.000 you have that error.
The best solution is when you do the updrage do not you consider these objects you need to delete. Exports all objects except the objects you want to delete.
I did see some other posts on this, but they were rather old and there does not appear to be any solutions at this point.
I'm trying to determine where a particular table(s) that SSIS is loading during a monthly job is being used in other packages. The package that loads these tables have in the past several months been taking much longer than before, and I'm trying to see if I can eliminate this load all together.
I just happened to check the Allocation packages in our database to see how the tables were being used, and discovered that I can't find anywhere when/where those tables are being used. Is there a function or query I can run in SSMS or elsewhere to determine how to find this information?
Thx in advance - please let me know if I need to clarify something.
The packages are just XML files. If you have the packages somewhere on your file system you can use any program that searches through text files.
I'm not sure about older SSIS projects but with an SSIS project in Data Tools for SQL Server 2012 you can just use the build in search function to search through your entire solution. It will also search in the XML of all the packages.
If you don't have this particular information saved anywhere already in your documentation then I think you are going to have some difficulty in finding an accurate way to retrieve this information. However, there are a few automated data collection options that might help you get most of the way there.
The first option is that because all SSIS Packages are essentially glorified XML that is being fed into an engine you can perform a patterned search on the packages like GREP to look for that particular table name. Any packages that dynamically retrieve and build the table name though would not be found through this method.
Another option would be to run a server side SQL trace with a pattern match based on the table name(s) and limited to the host or application name of SSIS. Run over the course of a month or so would make for a fairly accurate list.
I haven't used it myself, but the DOC xPress tool from PragmaticWorks might be what you're looking for.
I'd like to know your approach/experiences when it's time to initially populate the Grails DB that will hold your app data. Assuming you have CSVs with data, is is "safer" to create a script (with whatever tool fits you) that:
1.-Generates the Bootstrap commands with the domain classes, run it in test or dev environment and then use the native db commands to export it to prod?
2.-Create the DB's insert script assuming GORM's version = 0 and incrementing manually the soon-to-be autogenerated IDs ?
My fear is that the second approach may lead to inconsistencies for hibernate will have the responsability for the IDs generation and there may be something else I'm missing.
Thanks in advance.
Take a look at this link. This allows you to run groovy scripts in the normal grails context giving you access to all grails features including GORM. I'm currently importing data from a legacy database and have found that writing a Groovy script using the Groovy SQL interface to pull out the data then putting that data in domain objects appears to be the easiest thing to do. Once you have the data imported you just use the commands specific to your database system to move that data to the production database.
Update:
Apparently the updated entry referenced from the blog entry I link to no longer exists. I was able to get this working using code at the following link which is also referenced in the comments.
http://pastie.org/180868
Finally it seems that the simplest solution is to consider that GORM as of the current release (1.2) uses a single sequence for all auto-generated ids. So considering this when creating whatever scripts you need (in the language of your preference) should suffice. I understand it's planned for 1.3 release that every table has its own sequence.
Are there tools that make the job easier? If command-line only tools exist, then can anyone speculate if there is a market for a GUI tool? For example, you can create a relational database by modeling visually. Should the same notion exist for LDAP?
Apache Directory Studio includes an ldif-Editor. It is still a text editor but with syntax highlighting, autocompletion and group collapsing for ldif files:
http://directory.apache.org/studio/
I don't know if there are any tools but it isn't that hard to create them by hand.
If you are using IPlanet LDAP then they had a nice interface for creating and modifying schemas though. :)
I don't know if you would consider that to be by hand otherwise that is one tool to use.
I've done some LDIF handling using Perl and the Net::LDAP::LDIF module and it made scripting custom LDAP conversions very easy.
Have you looked at the command-line tool, LDIFDE.exe? Should be on your domain controller.
Business people give me Excel spreadsheets with inconsistent formatting of user and group data and want it loaded right away (then they come back with a new version and tell me they've only added some new users, but some are missing, some data is now invalid, there's a missing column etc.) They want unique passwords assigned, group memberships set up based on department id fields, and so forth.
Then they come back two weeks later and want to know about the differences between that spreadsheet and one from six months ago. Sigh.
I generally just do it all with a few hand-crafted Python scripts.
A lot of times you may be copying objects from one tree to another. Or backing them up. In that case, most LDAP tools have some way of exporting as LDIF. Then you can easily modify the files as needed.
Or copy examples to reuse.
I have seen a number of tools that will do tasks and output the results as LDIF, which can be handy, but they are basically point usage tools.
We are currently reviewing how we store our database scripts (tables, procs, functions, views, data fixes) in subversion and I was wondering if there is any consensus as to what is the best approach?
Some of the factors we'd need to consider include:
Should we checkin 'Create' scripts or checkin incremental changes with 'Alter' scripts
How do we keep track of the state of the database for a given release
It should be easy to build a database from scratch for any given release version
Should a table exist in the database listing the scripts that have run against it, or the version of the database etc.
Obviously it's a pretty open ended question, so I'm keen to hear what people's experience has taught them.
After a few iterations, the approach we took was roughly like this:
One file per table and per stored procedure. Also separate files for other things like setting up database users, populating look-up tables with their data.
The file for a table starts with the CREATE command and a succession of ALTER commands added as the schema evolves. Each of these commands is bracketed in tests for whether the table or column already exists. This means each script can be run in an up-to-date database and won't change anything. It also means that for any old database, the script updates it to the latest schema. And for an empty database the CREATE script creates the table and the ALTER scripts are all skipped.
We also have a program (written in Python) that scans the directory full of scripts and assembles them in to one big script. It parses the SQL just enough to deduce dependencies between tables (based on foreign-key references) and order them appropriately. The result is a monster SQL script that gets the database up to spec in one go. The script-assembling program also calculates the MD5 hash of the input files, and uses that to update a version number that is written in to a special table in the last script in the list.
Barring accidents, the result is that the database script for a give version of the source code creates the schema this code was designed to interoperate with. It also means that there is a single (somewhat large) SQL script to give to the customer to build new databases or update existing ones. (This was important in this case because there would be many instances of the database, one for each of their customers.)
There is an interesting article at this link:
https://blog.codinghorror.com/get-your-database-under-version-control/
It advocates a baseline 'create' script followed by checking in 'alter' scripts and keeping a version table in the database.
The upgrade script option
Store each change in the database as a separate sql script. Store each group of changes in a numbered folder. Use a script to apply changes a folder at a time and record in the database which folders have been applied.
Pros:
Fully automated, testable upgrade path
Cons:
Hard to see full history of each individual element
Have to build a new database from scratch, going through all the versions
I tend to check in the initial create script. I then have a DbVersion table in my database and my code uses that to upgrade the database on initial connection if necessary. For example, if my database is at version 1 and my code is at version 3, my code will apply the ALTER statements to bring it to version 2, then to version 3. I use a simple fallthrough switch statement for this.
This has the advantage that when you deploy a new version of your application, it will automatically upgrade old databases and you never have to worry about the database being out of sync with the software. It also maintains a very visible change history.
This isn't a good idea for all software, but variations can be applied.
You could get some hints by reading how this is done with Ruby On Rails' migrations.
The best way to understand this is probably to just try it out yourself, and then inspecting the database manually.
Answers to each of your factors:
Store CREATE scripts. If you want to checkout version x.y.z then it'd be nice to simply run your create script to setup the database immediately. You could add ALTER scripts as well to go from the previous version to the next (e.g., you commit version 3 which contains a version 3 CREATE script and a version 2 → 3 alter script).
See the Rails migration solution. Basically they keep the table version number in the database, so you always know.
Use CREATE scripts.
Using version numbers would probably be the most generic solution — script names and paths can change over time.
My two cents!
We create a branch in Subversion and all of the database changes for the next release are scripted out and checked in. All scripts are repeatable so you can run them multiple times without error.
We also link the change scripts to issue items or bug ids so we can hold back a change set if needed. We then have an automated build process that looks at the issue items we are releasing and pulls the change scripts from Subversion and creates a single SQL script file with all of the changes sorted appropriately.
This single file is then used to promote the changes to the Test, QA and Production environments. The automated build process also creates database entries documenting the version (branch plus build id.) We think this is the best approach with enterprise developers. More details on how we do this can be found HERE
The create script option:
Use create scripts that will build you the latest version of the database from scratch, which is empty except the default lookup data.
Use standard version control techniques to store,branch,tag versions and view histories of your objects.
When upgrading a live database (where you don't want to loose data), create a blank second copy of the database at the new version and use a tool like red-gate's link text
Pros:
Changes to files are tracked in a standard source-code like manner
Cons:
Reliance on manual use of a 3rd party tool to do actual upgrades (no/little automation)
Our company checks them in simply because someone decided to put it in some SOX document that we do. It makes no sense to me at all, except possible as a reference document. I can't see a time we'd pull them out and try and use them again, and if we did we'd have to know which one ran first and which one to run after which. Backing up the database is much more important then keeping the Alter scripts.
for every release we need to give one update.sql file which contains all the new table scripts, alter statements, new/modified packages,roles,etc. This file is used to upgrade the database from 1 version to 2.
What ever we include in update.sql file above one all this statements need to go to individual respective files. like alter statement has to go to table as a new column (table script has to be modifed not Alter statement is added after create table script in the file) in the same way new tables, roles etc.
So whenever if user wants to upgrade he will use the first update.sql file to upgrade.
If he want to build from scrach then he will use the build.sql which already having all the above statements, it makes the database in sync.
sriRamulu
Sriramis4u#yahoo.com
In my case, I build a SH script for this work: https://github.com/reduardo7/db-version-updater
How is an open question
In my case I am trying to create something simple that is easy to use for developers and I do it under the following scheme
Things I tested:
File-based script handling in git using GitlabCI
It does not work, collisions are created and the Administration part has to be done by hand in case of disaster and the development part is too complicated
Use of permissions and access via mysql clients
There is no traceability on changes to the database and the transition to production is manual
Use of programs mentioned here
They require uploading the structures and many adaptations and usually you end up with change control just like the word
Repository usage
Could not control the DRP part
I could not properly control the backups
I don't think it is a good idea to have the backups on the same server and you generate high lasgs for the process
This was what worked best
Manage permissions per user and generate traceability of everything that is sent to the database
Multi platform
Use of development-Production-QA database
Always support before each modification
Manage an open repository for change control
Multi-server
Deactivate / Activate access to the web page or App through Endpoints
the initial project is in:
In case the comment manager reads this part, I understand the self-promotion but please just remove this part and leave the rest since I think it complies with the answer to the question reacted in the post ...
https://hub.docker.com/r/arelis/gitdb
I hope this reaches you since I see that several
There is an interesting article with new URL at: https://blog.codinghorror.com/get-your-database-under-version-control/
It a bit old but the concepts are still there. Good Read!