My issue is that I have a SQL database that is the output of a SQL project in an ASP.NET solution. This output consists of 2 datafiles, lets call them MainDB.mdf and DocContent.mdf, and the necessary log files. DocContent.mdf has one table, doc_content, which, you guessed it, stores the bytes for documents. There are multiple dev's working on this project, and we each have a (localdb) instance of the DB to use.
What I need to make happen is to attach MY MainDB.mdf referring to dev B's DocContent.mdf. The data FK's match 1:1, the schema is identical, the only difference is that Dev B populated the varbinary field with uploaded documents.
SQL seems to be doing everything in its power to prevent this. I get:
I realize I could do a Data Compare and pump the data in that way, but unfortunately Dev B and I are not at the same location and the logistics of him sending his full DB output would be harder than finding a work around with what I've got (I hope, anyway).
Surely there is a way to tell SQL to shut up and just use the datafile provided?
The only info I could find on this subject referred to attaching a DB without its LDF. Anyone know how to make this scenario work?
So, for anyone interested, the short answer is 'no'. There are a number of convoluted steps that you you can take to force the database into SUSPECT mode and possible even get it into single user emergency mode, but it was not worth the effort and the end result is that you will have to hack into the sys tables in order to make it limp along this way.
In the end it was easier to get a full transfer of Dev B's data files.
Related
Random question, but I was wondering if there is a way to determining the SQL server version of a database before attaching it (i.e. whilst in a folder on the network).
The reason I ask is because we receive extremely large databases that take ages to attach only to then fail because they are not the correct version.
I was hoping there might be a tip to save time.
Many Thanks
[This is a different approach to a similar question I did couple days ago, I recognize the approach of "cloning" was wrong as this would involve too many changes and offline the actual DB]
EXPLANATION:
I have a set of systems-installations (WebApp + SQL Server database), where the databases have two groups of objects, one small "core" and second and large "instance_specific".
Those systems need to be rebuilt base on another system from time to time (like an Excel financial scenario case), a rebuild of a system will be best if "core" can stay, and everything else delete and importing from a source database (the source database can be different from time to time)
SO FAR:
CREATE DATABASE Database1_copy AS COPY OF Database1;
is not an option as will need to rebuild all core and neither Azure manage well DB created from SSMS, neither establish the needed connections, neither maintain the users and roles... (not good idea to change the db)
Same from recovery from backup / Import Data-Tier Application ...
Script seems really not a good option, doesn't include SEQUENCES (probably other objects as well) and it weight too much to be practical (see notes at the end). How should I load a 360mb script on SSMS to execute it on the destination system?
SQLAzureMW v5.15.6 Release Binary for SQL Server 2014, not able really to try as I run out of memory and trow and exception (with 16gb ram). Databases too big, or buggy ?
Solutions based on manual usage of backup sql functions, as:
https://www.mssqltips.com/sqlservertip/1243/auto-generate-sql-server-database-restore-scripts/
are not supported in Azure ...
To get an instance "clear" and ready to receive the backup I really like the idea of the next post, filtering objects out that belong to core:
Drop all the tables, stored procedures, triggers, constraints and all the dependencies in one sql statement
And due the mount of data, it would be great if "data-flows" goes from db to db directly, not back to local, and even better if it could be executed from the destination DB
SO:
In my best dreams I imagine a SP (similar to the one form bellow on stackoverflow) that goes object by object, reconstruct-it, and loads the data within, and even initialize sequences to the last used values.
I try to copy TABLES (with constrains and index, AND DATA), SEQUENCES (with acual value), VIEWS, SPs, FUNCTIONS, and I think that's all (do not need to pass USERS and ROLES)
I come also to Apexsql and RedBeltTools, haven't try yet, but maybe the solution, even I would prefer not to rely in 3rd party software that runs locally.
Am I out of options?
Should I start digging how to build my own SP migration tool?
(I am not really sure how/where to start ...)
Just some numbers of an actual source database:
CHECK_CONSTRAINT 12
DEFAULT_CONSTRAINT 259
FOREIGN_KEY_CONSTRAINT 145
PRIMARY_KEY_CONSTRAINT 162
SEQUENCE_OBJECT 7
SERVICE_QUEUE 3
SQL_INLINE_TABLE_VALUED_FUNCTION 1
SQL_SCALAR_FUNCTION 27
SQL_STORED_PROCEDURE 765
SQL_TABLE_VALUED_FUNCTION 6
UNIQUE_CONSTRAINT 54
USER_TABLE 268
VIEW 42
Other than Users and Roles, I think nothing else is missed here, isn't it?
Script DB delivers a 360 mb (over 500k lines)
and it seems to not include SEQUENCES at least
(SMSS scripting let me just chose: Tables/Views/SPs/UserDefFunctions/Users/DatabaseRole)
Since you have mention you want to replicate 90% of anAzureSQL database to an existent one. I could see this scenario as comparing two databases in schema and data and trying to send selected changes from an origin db to a destiny db.
That said I would recommend using SQL Server Data Tools, it has a complete suite of tools that could aid you in that scenario, and multiple ways to update the destiny db: inline, via script, via dacpac, via bacpac.
Also since your question is so open, I don't know if it will solve your scenario, but try looking at it and its compare functionality.
I have inherited a custom built tool that is poorly designed and unstable, and I have a great opportunity to rebuild it from scratch. This is an internal tool only that works almost entirely in Access, and its purpose is to provide higher detail on parts that cost the company over a certain dollar amount.
How it works:
1) The raw data (new part numbers) gets pulled nightly from the EDW via macros in Access.
2) The same macros then join two tables (part numbers from one, names from another). Any part under a certain dollar amount is removed, and the new data is appended to the existing Access database.
3) During the day employees can then open a custom Access form to add more details about the part. Different questions are asked depending on the part category.
4) The completed form is forwarded to management, and the information entered is retained in the Access database – it does not write back to the EDW.
5) Managers can also pull some basic reports from the database, based on overall costs.
The problems:
1) Currently everyone has to have Access installed on their work stations, and whenever there is an update the new database gets pushed to their stations. This is not considered an ideal situation by management or IT.
2) If anyone has left the tool open accidentally at the end of the day the database is locked out, therefore the macros cannot run and the tool cannot be updated with new part numbers.
3) If the tool cannot update for a few days in a row the database can become corrupted. We can restore from the last good backup, in the past this has resulted in the loss of multiple days of work.
Ideally we want to take the tool completely out of Access. I am building a SharePoint site that can host the tool, which (if I can get it right) will eliminate the need of Access on end-user stations with a database push. However the SharePoint form would need read/write ability.
The big question is: How do I build this?
I have a completely open path of possibilities – I can design it work any way I want, using any tools or platform I want, as long as it works. It does not have to update automatically, as I already run a number of SQL scripts at the start of my day and adding one more is inconsequential.
The resources I have at my immediate disposal are: SharePoint (with designer), Access, Toad, and SQL Server. The database can be hosted on a shared network drive.
I am a recent college graduate with basic SQL knowledge. I have about a year to produce a final product, but would like to get it up and running far sooner if possible.
Any advice on what direction to pursue would be very helpful, thank you.
Caveat: I've never worked with SQL Server, so I don't know all of it's capabilities (I'm an Oracle developer).
What I'd do in your situation is something like the following (although not necessarily in this exact order):
Get a SQL Server database set up to host your tables.
Create the tables etc
Migrate test data across (I'm assuming you have a dev/uat/test environment for your current system! If you haven't, make sure you set up at least a separate test environment to prod for your new db!)
Write stored procs to do the work for adding new parts, updating existing data, etc etc
Set up an automated job on the db (I'm assuming SQL Server can do this!) to do the overnight processing.
Create a separate db user with the necessary permissions to call the stored procedures
Get your frontend to call the stored procs with the relevant parameters using the db user you created in step 6 to connect to the db.
You'd also have to think about transaction control to try and mitigate the case where users go home at the end of the day without committing their work - Does the db handle the commits/rollbacks or does Sharepoint?
Once you've worked out everything in your test environment, it's then a case of creating the prod db, users and objects, and then working out the best way of migrating the prod data across.
Good luck.
Don't forget to get backups for the new db set up as well.
Is there a tool (for PosgreSQL, ideally), which can make a small, but consistent sample of a big database?
The this is we need a testing database, but we don't want to fully copy the production one. First, because it is too big and second, the nature of testing implies that the testing database will be recreated several times in the process.
Obviously, you can not simply take random rows from some tables, because this will violate the hell out of foreign keys and what not. So, I wonder is there a tool available that can do that?
What about generating some mock data with a tool like databene benerator, just as much as you want, and store them for reuse.
I once built such a tool for the IDMS system.
I was in the process of making it work for SQL systems too when the managers of the company we were mergered into told me I could not continue wasting my time on such futile and unnecessary pieces of software.
Until this day, I have still neither seen nor heard of any commercially available thing that matches what I achieved way back then.
Yes, I wrote a tool to do this: https://github.com/mla/pg_sample
From the README:
pg_sample is a utility for exporting a small, sample dataset from a
larger PostgreSQL database. The output and command-line options
closely resemble the pg_dump backup utility (although only the
plain-text format is supported).
The sample database produced includes all tables from the original,
maintains referential integrity, and supports circular dependencies.
You can use pg_dump --schema-only to dump only the schema of the database. Then use pg_restore to load the dump into a new database. From there you have a few options:
Create your data by hand; this will allow you to cover edge cases but will take a while if you want to test on a lot of data.
Script a few queries to import random sections of each table on your database. As you said, this will violate foreign key constraints, but when it does just ignore the failure. Keep track of the number of successes and keep going until you have as many data items as you want. This may not work depending on your schema, however, if you have very restrictive constraints, because it might take too long to hit on succeeding data.
Back in my Oracle days we would have a test database with a very small auto generated set of data. At the time it was about a 5th of the production database size. We would than copy the stats from the production database and put them into our test database to make it think it had billions of rows in tables when in reality it only had 500,000. This allowed us to get explain plans in test that would we would get in production. It has it's values, but doesn't solve all your question and I'm not sure how easy or even feasible it is to mess with PostgreSQL's stats.
You can create custom scripts to create meaningful copy of production data or use commercial products such as Informatica data subset or IBM optim
Please use IBM OPTIM to production database sample. If you have Prod data available in DB, go for OPTIM TDM or if there is no Prod data go for IBM OPTIM TDF (Test data Fabricator)
I'm writing a system at the moment that needs to copy data from a clients locally hosted SQL database to a hosted server database. Most of the data in the local database is copied to the live one, though optimisations are made to reduce the amount of actual data required to be sent.
What is the best way of sending this data from one database to the other? At the moment I can see a few possibly options, none of them yet stand out as being the prime candidate.
Replication, though this is not ideal, and we cannot expect it to be supported in the version of SQL we use on the hosted environment.
Linked server, copying data direct - a slow and somewhat insecure method
Webservices to transmit the data
Exporting the data we require as XML and transferring to the server to be imported in bulk.
The data copied goes into copies of the tables, without identity fields, so data can be inserted/updated without any violations in that respect. This data transfer does not have to be done at the database level, it can be done from .net or other facilities.
More information
The frequency of the updates will vary completely on how often records are updated. But the basic idea is that if a record is changed then the user can publish it to the live database. Alternatively we'll record the changes and send them across in a batch on a configurable frequency.
The amount of records we're talking are around 4000 rows per table for the core tables (product catalog) at the moment, but this is completely variable dependent on the client we deploy this to as each would have their own product catalog, ranging from 100's to 1000's of products. To clarify, each client is on a separate local/hosted database combination, they are not combined into one system.
As well as the individual publishing of items, we would also require a complete re-sync of data to be done on demand.
Another aspect of the system is that some of the data being copied from the local server is stored in a secondary database, so we're effectively merging the data from two databases into the one live database.
Well, I'm biased. I have to admit. I'd like to hypnotize you into shelling out for SQL Compare to do this. I've been faced with exactly this sort of problem in all its open-ended frightfulness. I got a copy of SQL Compare and never looked back. SQL Compare is actually a silly name for a piece of software that synchronizes databases It will also do it from the command line once you have got a working project together with all the right knobs and buttons. Of course, you can only do this for reasonably small databases, but it really is a tool I wouldn't want to be seen in public without.
My only concern with your requirements is where you are collecting product catalogs from a number of clients. If they are all in separate tables, then all is fine, whereas if they are all in the same table, then this would make things more complicated.
How much data are you talking about? how many 'client' dbs are there? and how often does it need to happen? The answers to those questions will make a big difference on the path you should take.
There is an almost infinite number of solutions for this problem. In order to narrow it down, you'd have to tell us a bit about your requirements and priorities.
Bulk operations would probably cover a wide range of scenarios, and you should add that to the top of your list.
I would recommend using Data Transformation Services (DTS) for this. You could create a DTS package for appending and one for re-creating the data.
It is possible to invoke DTS package operations from your code so you may want to create a wrapper to control the packages that you can call from your application.
In the end I opted for a set of triggers to capture data modifications to a change log table. There is then an application that polls this table and generates XML files for submission to a webservice running at the remote location.