I have a complex XSD schema and hundreds of XML files conforming to the schema.
How do I automate the creation of related SQL Server tables to store the XML data?
I've considered creating C# classes from the XSD schema using the xsd.exe tool and letting something like Subsonic figure out how to make a shiny database out of it, but not sure if it's the best way to approach it.
Has anyone managed to elegantly import XSD files into SQL Server?
A similar question with good answers: How can I create database tables from XSD files?
I suggest you use SQL Server Integration Services, which comes with SQL Server 2008 or 2005 (Or Data Transformation Services if your stuck with 2000).
Unfortunately it doesn't come with the free "Express" version of SQL Server, however SQL Server Developer edition can be had for < $100 which has the full SQL Server Standard functionality and would suit your needs.
SSIS is a big topic and I'm not going to go over all of the bells and whistles here but basically you:
Create a new SSIS project using BIDS (Business Intelligence Development Studio, a modified Visual Studio that comes with SSIS)
Drag a new Data Flow Task onto the Control Flow surface, then click the data flow tab.
Drag an "XML source" from toolbox into data flow panel, and then configure the XSD and XML file locations.
Drag an ADO.NET data destination from the toolbox onto the dataflow and connect one of the the outputs from the XML source to the input of the ADO.NET destination. If you want to create a new table based on the data output from the xml schema as opposed to using an existing one click on "New" when specifying the Connection Manager Settings in the ADO.NET Destination and it generate and execute the appropriate create table statement. Repeat this for any other outputs from the XML source (there will be one for each logical flat table generated from the schema).
You will most probably need to use other data transformation objects first to transform the data before it loaded into SQL server, but that is the general gist of it. If you need to run the process for a large amount of XML files you could put the task in a control loop and use a variable to set the XML file location.
The MS Documentation on using an XML source in SSIS is here: http://msdn.microsoft.com/en-us/library/ms140277(v=SQL.100).aspx
Just found XSD2DB on Sourceforge, according to the site:
XSD2DB is a command line tool written
in C#, that will read a Microsoft
ADO.NET compatible DataSet Schema File
(XSD) and generate a database.
Checking it out.
Related
I am working on a little Power BI Project. The data source for this project is mostly excel files with untransformed dynamic data.
Initially I used these Excel sheets as the data source and transformed them in PowerQuery prior to building the datamodel. However, now there is a need to load the historical data into SQL server and connect Power BI to SQL as the data source.
Since the source files are not cleaned and transformed prior to importing them into Power Query, I cannot import them directly using the SSIS package - foreachloop container for multiple dynamic files.
Is there a way I can get the data back into SQL Server from PowerQuery after transformation? Is this where PowerQuery Source come into picture? What does it do exactly?
It seems that PowerQuery Source will be the best choice in your case. There is a good article on Microsoft explaining it.
You will need to to three steps:
For each file which you use as a source, create a connection with PowerQuery type in SSIS connection manager;
Using PowerQuery source, copy the full query from Power BI advanced query editor to the "Query" field in the source settings;
On "Connection Managers" tab of the source, map each power query source (each file) with the corresponding connection which you created in step 1.
The output of this source is a list of columns which you can write to the DB using usual data flow task appraoch (like mapping this source to OLE DB destination)
What's the correct way of exporting data from Excel 2013 file to SQL Server database? The data from the Excel file should be transferred into SQL when saving excel file to a database.
I know many answers for this are available but my question is bit different: every time the excel data changes, or the user clicks on save button the data in the database also needs to be updated.
The easiest way to do this is with an SSIS package. SSIS (SQL Server Integration Services) is a package built in to SQL which allows transformations between data formats.
You can create a package by right-clicking on your target database in SQL Server Management Studio and selecting Tasks > Import Data. In the wizard that comes up asking for a data source choose "Microsoft Excel" from the top drop-down labelled as Data Source, then follow the wizard through. You'll have the choice of importing the Excel data into a new table or mapping it into an existing table.
If you want do this programmatically, you can save your package at the end of the wizard and then invoke it via code. But that's a different question.
What you want is not possible (as long as I know off). You can use SSIS package to migrate the Excel sheet into SQL Server, but is imposible to determine if someone "click save" or do some changes on the excel file. SSIS package can be programed to run on schedule or by demand. You should investigate SSIS packages. It is not easy to learn, but do what you need.
Maybe you would like to try a tool I have developed? It's an Excel Add-In that exports Excel data to SQL Server.
There is a feature to automatically export the data to SQL Server every time you press the save button in Excel. If you need to update the database every time a cell value change, you will need to add a few lines of VBA-code that will push the data to SQL Server.
To beta testers I currently give away a free license, so if you are interested in testing it out, send me an email:)
www.sqlpreads.com
I am in the process of migrating a web application database from SQL Server 2000 to SQL Server 2008. Currently there is a DTS package that is used to deploy content changes from a staging database to the production database.
The DTS package is using a Copy SQL Server Objects task with the following options selected: Copy Data (Append Data) and Use Collation. The specific tables to copy are selected in the "Select Objects" dialog.
Because this is the only DTS package we have, it doesn't make much sense to learn and implement an SSIS solution, IMO, so I want to recreate the functioning of the DTS package using only T-SQL.
Writing the Insert and Select is not a problem. What I need to know is how the "Append Data" option works.
Is it looking at each row in the source, finding matching rows in the destination, comparing and updating as necessary OR is is ignoring existing rows and simply appending new rows?
If it is indeed comparing and updating, is it safe to use the SQL Server Checksum function on the data as a method of comparison against the target or is there a better way? Ideally, I'd like to avoid any schema changes.
Please check this msdn article : Migrating DTS Packages to Integration Services
You might be able to migrate the single DTS package to SSIS package very easily using the tool noted in the article.
I would like to copy a table from one database to another. I know you can easily do the following if the databases are on the same SQL Server.
SELECT * INTO NewTable FROM existingdb.dbo.existingtable;
Is there any easy way to do this if the databases are on two different SQL Servers, without having to loop through every record in the original table and insert it into the new table?
Also, this needs to be done in code, outside of SQL Server Management Studio.
Yes. add a linked server entry, and use select into using the four part db object naming convention.
Example:
SELECT * INTO targetTable
FROM [sourceserver].[sourcedatabase].[dbo].[sourceTable]
If it’s only copying tables then linked servers will work fine or creating scripts but if secondary table already contains some data then I’d suggest using some third party comparison tool.
I’m using Apex Diff but there are also a lot of other tools out there such as those from Red Gate or Dev Art...
Third party tools are not necessary of course and you can do everything natively it’s just more convenient. Even if you’re on a tight budget you can use these in trial mode to get things done….
Here is a good thread on similar topic with a lot more examples on how to do this in pure sql.
SQL Server(2012) provides another way to generate script for the SQL Server databases with its objects and data. This script can be used to copy the tables’ schema and data from the source database to the destination one in our case.
Using the SQL Server Management Studio, right-click on the source database from the object explorer, then from Tasks choose Generate Scripts.
In the Choose objects window, choose Select Specific Database Objects to specify the tables that you will generate script for, then choose the tables by ticking beside each one of it. Click Next.
In the Set Scripting Options window, specify the path where you will save the generated script file, and click Advanced.
From the appeared Advanced Scripting Options window, specify Schema and Data as Types of Data to Script. You can decide from here if you want to script the indexes and keys in your tables. Click OK.
Getting back to the Advanced Scripting Options window, click Next.
Review the Summary window and click Next.
You can monitor the progress from the Save or Publish Scripts window. If there is no error click Finish and you will find the script file in the specified path.
SQL Scripting method is useful to generate one single script for the tables’ schema and data, including the indexes and keys. But again this method doesn’t generate the tables’ creation script in the correct order if there are relations between the tables.
Microsoft SQL Server Database Publishing Wizard will generate all the necessary insert statements, and optionally schema information as well if you need that:
http://www.microsoft.com/downloads/details.aspx?familyid=56E5B1C5-BF17-42E0-A410-371A838E570A
Generate the scripts?
Generate a script to create the table then generate a script to insert the data.
check-out SP_ Genereate_Inserts for generating the data insert script.
Create the database, with Script Database as... CREATE To
Within SSMS on the source server, use the export wizard with the destination server database as the destination.
Source instance > YourDatabase > Tasks > Export data
Data Soure = SQL Server Native Client
Validate/enter Server & Database
Destination = SQL Server Native Client
Validate/enter Server & Database
Follow through wizard
I am trying come up with a way to pull the tables out of an Access database, automate the creation of those same tables in a SQL 2008 DB, and move the data to the new tables. This process will happen on a regular basis and there may be different tables each time.
I would like to do this totally in SSIS.
C# SQL CLR objects are an option.
The main issue I have been running into is how to get the Access table's schema and then convert that to a SQL script that I can run via SSIS.
Any ideas?
TIA
J
SSIS cannot adapt to new tables at runtime. (You can change connections, move a source to a table with a different name, but the same schema) So, it's not really easy to do what I think you are saying: Upsize an arbitrary set of tables in an Access DB to SQL (mirroring their structure and data, naming, etc), so that I can then write some straight SQL to transform the data into another SQL database or the same part of the database.
You can access the SSIS object model from C# and build a package (or modify a template package) programmatically and then execute it. This might offer the best bang for your buck, but the SSIS object model is kind of deep. The SSIS Team blog have finally started putting up examples (a year after I had to figure a lot of this out for myself)
There is always the upsizing wizard, and I'm sure there are some third party tools.