I would like to use BULK INSERT to load a few hundred raw data tables into SQL Server. The format of these tables would be similar, although not identical (they come from excel sheets which are not tightly version controlled).
I want to know if there is a way to dynamically generate the table required on SQL Server depending on the headers in the file to be loaded, and then do the BULK INSERT thereafter.
You can connect to those Excel tables using OPEN ROWSET. Then, do the following:
SELECT *
FROM Excel
INTO NewTable
WHERE 0=1
This will transfer the schema. Is this what you want?
You can make an excel script that generates sql statements for creation of the tables, then execute the file before bulk insert the data.
Related
I am building a SSIS package in which package i need to transfer from
an odata source some tables into sql server.
So far i have implement an "insert into" query to the sql server from the tables i read from odata Source. Because the number of tables are 10+ is there a way that i can do "select into" query for faster transfer of those tables in SSIS ?
SSIS has no build in operation to create a table on a destination based on a data set, which is what SELECT ... INTO does.
There is no easy tweak to do this either, SSIS is mostly based for static metadata ETLs, that is performing operations between different sources and destinations with consistent structures and data types. You might achieve what you need with custom scripts, but that would be as well completely outside of SSIS.
If you already know the data you will be inserting into, create the destination tables first (with CREATE TABLE) and then use SSIS to map the corresponding columns. If your destination tables will be dynamic then you will have a hard time using regular SSIS operations to match the metadata of each table, since this is set at design time.
If the problem isn't the table's column data type but the speed of the operation (SELECT ... INTO has minimal logging), then the fastest option is using the bulk insert operation on the destination component when working with SQL Server. It will be faster than regular inserts, but usually slower than performing a SELECT ... INTO directly from SQL.
I have two different databases in two different SQL Servers. The databases are identical in schema but contain different data in one of the tables.
I want to copy all the data from one table in one database to the same table in the other database so that I can get rid of the database from which I am copying the data.
The data is too large so I cannot create data scripts and run it onto other database.
How can I achieve this?
There are many ways like ssis transfer,select * into ,but i prefer below way if you are just transferring data
create a linked server on source server for destination server,then you could refer destination server with four part name
Assuming linked server of source is A and destination server is B,data moving is as simple as
insert into B.databasename.Schema.Table
select * from table---this is in source server and db
if data is huge and you may worry about time outs,you can write a simple script which can do in batches like
While (1=1)
begin
insert into B.databasename.Schema.Table
select top 10000* from table---this is in source server and db
if (##rowcount=0)
break
end
Creating linked server ,you can follow this
You have the following options available to you. Not all of these will work, depending on your exact requirements and the networking arrangements between the servers.
SQL Server Management Studio - Import & Export Wizard: this is accessed from the right-click menu for a database > Tasks > Import Data (or Export Data).
SQL query using a Linked Server: a Linked Server configured between the two servers allows you to reference databases on one from the other, in much the same way as if they were on the same server. Any valid SQL query approach for transferring data between two tables within one database will then work, provided you fully-qualify the table names as Server.Database.Schema.Table.
SSIS: create an SSIS package with both servers as connections, and a simple workflow to move the data from one to the other. There is plenty of information available online on how to use SSIS.
Export to flat-file format then import: this could be done using the Import/Export Wizard above or SSIS, but instead of piping the data directly between the two servers, you would output the data from the source table into a suitable flat-file format on the filesystem. CSV is the most commonly used format for this. This file can then be moved to the destination server using any file transfer approach (compressed e.g. to a Zip file if desired), and imported into the destination table.
Database backup and restore: Similar to (4), but instead of using a flat file, you could create a backup of the source database via Tasks > Back Up... You then move that backup as a file (just like the CSV approach), and restore it onto the destination server. Now you have two databases on the destination server, and can move data from one to the other locally.
I hope, this query helps you!!!
INSERT INTO [dbo].[tablename] (Column1, Column2,Column3)
(select Column1, Column2,Column3, from [Database1].[dbo].[tablename]
Thanks!!!
I need suggestion on best approach from below listed options. I need to validate excel file data and load it to SQL Server
Validations include
Non Duplicate columns
Mandatoty fields present
Fields not present in Database
In case of error I would write in errorlog table in database
Below is my approach
Load the Data into a Temp Table in Database
Run the Validations
Log the Error
On success load it to main tables
Please let me know if you have any other better ideas for this scenario
Here are couple of approaches that are possible:
Using SSIS
Create excel connection manager then use dataflow task with OLEDB Source, lookup transform (to eliminate the records NOT needed), OLEDB destination
directly into main table.
You can also choose to redirect or ignore rows that do not satisfy the transformations.
(use can use bulk insert task if the excel is really large instead of dealing RBAR)
2. Using TSQL
BULK INSERT or BCP or use OPENROWSET into staging table. Beware that you need to have approriate drivers installed (JET for x32 or ACE for x64 SQL Server).
Then do error handling by logging to error table (raiseerror, try-catch) before loading to main table.
We have 2 databases and we need data to be transferred from db 1 to db 2. How can I do that (in SYBASE there are proxy tables) in SQL Server?
As #Nathan says just BULK INSERT the data. Assuming both databases are on the same server then you reference the table usually as databasename.schema.tablename thus db1.dbo.table1 or db2.dbo.table1
As such you can also just create a view in the destination data to use as a 'proxy' and pull the data without actually copying it. The view would be in db2 and be something like:
CREATE VIEW table1 AS SELECT * FROM db1.dbo.table1
I think INSERT INTO would be a good way to go.
http://msdn.microsoft.com/en-us/library/aa933206(v=sql.80).aspx
First of all, you can create a linked server in your destination server to the other server. Then you can do the INSERT INTO.
If you don't want to do that (or are unable), then dump the data to a file and do the very fast BULK INSERT to get the data into your new table.
I want to use Access database with query for pivot table in Excel.
How do I do that?
Open Excel and select Data->Pivot Table....->External data source->Get Data.
Then select Microsoft Access MDB and find your MDB file. Select the
appropriate tables and/or queries and fields.