I am trying to insert from a .txt file (or .csv would be better!) but I want to be able to pick specific columns from the .txt (or .csv file), such as column #1 #2 #5 #10 etc..
Is this possible to do?
At the minute the following MSSQL will insert the entire contents and requires matching number of fields on the database it is inserting into.
BULK
INSERT TableName
FROM 'C:\Users\User\Desktop\test.txt'
WITH
(
FIELDTERMINATOR = ',',
ROWTERMINATOR = '\n'
)
GO
Here is an example of one row of data from the .csv:-
"1A","SOUTH BANK UNIVERSITY","","","","103","","BOROUGH ROAD","","","","LONDON","","SURREY","SOUTHWARK","1","L","38026","SE1 0AA","53180","17940","BEGE","Q71","08Q ","064"
Any help would be much appreciated!
You can use OPENROWSET
You'll need the Microsoft.ACE.OLDB.xx.0 provider installed.
(xx on my server = 15.0, or 12.0)
If your file does not have column names:
SELECT F1, F2, F5, F10
FROM OPENROWSET (
'Microsoft.ACE.OLEDB.15.0',
'Text;Database=C:\Projects\data;Extended Properties="HDR=No;IMEX=1"',
'SELECT * FROM data.txt') a
If your file has column names (specify the name of the column)
SELECT Column1, Column2, Column5, Column10
FROM OPENROWSET (
'Microsoft.ACE.OLEDB.15.0',
'Text;Database=C:\Projects\data;Extended Properties="HDR=Yes;IMEX=1"',
'SELECT * FROM data.txt') a
You'll need to enable 'Ad Hoc Distributed Queries' using sp_configure
EXEC sp_configure 'show advanced options', 1
RECONFIGURE
GO
EXEC sp_configure 'ad hoc distributed queries', 1
RECONFIGURE
GO
You also might have to "allow in process" under the provider properties.
(Server Object | Linked Servers | Providers)
Hope that helps.
If you save your data into a csv or excel format, you can use the SQL Server Management Studio function to import the data from your sheet with a wizard instead of using commands.
To access this function, right click on your database, then Tasks -> Import Data. Choose "Flat File Source" on the next page if you wish to use your .txt/.csv file. Choose Excel if you want to save your csv as an Excel file and import that instead.
Just follow the wizard from there. You can import the columns into a new or existing table.
As a thought, it is probably easier to save your csv file into Excel and then just manually delete the columns you don't need prior to the import.
Related
I want to import an Excel sheet into a SQL Server database but I'm not able to do that. I tried the procedure manually by right click on database name then import data then data source then file destination by finish. All processes are ok. I got all the green tick marks but 0 rows transferred. I don't what to do, I tried it so many times but not able to that. Please help.
USE [DatabaseName]
GO
SELECT * INTO Schema.TableName
FROM OPENROWSET('Microsoft.ACE.OLEDB.12.0',
'Excel 12.0; Database=DriveLetter:\FolderName\FileName.xlsx', [Sheet1$]); -- Modify the path to the file
GO
I am trying to move tables from access to SQL Server programmatically.
I have some limitation in the system permissions, ie: I cannot use OPENDATASOURCE or OPENROWSET.
What I want to achieve is to transfer some table from Access to SQL Server and then work on that tables through vba (excel)/python and T-SQL.
The problem is in the timing that it is required to move the tables.
My current process is:
I work with vba macros, importing data from excel and making same transformation in access, to then import into the SQL Server
destroy the table in the server: "DROP TABLE"
re-importing the table with DoCmd.TransferDatabase
What I have notice is that the operation seems to be done based on a batch of rows and not directly. It is taking 1 minutes and half each 1000 rows. The same operation on Access it would have taken few seconds.
I understood that it is a specific way of SQL Server to use import by batches of 10 rows, probably to have more access on data: Micorsoft details
But in the above process I just want a copy the table from access to the SQL as fast as possible as then I would avoid cross platform links and I will perform operation only on the SQL Server.
Which would be the faster way to achieve this goal?
Why are functions like OPENDATASOURCE or OPENROWSET are blocked? Do you work in a bank?
I can't say for sure which solution is the absoute fastest, but you may want to consider exporting all Access tables as separate CSV files (or Excel files), and then run a small script to load each of those files into SQL Server.
Here is some VBA code that saves separate tables as separate files.
Dim obj As AccessObject, dbs As Object
Set dbs = Application.CurrentData
For Each obj In dbs.AllTables
If Left(obj.Name, 4) <> "MSys" Then
DoCmd.TransferText acExportDelim, , obj.Name, obj.Name & ".csv", True
DoCmd.TransferSpreadsheet acExport, acSpreadsheetTypeExcel9, obj.Name, obj.Name & ".xls", True
End If
Next obj
Now, you can very easily, and very quickly, load CSV files into SQL Server using Bulk Insert.
Create TestTable
USE TestData
GO
CREATE TABLE CSVTest
(ID INT,
FirstName VARCHAR(40),
LastName VARCHAR(40),
BirthDate SMALLDATETIME)
GO
BULK
INSERT CSVTest
FROM 'c:\csvtest.txt'
WITH
(
FIELDTERMINATOR = ',',
ROWTERMINATOR = '\n'
)
GO
--Check the content of the table.
SELECT *
FROM CSVTest
GO
--Drop the table to clean up database.
DROP TABLE CSVTest
GO
https://blog.sqlauthority.com/2008/02/06/sql-server-import-csv-file-into-sql-server-using-bulk-insert-load-comma-delimited-file-into-sql-server/
Also, you may want to consider one of these options.
https://www.online-tech-tips.com/ms-office-tips/ms-access-to-sql-database/
https://support.office.com/en-us/article/move-access-data-to-a-sql-server-database-by-using-the-upsizing-wizard-5d74c0df-c8cd-4867-8d07-e6e759d72924
I have an Excel file generated by another program that has multiple Excel sheets.
The column structure is the same except that just the first sheet has headers, the rest are just continuing from where the last sheet ended - it is an .xls format that allows only 65563 rows per sheet.
I want to import them in one table in SQL Server Management Studio.
I think I have some tutorial for you. It helps you to do it quickly to insert a record from Excel to database.
This is sample code:
INSERT INTO Employee
SELECT *
FROM OPENROWSET('Microsoft.ACE.OLEDB.12.0',
'Excel 12.0 Xml; HDR=YES; IMEX=1;
Database=C:\DataFiles\EmployeeData1.xlsx',
[vEmployee$]);
For more detail, visit this link: tutorial link
Or you can also do it by this way: Tutorial for excel to database
There are a few things you can do.
Given below is the script to migrate data from Excel 2003:
SELECT * FROM OPENROWSET('Microsoft.Jet.OLEDB.4.0'
,'Excel 4.0;Database=D:\test.xls','SELECT * FROM [sheet1$]')
But, when the same code is used for Excel 2007 & above, it gives the following error:
SELECT * FROM OPENROWSET('Microsoft.Jet.OLEDB.4.0'
,'Excel 4.0;Database=D:\test.xlsx','SELECT * FROM [sheet1$]')
Msg 7399, Level 16, State 1, Line 1
The OLE DB provider “Microsoft.Jet.OLEDB.4.0” for linked server “(null)” reported an error. The provider did not give any information about the error.
Msg 7303, Level 16, State 1, Line 1
Cannot initialize the data source object of OLE DB provider “Microsoft.Jet.OLEDB.4.0” for linked server “(null)”.
Lets solve this issue step by step.
Step 1 :
Download Microsoft Access Database Engine 2010
Step 2 :
Install Microsoft Access Database Engine 2010 in your machine.
Step 3 :
Open SQL Server Management Studio. (If already open, close and re-open)
Step 4 :
Enable Ad hoc queries.
sp_configure 'show advanced options', 1;
RECONFIGURE;
sp_configure 'Ad Hoc Distributed Queries', 1;
RECONFIGURE;
GO
Step 5 :
Close the excel sheet that you need to import (if open).
Step 6 :
Give the correct path, file name and excel sheet name with $ sign in the script given below and execute the query.
This information must be correct.
SELECT * FROM OPENROWSET ('Microsoft.ACE.OLEDB.12.0',
'Excel 12.0;Database=E:\test.xlsx;HDR=YES;IMEX=1',
'SELECT * FROM [sheet1$]');
If your first row has a column name then “HDR=YES“, else it should be No.
In case you need this query result in temporary tables, given below is the script to insert Excel records into temp table.
SELECT * into Temp FROM OPENROWSET ('Microsoft.ACE.OLEDB.12.0',
'Excel 12.0;Database=E:\test.xlsx;HDR=YES;IMEX=1',
'SELECT * FROM [sheet1$]');
Or, similarly...
---- Has column names
SELECT * FROM OPENROWSET ('Microsoft.ACE.OLEDB.12.0', 'Excel 12.0;Database=E:\test.xlsx;HDR=YES;IMEX=1', 'SELECT * FROM [sheet1$]');
---- Does not have column names
SELECT * FROM OPENROWSET ('Microsoft.ACE.OLEDB.12.0', 'Excel 12.0;Database=E:\test.xlsx;HDR=NO;IMEX=1', 'SELECT * FROM [sheet2$]');
---- Does not have column names
SELECT * FROM OPENROWSET ('Microsoft.ACE.OLEDB.12.0', 'Excel 12.0;Database=E:\test.xlsx;HDR=NO;IMEX=1', 'SELECT * FROM [sheet3$]');
I have created a table called DimInternationalFunction.
IF EXISTS (SELECT * FROM sys.objects WHERE object_id = OBJECT_ID(N'[DimInternationalFunction]') AND type in (N'U'))
DROP TABLE [DimInternationalFunction]
Go
Create Table DimInternationalFunction
(IntFunctionKey int NOT NULL identity primary key,
SubSubFunctionString char(10),
FunctionCode char(3),
SubFunctionCode char(6),
SubSubFunctionCode char(10),
SubSubFunctionName nvarchar(60),
SubFunctionName nvarchar(60),
FunctionName nvarchar(60))
I have initially inserted records in this table in SSMS.
After inserting the initial records manually in SSMS, now my manager wants me to insert "new records only" using SSIS.
I have tried using this in SSMS and it worked. Either it gives me 0 records inserted or sometimes it gives me 5 records inserted as a result. My manager wants me to do this in SSIS.
I tried using this script inside the OLE DB Source under Data Access Mode: SQL Command and SQL Command text:
insert into DWResourceTask.dbo.DimInternationalFunction
select f.SubSubFunctionString,
f.FunctionCode,
f.SubFunctionCode,
f.SubSubFunctionCode,
f.SubSubFunctionName,
f.SubFunctionName,
f.FunctionName
from ODS_Function F
where FunctionViewCode = 'INT'
and not exists (select * from DWResourceTask.dbo.DimInternationalFunction I
where (f.SubSubFunctionString=i.SubSubFunctionString
and f.FunctionCode=i.FunctionCode
and f.SubFunctionCode=i.SubFunctionCode
and f.SubSubFunctionCode=i.SubSubFunctionCode
and f.SubSubFunctionName=i.SubSubFunctionName
and f.SubFunctionName=i.SubFunctionName
and f.FunctionName=i.FunctionName)
)
The error message that I got after clicking preview is
The component reported the following warnings:
Error at Int Function [International Function Table [33]]: No column information was returned by the SQL command.
Choose OK if you want to continue with the operation.
Choose Cancel if you want to stop the operation.
Is there another component in SSIS that can do this? or can I just use either exec sql task component or ole db source?
I am thinking of using exec sql task connected to a data flow task, inside the data flow task I will put ole db source containing a staging table and do a delete on that or is there any other way to do it. Please help. Thanks in advance.
You could do it with an Execute SQL task.
If you want to do it "the pure SSIS way", you could use a lookup component. Set the "rows with no matching" handler to "Redirect to no match output", and configure the target table as connection. Then use the "No Match Output" only, ignoring the "Match Output". And send the records from the "No Match Output" to the target.
In spite of its name, the "Lookup" component can be used to filter data in many cases.
But I would assume the Execute SQL task would be more efficient for large data sets, keeping all data within the database engine.
How to export a table to a text file?
I need to get the INSERT script (structure and data) for an already existing table.
In SQL2k, try to read about bulk copy, the command should be bcp i think
examples from MS help file has something like ...
Exporting data from table to text file
bcp "SELECT au_fname, au_lname FROM pubs..authors ORDER BY au_lname" queryout Authors.txt -c -Sservername -Usa -Ppassword
Importing data from text file to table:
The command to bulk copy data from Newpubs.dat into publishers2 is:
bcp pubs..publishers2 in newpubs.dat -c -t , -r \n -Sservername -Usa -Ppassword
Alternatively, you can use the BULK INSERT statement from a query tool, such as SQL Query Analyzer, to bulk copy data:
BULK INSERT pubs..publishers2 FROM 'c:\newpubs.dat'
WITH (
DATAFILETYPE = 'char',
FIELDTERMINATOR = ',',
ROWTERMINATOR = '\n'
)
Gath
In SQL Server 2005 Management Studio, you can use the Import/Export Wizard (not sure if you specifically needed a script, or simply a way to export the structure/data to a file, but this suggestion will do it without an actual script):
right-click on the database containing the table
select Tasks->Export Data
Choose a Data Source screen: (after the welcome screen) leave defaulted values, click Next
Choose a Destination: "Flat File Destination" for the Destination field. Then fill in the file name/path and the other options as you wish, click Next
Select Copy data..., click Next
Select the table to export, click Next
On the Save and Execute Package screen, you can just leave Execute Immediately selected, or if you'd like to save the resulting "script" as a SSIS package you can select that option also. Click Next, then Finish, to execute the export
Your resulting file will have the contents of the table. If you then need to "insert" this data into a different db you can use the "Import Data" option of the Wizard to import the data from the text file into the other database/table.
try this:
http://vyaskn.tripod.com/code.htm#inserts
You can build the INSERT statement programatically by fetching the column info from the information_schema where each row of data describes a column:
SELECT table_name,
ordinal_position,
column_name,
data_type,
is_nullable,
character_maximum_length
FROM information_schema.columns
WHERE table_name LIKE '%TableName%'
ORDER BY ordinal_position
For exporting data BCP is the tool and the BOL has several decent examples:
bcp AdventureWorks.Sales.Currency out Currency.dat -T -c
You can run a insert script generator like this one
or a desktop tool like this link