I am trying to use BULK INSERT to add rows to an existing table from a .csv file. For now I have a small file for testing purposes with the following formatting:
UserID,Username,Firstname,Middlename,Lastname,City,Email,JobTitle,Company,Manager,StartDate,EndDate
273,abc,dd,dd,dd,dd,dd,dd,dd,dd,dd,dd
274,dfg,dd,dd,dd,dd,dd,dd,dd,dd,dd,dd
275,hij,dd,dd,dd,dd,dd,dd,dd,dd,dd,dd
And this is what my query currently looks like:
BULK INSERT DB_NAME.dbo.Users
FROM 'C:\data.csv'
WITH
(
FIRSTROW = 2,
FIELDTERMINATOR = ',',
ROWTERMINATOR = '\n'
)
When I execute this query it returns 1 row affected. I checked the entry in the table and noticed that the data in the file is inserted in the table as a single row.
What could be causing this? What I am trying to accomplish is insert those rows in individual rows in the table. See (long)image below
The first column is actually an IDENTITY column so in the file I just specified a integer even though it will be overwritten by the auto generated ID as I am not sure how to tell the query to start inserting from the second field yet.
There are more columns created in the actual table than specified in the file as not everything needs to be filled. Could that be causing it?
The problem is that you are loading data into the first column. To skip a column create a view over your table with just the columns you want to load and BULK INSERT into view. See below example (from MSDN: https://msdn.microsoft.com/en-us/library/ms179250.aspx):
CREATE VIEW v_myTestSkipCol AS
SELECT Col1,Col3
FROM myTestSkipCol;
GO
USE AdventureWorks2012;
GO
BULK INSERT v_myTestSkipCol
FROM 'C:\myTestSkipCol2.dat'
WITH (FORMATFILE='C:\myTestSkipCol2.xml');
GO
What I would recommend you do instead is to create a staging table which matches the file exactly. Load data into that and then use INSERT statement to copy it into your permanent table. This approach is much more robust and flexible. For example, after loading the staging table you can perform some data validation or cleanup before loading the permanent table.
Related
I am trying to move all the data from fælge column in the test table into the fælgen column in table test to with this query
INSERT INTO [dbo].[test2] ([Fælgen])
SELECT Fælge
FROM [dbo].[test];
but I am getting a error with it saying that it cant insert the null into column ET which is not that column i am trying to insert my data to
Msg 515, Level 16, State 2, Line 2
Cannot insert the value NULL into column 'ET', table 'OminiData.dbo.test2'; column does not allow nulls. INSERT fails.
While you are going to Insert data to table you need to make sure that all the constraints are passed and nothing violate them.
You need to make sure that all Non-Null columns have value while inserting the new record. In fact you can not Insert value to single column in a table without considering the constraints and providing value for non-null-able columns.
Finally to overcome this error you have two choice:
Update the destination column(If there is a one-one or one-many relation between Test and Test2, This will need using Update based on Join, other wise a simple Update will be the solution)
Provide non-null value for ET
INSERT INTO [dbo].[test2] ([Fælgen], [ET])
SELECT Fælge , '' as ET
FROM [dbo].[test];
Press right mouse button on the table, chose Script table as... -> INSERT. There will be a script template for insertion. All columns mentioned in this script should have values determined by there datatypes.
I was working on same error . found this from this link SQL Server: copy data from one column to another column?
Using Merge I was able to copy data from one column to another in two different table.
Make sure data type for both columns need to get copied, and row count for both tables.
You cannot insert values to a single DB columns. You always insert entire rows. So your statement doesn't really mean insert some values to this column. What it really means is insert some values to this column and null/default values to all other columns. If any of those other columns does not allow nulls and doesn't have a non-null default, insert will fail.
Background: I am trying to have an after update trigger which stores the changed values dynamically into another table. Since this trigger should be generic and easy to transfer to other tables and won't cause problems, if I add additional columns (If my whole code should be required to solve this, I'll update the question)
While trying to do this, I encounter following issue: I want to store the inserted table into an temporary table, which I do in this way:
SELECT *
INTO #tempINSERTED
FROM INSERTED
But the original table contains both: ntext and timestamp columns which aren't allowed in temporary tables.
Another approach I tried, was looping through the system table INFORMATION_SCHEMA.COLUMNS and build a SQL statement as a string excluding non-copyable columns, but this way I cannot access the inserted table. - I already figured I cannot access inserted if I use sp_executesql.
So my question: is there a way to access the inserted table and exclude non-copyable columns as ntext, text, image ?
Thanks in advance
You want the triggers to run fast. So the better approach would be to generate the create trigger code rather than looping through the fields in the trigger itself. Then if the table schema changes you will need to regenerate the trigger.
For your #TEMPINSERTED table you can use nvarchar(max) in place of ntext,
varchar(max) for text and varbinary(max) in place of image. You can also use and binary(8) or bigint in place of timestamp.
I would suggest using a table variable instead of an #temptable. I.e.:
declare #tempTable table (
fieldname int, -- and so on
)
I excluded values of identity column of the table in the text file i used to load. This resulted error while loading. Please let me know how to deal with this scenario.
Your source csv file should include all columns, even the identity column.
The destination table with the identity column will create its own values when you bulk upload and it will ignore the column in your source CSV file unless you specify the KEEPIDENTITY property.
See BULK INSERT (Transact-SQL)
I insert information into my table using a Bulk Insert to speed things up. Now I want to add a trigger to my table. But this trigger is run once with every Bulk Insert whereas I need to know what the rows that were inserted are during the latest bulk insert.
So, is there a query to know what the inserted rows during BULK INSERT were?
If you have an ID IDENTITY column, you could make a note of the ID before the BULK INSERT and then all rows with an ID higher than this value you wrote down have been bulk inserted.
You cannot have triggers on a per-row basis in SQL Server, nor can you do anything else (like using an OUTPUT statement) to capture the inserted rows, really. You just have to look at what's in the database before and after the BULK INSERT
How to do regular updates to a database table in SSIS. The table has foreign key constraints.
I have a package running every week, and I have to update the data in the table from a flat file. Most of the contents are the same with update values and other new rows.
UPDATE : My data file contains updated contents ( some rows missing, some rows added, some modified ). The data file does not have the Primary keys ( I create the primary keys when I first bulk insert the data from the data file ), on subsequent SSIS package runs, I need to update the table with new data file contents.
e.g.
table
---------------------------------------------
1 Mango $0.99
2 Apple $0.59
3 Orange $0.33
data file
---------------------------------------------
Mango 0.79
Kiwi 0.45
Banana 0.54
How would I update the table with data from the file. The table has foreign key constraints with other tables.
another approach, to load massive group data instead of dealing row by row:
On database
create an staging table (e.g. StagingTable [name], [price])
Create a procedure (you may need to change the objects names, and add
transaction control and error handling etc just a draft):
create procedure spLoadData
as
begin
update DestinationTable
set DestinationTable.Price = StagingTable.Price
from DestinationTable
join StagingTable
on DestinationTable.Name = StagingTable.Name
insert into DestinationTable
(Name, Price)
select Name, Price
from StagingTable
where not exists (select 1
from DestinationTable
where DestinationTable.name = StagingTable.Name)
end
On SSIS
Execute SQL Task with (truncate [staging_table_name])
Data Flow task transferring from your Flat File to the Staging Table
Execute SQL Task calling the procedure you created (spLoadData).
Following are the few thoughts/steps:
Create a Flat File Connection manger.
Take Data flow task.
Create Flat File Source with connection manager just created.
Take lookup transformation(s) as many as you need to get FK values based on your source file values.
Take a lookup transformation after all above lookups, to get all values from Destination table.
Keep Conditional split and compare source values and destination values.
If all columns matched then UPDATE, else INSERT.
Map above conditional split results accordingly to OLEDB Destnation/OLEDB Command.
Give a try and let me know the results/comments.