My goal is to take raw data I'm receiving from a file path and insert that into a new table or temporary table. The problem is there are no headers for the data I am getting. The file is a flat file with tab delimiters.
Information:
The file names all start with the same substring: DBA_NG_INBOUND_ELIG_
And the file path is the same for all files: \DBAediarchive\shared\edi\ediarchive\Inbound\TPV\Eligibility
Ideally I'd like to just store the last 7 days worth of files in a temporary table and delete that data and replace it every 7 days. But maybe I make a stored procedure to accomplish that (or all of this)
My solution would be something like this (there will be more than 1 column and I can refine the data types for the columns as well:
CREATE TABLE #TempTable (
Column1 VARCHAR(50),
Column2 VARCHAR(50),
Column3 VARCHAR(50),
Column4 VARCHAR(50),
Column5 VARCHAR(50),
Column6 VARCHAR(50),
Column7 VARCHAR(50),
Column8 VARCHAR(50),
Column9 VARCHAR(50),
Column10 VARCHAR(50)
)
BULK INSERT #TempTable
FROM '\\DBAediarchive\shared\edi\ediarchive\Inbound\TPV\Eligibility\DBA_NG_INBOUND_ELIG_*.txt'
WITH (
FIELDTERMINATOR = '\t',
ROWTERMINATOR = '\n',
FIRSTROW = 1
);
Is this an efficient solution? Additionally if someone has a stored procedure method that would pick up only 7 days worth of files from that path that would be an ideal solution.
Related
This is the sample procedure I am using.
create procedure PRO_ProcName
#description varchar(max),
#txn_no varchar
as
begin
declare #txn table (
id bigint,
description varchar(max),
txn_no varchar
);
declare #txn_id bigint;
insert into transactions
(
description,
txn_no
)
output
inserted.description,
inserted.txn_no
into #txn
values
(
#description,
#txn_no
)
select #txn_id = id from #txn;
end
I am getting an error like:
Column name or number of supplied values does not match table definition.
I know that it is because I have id field in my temporary table and it is not getting inserted in the insert into statement. I cannot give value for id because it is the auto increment primary key.
How can I tackle this situation and get id of inserted record into a variable?
The inserted table represents the data that exists in the target table after the insert - so it also includes the auto-generated values, whether they where generated by a default value definition or by an identity definition on the columns - so you need to add inserted.id to the output clause.
However, there are two more things wrong in your procedure.
The first and most important is the fact that you didn't specify a length to the #txn_no varchar parameter. SQL Server will implicitly specify the length of 1 char in this case.
The second is the fact that you are not specifying the columns list of #txn in the output clause.
Here is a improved version of your code with all these issues fixed:
create procedure PRO_ProcName
#description varchar(max),
#txn_no varchar(255) -- Note: I don't know the actual length you need
as
begin
declare #txn table (
id bigint,
description varchar(max),
txn_no varchar
);
declare #txn_id bigint;
insert into transactions
(
description,
txn_no
)
output
inserted.id,
inserted.description,
inserted.txn_no
into #txn(id, description, txn_no)
values
(
#description,
#txn_no
)
select #txn_id = id from #txn;
end
I cannot give value for id because it is the auto increment primary key.
No it isn't. You haven't declared it to be anything of the sort. So we need to fix that first:
declare #txn table (
id bigint IDENTITY PRIMARY KEY,
description varchar(max),
txn_no varchar
);
And then we fix it by specifying a column list in your INTO clause:
output
inserted.description,
inserted.txn_no
into #txn (description, txn_no)
It's always a good habit to specify column lists anyway.
Or if I've misinterpreted your question, and the id should be coming from transactions, then you just add inserted.id as another column in your OUTPUT clause. inserted represents that state of the table after the insert. So you can include columns from it in your OUTPUT clause even if you didn't specify them in the INSERT.
I want to create a temporary table from another table dynamically creating stored procedure.
I have a table named as Tblstructure. This table has only 1 column Colname. Data in the column is created at run time and is going to change every time. For time being I have data
Colname
-------
Region
Country
State
Metric
Value
I want to use the above table and create a dynamic table having column as
Region Country state Metric value.
Whenever users call the stored procedure, Procedure call the table tblstructure pull out all the rows from the table and transform them into column header.
How to do this?
I think what you need to look into are patterns for concatenating text stored across the rows in a table. This resource lays out some common patterns you should read about: https://www.simple-talk.com/sql/t-sql-programming/concatenating-row-values-in-transact-sql/
That said, the XML path pattern is the one I most frequently employ.
CREATE TABLE Tblstructure (Colname varchar(50));
INSERT INTO Tblstructure (Colname)
VALUES ('Region'), ('Country'), ('State'), ('Metric'), ('Value');
DECLARE #ColumnDeclaration VARCHAR(2000) =
STUFF(
(SELECT ', ' + Colname + ' VARCHAR(50) '
FROM Tblstructure
FOR XML PATH(''), type).value('.', 'varchar(max)')
, 1, 1, '')
SELECT 'CREATE TABLE SomeTempTable (' + #ColumnDeclaration + ');'
Output: CREATE TABLE SomeTempTable ( Region VARCHAR(50) , Country VARCHAR(50) , State VARCHAR(50) , Metric VARCHAR(50) , Value VARCHAR(50) );
Notice that I had to make an assumption about the type that should represent each field. I have assumed each field is a VARCHAR(50), but you would probably want to figure out a better way of inferring what type each field should be.
Working example: http://sqlfiddle.com/#!6/d4434/23
Edit
Also, keep in mind that this example wouldn't actually create the table, it just gives you the dynamic create table statement. To execute this, you would want to feed the resulting create table statement into sp_executesql.
When I import a CSV or text file and bulk insert it into my database, the process successfully adds all record to the table.
My problem is that the inserted string is in Arabic, which appears as symbols in my database table. How can i solve this problem?
Insert using query
You need to choose an Arabic collation for your varchar/char columns or use Unicode (nchar/nvarchar).
CREATE TABLE MyTable
(
MyArabicColumn VARCHAR(100) COLLATE Arabic_CI_AI_KS_WS,
MyNVarCharColumn NVARCHAR(100)
)
Both columns should work.
Bulk Insert from file
This article explains how to bulk insert unicode characters.
Test Table
USE AdventureWorks2012;
GO
CREATE TABLE myTestUniCharData (
Col1 smallint,
Col2 nvarchar(50),
Col3 nvarchar(50)
);
Bulk Insert
DATAFILETYPE='widechar' allows the use of Unicode character format when bulk importing data.
USE AdventureWorks2012;
GO
BULK INSERT myTestUniCharData
FROM 'C:\myTestUniCharData-w.Dat'
WITH (
DATAFILETYPE='widechar',
FIELDTERMINATOR=','
);
GO
SELECT Col1,Col2,Col3 FROM myTestUniCharData;
GO
I have a .txt file in the format:
123456
111111
122222
123344
121212
I wish to insert these into a temporary table along with an integer recording the order in which they are in the .txt file, such as:
Index Number
---------------
1 123456
2 111111
3 122222
4 123344
5 121212
Currently I'm doing this by having an IDENTITY column in my temporary table and doing a BULK INSERT using a FORMATFILE like so:
CREATE TABLE #tbl
(
idx int NOT NULL IDENTITY,
ItemNumber nchar(6)
)
BULK INSERT #tbl
FROM 'd:\MyNumberList.txt'
WITH
(
FORMATFILE='d:\MyFormatFile.xml'
)
However, I'm hoping theres a way of achieving this without the need for a FORMATFILE file.
Is there a way of doing this?
You can do this with a view. However, because you are using a temporary table here, and it's not possible to create a view on a temporary table, you'd addtionally need to make use of a synonym.
If you were importing into a regular table, you wouldn't need the synonym:
create synonym tbl for #tbl
GO
create view vtbl
as
select ItemNumber from tbl
GO
bulk insert vtbl from 'd:\MyNumberList.txt'
GO
select * from #tbl
GO
Okay so I have a table ERD designed like so... for regular bulk inserts
(source: iforce.co.nz)
And a tab delimited \t text file with information about each customer (consists of about 100,000+ records).
# columnA columnB columnC
data_pointA data_pointB data_pointC
And a stored procedure that currently does its intended job fine.
CREATE PROCEDURE import_customer_from_txt_para #filelocation varchar(100)
AS BEGIN
TRUNCATE TABLE dbo.[customer_stg]
DECLARE #sql nvarchar(4000) = '
BULK INSERT customer_stg
FROM ''' + #filelocation + '''
WITH
(
FIRSTROW=14,
FIELDTERMINATOR=''\t'',
ROWTERMINATOR=''\n''
)';
print #sql;
exec(#sql);
END
But my question is about the relationship between customer_table and customer_stg is it possible to include a customer_id within the customer_stg bulk insert? with something like so? ( I'm not sure how to apply the foreign key parameter #customer_sk to the bulk insert ).
CREATE PROCEDURE import_customer_from_txt_para #filelocation varchar(100), #customer_sk int
AS BEGIN
TRUNCATE TABLE dbo.[customer_stg]
DECLARE #sql nvarchar(4000) = '
BULK INSERT customer_stg
FROM ''' + #filelocation + '''
WITH
(
FIRSTROW=14,
FIELDTERMINATOR=''\t'',
ROWTERMINATOR=''\n''
)';
print #sql;
exec(#sql);
END
Preferably after each bulk-insert I'd wish to be able to relate the data between the two tables.
(source: iforce.co.nz)
Bulk inserts will either insert NULL or the default value for unspecified column (based on the KEEPNULLS argument), which of course will not work for your situation assuming you have (or will create) a constraint. I assume this is the case because otherwise you could just update your table directly after you run the insert.
I see two ways around this:
- If you have the ability, you can just macro-edit the text file before you run the bulk insert. Since I'm assuming that that isn't in the question...
- First of all, you will need to add your FK column to your _stg table if it's not already there. Then, in your stored procedure, create a temp table with the three columns specified in the input file:
CREATE TABLE dbo.#Temp_STG
(
columnA,
columnB,
columnC
)
Then, batch insert into that table. Then you can insert from the temp table to the main _stg table, but add a column:
INSERT dbo.Customer_STG
SELECT
T.columnA,
T.columnB,
T.columnC,
[your customer key]
FROM dbo.#Temp_STG AS T
Make sure you drop the temp table when you're done.
As a side note, do you need to use dynamic SQL for this task? It's generally best to avoid unless absolutely necessary.
I suppose another option would be setting the default value for the column to whatever you want, and turning KEEPNULLS off. But, I would definitely NOT recommend doing this when you can just use the solution described above.
See more: http://msdn.microsoft.com/en-us/library/ms188365.aspx