SQL Server: Importing and archiving weekly data - sql-server

Any ideas/suggestions appreciated....
I've been asked to come up with a simple way to import new data we receive from an outside vendor (text files). We get several text files and each needs to be imported into its own table. Some tables have to have the current/existing data moved into a table called TABLENAME_Previous (to work with various existing reports), then have the current table emptied out and the new data imported into it. Also, any data now in the "previous" table has to be appended to an archive table.
Here's an example:
customer.txt comes in from vendor....
First we move the contents of customers_previous to customers_arch
Next we move the contents of customers to customers_previous
Finally we import the new customers.txt file into the table customers
Has anyone ever written a SQL routine to do this, or knows where to find one, that wouldn't be too painful to modify?
Thanks

you may try something like this:
To copy your previous data to Archive
Insert into customers_arch select * from customers_previous
To Copy your Customer Data to Previous:
truncate table customers_previous;
insert into customers_previous select * from customers
Then to Load you text file use Bulk Insert to load your customer table after clearing it.
truncate table customers;
bulk insert customers
from 'd:\yourfolder\customers.txt'
WITH
(
FIELDTERMINATOR =',',
ROWTERMINATOR ='\n'
);
UPDATE:
Ok, Brian, to answer your other question, How to run it for multiple files saved in your WeeklyTable.
Suppose your WeeklyTable is like this:
Declare #WeeklyTable TABLE(ID int Identity(1,1), [FileName] varchar(50))
insert into #WeeklyTable Values
('Customers'),('Orders'), ('Order_Details')
You can create a dynamic query to run your script for each file.
Declare #Template varchar(max)
Set #Template = '
-- Start of [[FILENAME]] --------------------
Insert into [FILENAME]_arch select * from [FILENAME]_previous
GO
truncate table [FILENAME]_previous;
insert into [FILENAME]_previous select * from [FILENAME]
GO
truncate table [FILENAME];
bulk insert [FILENAME]
from ''d:\yourfolder\[FILENAME].txt''
WITH
(
FIELDTERMINATOR ='','',
ROWTERMINATOR =''\n''
);
'
Declare #s varchar(max)
Declare #FileName varchar(50)
Declare #ID int =0
Select TOP 1 #ID=ID, #FileName=[FileName] From #WeeklyTable Where ID>#ID order by ID
While ##ROWCOUNT>0 Begin
Set #s = REPLACE(#Template, '[FILENAME]', #FileName)
Print #s
-- EXEC(#s) -- Uncomment to EXEC the script.
Select TOP 1 #ID=ID, #FileName=[FileName] From #WeeklyTable Where ID>#ID order by ID
End

Related

Create a single table by importing all CSV files in a folder?

I have around 30-40 CSV files in a folder. For example, suppose folder 'Florida' has customer information from different cities of state Florida. each CSV file has customer information of one city. Now I want to create a table in SQL Server by importing all the CSV files from that folder to create a table for all customers in Florida. I wanted to know if there is any way I could perform this action for all CSV files at once. I am using SQL Server Management Studio (SSMS).
All the CSV files have same column names.
I am doing the following for one CSV file:
CREATE TABLE sales.cust (
Full_name VARCHAR (100) NOT NULL,
phone VARCHAR(50),
city VARCHAR (50) NOT NULL,
state VARCHAR (50) NOT NULL,
);
BULK INSERT sales.cust
FROM 'C:\Users..............\cust1.csv'
WITH
(
FIRSTROW = 2,
FIELDTERMINATOR = ',', --CSV field delimiter
ROWTERMINATOR = '\n', --Use to shift the control to next row
ERRORFILE = 'C:\Users\..............\cust1ErrorRows.csv',
TABLOCK
)
Suggestion to use command prompt only because of limited tools.
I thought of another solution you can use that could help you out and make it so you only have to import one file.
Create your table:
CREATE TABLE sales.cust (
Full_name VARCHAR (100) NOT NULL,
phone VARCHAR(50),
city VARCHAR (50) NOT NULL,
state VARCHAR (50) NOT NULL,
);
Using command Prompt do the following:
a. Navigate to your directory using cd "C:\Users..............\"
b. Copy the files into one giant file using:
copy *.csv combined.csv
Import that file using GUI in SSMS
Deal with the headers
delete from sales.cust where full_name = 'Full_name' and phone = 'phone'
You can only do this because all columns are varchar.
Here is one route to get all the files into a table---
-- from Rigel and froadie # https://stackoverflow.com/questions/26096057/how-can-i-loop-through-all-the-files-in-a-folder-using-tsql
-- 1.Allow for SQL to use cmd shell
EXEC sp_configure 'show advanced options', 1 -- To allow advanced options to be changed.
RECONFIGURE -- To update the currently configured value for advanced options.
EXEC sp_configure 'xp_cmdshell', 1 -- To enable the feature.
RECONFIGURE -- To update the currently configured value for this feature.
-- 2.Get all FileNames into a temp table
--for repeatability when testing in SMSS, delete any prior table
IF OBJECT_ID('tempdb..#tmp') IS NOT NULL DROP TABLE #tmp
GO
CREATE TABLE #tmp(csvFileName VARCHAR(100));
INSERT INTO #tmp
EXEC xp_cmdshell 'dir /B "C:\\Users..............\\*.csv"';
-- from Chompy #https://bytes.com/topic/sql-server/answers/777399-bulk-insert-dynamic-errorfile-filename
-- 3.Create sql prototype of the Dynamic sql
---- with CSV field delimiter=',' and CSV shift the control to next row='\n'
DECLARE #sqlPrototype nvarchar(500)
SET #sqlPrototype = N'BULK INSERT sales.cust
FROM ''C:\\Users..............\\xxxx''
WITH ( FIRSTROW = 2,
FIELDTERMINATOR = '','',
ROWTERMINATOR = ''\n'',
ERRORFILE = ''C:\\Users..............\\xxxx_ErrorRows.txt'',
TABLOCK)'
-- 4.Loop through all of the files
Declare #fileName varchar(100)
While (Select Count(*) From #tmp where csvFileName is not null) > 0
Begin
Select Top 1 #fileName = csvFileName From #tmp
-- 5.Replace real filename into prototype
PRINT(#filename)
DECLARE #sqlstmt nvarchar(500)
Set #sqlstmt = replace(#sqlPrototype, 'xxxx', #filename)
--print(#sqlstmt)
-- 6.Execute the resulting sql
EXEC sp_executesql #sqlstmt;
-- 4A.Remove FileName that was just processed
Delete from #tmp Where csvFileName = #FileName
End
Caution--If ErrorFile exists, then BulkInsert will fail.

Copy entire SQL table to another and truncate original table

I am writing a stored procedure that will copy the entire contents of a table called "CS_Consolidation" into a backup table called "CS_ConsolidationBackup2016" all fields are exactly the same and the new data everyday must just be added after which the original table must be truncated.
I am however having a problem with my procedure and how it is written if anyone can help:
CREATE PROCEDURE BackUpData2
AS
BEGIN
SET NOCOUNT ON;
SELECT *
INTO [dbo].[CS_ConsolidationBackUp]
FROM [dbo].[CS_Consolidation]
TRUNCATE TABLE [dbo].[CS_Consolidation]
GO
Why do you want to copy the data and then delete the original? This is entirely more complicated and stressful to the system then you need. There is no need to create a second copy of the data so that you can just turn around and drop the first copy.
A much easier path would be to rename to current table and then create you new primary table.
EXEC sp_rename 'CS_Consolidation', 'CS_ConsolidationBackUp';
GO
select *
into CS_Consolidation
from CS_ConsolidationBackUp
where 1 = 0; --this ensures no rows but the entire structure is copied.
If you are looking to create one backup table daily, would something like this work?
DECLARE #BackupTableName nvarchar(250)
SELECT #BackupTableName = 'CS_ConsolidationBackUp' + CAST(CONVERT(date, getdate()) as varchar(250))
IF EXISTS(SELECT TABLE_NAME FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_NAME = #BackupTableName)
BEGIN
EXEC('DROP TABLE [' + #BackupTableName + ']')
END
EXEC('SELECT * INTO [dbo].[' + #BackupTableName + '] FROM [dbo].[CS_Consolidation]')
TRUNCATE TABLE [dbo].[CS_Consolidation]
You are missing and "end" statement before "go". This is the correct code:
CREATE PROCEDURE BackUpData2
AS
BEGIN
SET NOCOUNT ON;
SELECT *
INTO [dbo].[CS_ConsolidationBackUp]
FROM [dbo].[CS_Consolidation]
TRUNCATE TABLE [dbo].[CS_Consolidation]
end
GO

SQL inserting from csv with converting filepath to binary image value at the same time

I'm trying to make sql script that will add to a table some rows with different data types in each field.
table has 4 columns - ID, first name, last name, image
I have a csv file with about 4000 rows with those columns (last one as a path to image file):
ex.
1,'John','Smith','d:\images\johnsmith.jpg'
I can add single row with this code:
INSERT INTO [Employee] ([ID],[First_Name],[Last_Name],[Image])
SELECT
1 AS [ID]
,'John' AS [First_Name]
,'Smith' AS [Last_Name]
, * FROM OPENROWSET(BULK 'd:\images\johnsmith.jpg', SINGLE_BLOB) AS [Image];
GO
But I would like to import this that like this
BULK INSERT [Employee]
FROM 'd:\tmp\data.csv'
WITH
(
FIRSTROW = 2,
FIELDTERMINATOR = ',', --CSV field delimiter
ROWTERMINATOR = '\n', --Use to shift the control to next row
TABLOCK
)
GO
however with this method I cannot not convert filepath to image data type.
Maybe there is a way to run a command saved in csv fil, so I could make the file something like:
1,'John','Smith','* FROM OPENROWSET(BULK 'd:\images\johnsmith.jpg', SINGLE_BLOB)'
Is there any simple way to do this? I'm not very good with sql
First of all - Be sure not to use the IMAGE datatye, use varbinary(max)
https://msdn.microsoft.com/en-us/library/ms187993.aspx
Second of all, you will run into a problem with OPENROWSET. It does not allow for variables in its parameters, so you cannot make the
, * FROM OPENROWSET(BULK 'd:\images\johnsmith.jpg', SINGLE_BLOB) AS [Image];
query work inside a select (with different files)
For a brute force solution, do it in two steps. First load the CSV into a table.
BULK INSERT [#TempEmployee]
FROM 'd:\tmp\data.csv'
WITH
(
FIRSTROW = 2,
FIELDTERMINATOR = ',', --CSV field delimiter
ROWTERMINATOR = '\n', --Use to shift the control to next row
TABLOCK
)
Then create the bulkloads dynamically:
create table #TempEmployee (
[ID] nvarchar(max),
[First_Name] nvarchar(max),
[Last_Name] nvarchar(max),
[FilePath] nvarchar(max))
Declare #SQL nvarchar(max)
Declare TempEmployees CURSOR FOR
SELECT N'
INSERT INTO [Employee] ([ID],[First_Name],[Last_Name],[Image])
SELECT
'''+[ID]+''' AS [ID]
,'''+[First_Name]+''' AS [First_Name]
,'''+[Last_Name]+''' AS [Last_Name]
, * FROM OPENROWSET(BULK '''+[FilePath]+''', SINGLE_BLOB) AS [Image];' as [SQL]
FROM #TempEmployee
OPEN TempEmployees
FETCH NEXT FROM TempEmployees
INTO #SQL
WHILE ##FETCH_STATUS = 0
BEGIN
exec (#SQL);
FETCH NEXT FROM TempEmployees
INTO #SQL
END
CLOSE TempEmployees
DEALLOCATE TempEmployees
DROP TABLE #TempEmployee
Another solution would be to create a filetable in the database, and compy all of the image files to the filetable. Now load the csv file, and you have the employee data including filename and path to image in the Employee table and the images in the Filetable.
For an overview of Filetables see here:
http://www.databasejournal.com/features/mssql/filestream-and-filetable-in-sql-server-2012.html
You've got to have SQL Server 2012 or later for that to work

Bulk insert with many files inside folder

I want to read xml files with sql server. I show below how I do it.
DECLARE #testxml TABLE (IntCol int, XmlCol xml);
INSERT INTO #testxml(XmlCol)
SELECT * FROM OPENROWSET(
BULK 'C:\XMLs\32056963_0001515351.xml',
SINGLE_BLOB) AS x;
SELECT * FROM #testxml
All is ok. But I need to read many files inside a folder, so I'm using:
EXEC master.sys.xp_dirtree 'C:\XMLs\',0,1;
But how can I doing a dynamic bulk insert in order to insert all xml files in the folder to #testxml?
I don't know if there is some way to do a bulk insert of all the files at once. I would suggest to execute your import query for each file, using dynamic queries. But in order to be able to fetch the data from the main query, you should insert the data in a temporary table, because the scope of the table variable will be limited to the dynamic query.
-- Get the file names
CREATE TABLE #files (
subdirectory NVARCHAR(255),
depth INT,
file BIT
)
INSERT INTO #files
EXEC master.sys.xp_dirtree 'C:\XMLs\',0,1;
-- Iterate through the XML files
DECLARE #filesCursor CURSOR;
SET #filesCursor = CURSOR FOR
SELECT subdirectory
FROM #files
WHERE file=1 AND LEN(subdirectory)>4 AND LOWER(RIGHT(subdirectory,4))='.xml'
DECLARE #fileName NVARCHAR(255), #query NVARCHAR(MAX);
FETCH NEXT FROM #filesCursor INTO #fileName;
-- Temporary table to store the data
CREATE TABLE #testxml (IntCol int, XmlCol xml);
WHILE ##fetch_status = 0
BEGIN
-- Build and execute the query for each file
SET #query = 'INSERT INTO #testxml(XmlCol) SELECT * FROM OPENROWSET(BULK ''C:\XMLs\' + #fileName + ''',SINGLE_BLOB) AS x';
EXECUTE sp_executesql #query;
FETCH NEXT FROM #filesCursor INTO #fileName;
END
-- Closing and deallocating cursor
CLOSE #filesCursor;
DEALLOCATE #filesCursor;
-- Get the data from the temp table into your table variable.
-- If it is not necessary to use a table variable, you could read
-- the data directly from the temp table
DECLARE #testxml TABLE (IntCol int, XmlCol xml);
INSERT INTO #testxml
SELECT * FROM #testxml;
-- Deleting temp tables, as they won't be used anymore
DROP TABLE #testxml;
DROP TABLE #files;

SSIS : how to convert the source column from ID to Value

I'm creating a SSIS package to load data from a CSV file to SQL table. The sample CSV file is
EMP_ID,EMP_NAME,DEPT_ID,MANAGER_ID,SALARY
1801,SCOTT,20,1221,3000
1802,ALLEN,30,1221,3400
I need to load data into a SQL Server table, but while loading I need to load Department Name and Manager Name instead of their IDs. So I need to convert the CSV source to
1801,SCOTT,FINANCE,JOHNSON,3000
1802,ALLEN,HR,JOHNSON,3400
The values for Department Name and Manager name come from the SQL Server database only. But how do I query and convert ID to text values?
I'm new to SSIS, please suggest how can I achieve this.
Thanks
John
CREATE PROCEDURE [dbo].[BulkInsert]
(
-- Declare Parameters here for your CSV file
)
AS
BEGIN
SET NOCOUNT ON;
declare #query varchar(max)
CREATE TABLE #TEMP
(
[FieldName] [int] NOT NULL ,
[FieldName] int NOT NULL,
)
SET #query = 'BULK INSERT #TEMP FROM ''' + PathOfYourTextFile + ''' WITH ( FIELDTERMINATOR = '','',ROWTERMINATOR = ''\n'')'
--print #query
--return
execute(#query)
BEGIN TRAN;
MERGE TableName AS Target
-- Now here you can get the value Department Name and Manager Name by using Target.Id --in the table from where you mant to get the value of the Manager Name
USING (SELECT * FROM #TEMP) AS Source
ON (Target.YourTableId = Source.YourTextFileFieldId)
-- In the above line we are checking if the particular row exists in the table(Table1) then update the Table1 if not then insert the new row in Table-1.
WHEN MATCHED THEN
UPDATE SET
Target.SomeId= Source.SomeId
WHEN NOT MATCHED BY TARGET THEN
-- Insert statement
The above code is just an example for you by taking the help from this you can edit in your code. And one more important thing for you, Bulk Insert is one of the great way to save the CSV files. So try to use this..:)
In SSIS package from Data Flow tab use LOOKUP process from the Toolbox. You'll specify the table to get your string values from and which columns to use for the join and the column to substitue your IDs with.

Resources