I'm creating a SSIS package to load data from a CSV file to SQL table. The sample CSV file is
EMP_ID,EMP_NAME,DEPT_ID,MANAGER_ID,SALARY
1801,SCOTT,20,1221,3000
1802,ALLEN,30,1221,3400
I need to load data into a SQL Server table, but while loading I need to load Department Name and Manager Name instead of their IDs. So I need to convert the CSV source to
1801,SCOTT,FINANCE,JOHNSON,3000
1802,ALLEN,HR,JOHNSON,3400
The values for Department Name and Manager name come from the SQL Server database only. But how do I query and convert ID to text values?
I'm new to SSIS, please suggest how can I achieve this.
Thanks
John
CREATE PROCEDURE [dbo].[BulkInsert]
(
-- Declare Parameters here for your CSV file
)
AS
BEGIN
SET NOCOUNT ON;
declare #query varchar(max)
CREATE TABLE #TEMP
(
[FieldName] [int] NOT NULL ,
[FieldName] int NOT NULL,
)
SET #query = 'BULK INSERT #TEMP FROM ''' + PathOfYourTextFile + ''' WITH ( FIELDTERMINATOR = '','',ROWTERMINATOR = ''\n'')'
--print #query
--return
execute(#query)
BEGIN TRAN;
MERGE TableName AS Target
-- Now here you can get the value Department Name and Manager Name by using Target.Id --in the table from where you mant to get the value of the Manager Name
USING (SELECT * FROM #TEMP) AS Source
ON (Target.YourTableId = Source.YourTextFileFieldId)
-- In the above line we are checking if the particular row exists in the table(Table1) then update the Table1 if not then insert the new row in Table-1.
WHEN MATCHED THEN
UPDATE SET
Target.SomeId= Source.SomeId
WHEN NOT MATCHED BY TARGET THEN
-- Insert statement
The above code is just an example for you by taking the help from this you can edit in your code. And one more important thing for you, Bulk Insert is one of the great way to save the CSV files. So try to use this..:)
In SSIS package from Data Flow tab use LOOKUP process from the Toolbox. You'll specify the table to get your string values from and which columns to use for the join and the column to substitue your IDs with.
Related
How can I find the specific value from all stored procedures in my SQL Server database?
To be more specific, I want to know the value that is inserted into the specified column on the specified table.
For example:
A database has 3 stored procedures; dbo.sp_1, dbo.sp_2, dbo.sp_3
And that database also has a [Log] table which has a [number] column
dbo.sp_1
INSERT INTO [dbo].[Log] ([number])
VALUES (1);
dbo.sp_2
INSERT INTO [dbo].[Log] ([number])
VALUES (2);
dbo.sp_3
INSERT INTO [dbo].[Log] ([number])
VALUES (4);
So, the query results I expect are as follows:
I found a snippet that was used for a somewhat-similar task, however, I did not have to parse values. This may get you close if you really have to parse the sql. Sorry, the rest of the parsing will be left up to you
DECLARE #NonEscapeTextToFindLike NVARCHAR(MAX) = 'INSERTINTO\[dbo\].\[LOG\](\[number\])VALUES('
DECLARE #NonEscapeTextToFind NVARCHAR(MAX) = 'INSERTINTO[dbo].[LOG]([number])VALUES('
;WITH Procs AS
(
SELECT
StoredProcedure = name,
StoredProcedureText = OBJECT_DEFINITION(object_id),
NoBlankText = REPLACE(REPLACE(REPLACE(REPLACE(OBJECT_DEFINITION(object_id),' ',''),CHAR(9),''),CHAR(10),''),CHAR(13),'')
FROM
sys.procedures
)
SELECT
*,
StartOfPossibleInt = CHARINDEX(#NonEscapeTextToFind, Procs.NoBlankText) + LEN(#NonEscapeTextToFind)
FROM
Procs
WHERE
Procs.NoBlankText LIKE '%'+#NonEscapeTextToFindLike+'%' ESCAPE '\'
I am currently working on getting a set of records from a view in the Oracle database and trying to insert/update them in to the table in the SQL Server table depending on a column using BizTalk.
For this I created a stored procedure:
Create PROCEDURE [dbo].[uspInsertorUpdateDepartment]
#dept_name varchar(64),
#jax_dept_id char(32)
AS
BEGIN
SET NOCOUNT ON;
IF (SELECT TOP (1) 1 FROM afm.[jax_dept]
WHERE jax_dept_id = #jax_dept_id) IS NULL
INSERT INTO afm.[jax_dept](dept_name, jax_dept_id)
VALUES (#dept_name,#jax_dept_id)
ELSE
UPDATE afm.[jax_dept]
SET dept_name = #dept_name
WHERE jax_dept_id = #jax_dept_id
END
I created the schema for the stored procedure using consume adapter service. Used them in the mapping and the orchestration. Though I was not able to use the lopping functoid in the mapping
So removed the lopping and deployed the application. And tried to run and it ran without any error but just insert the first record from the oracle view in to the SQL Server database leaving all the other records. How can this be approached so the entire set of records from the oracle is inserted/updated in to SQL Server database.
Here I converted the separate update and insert into one merge statement:
Create PROCEDURE [dbo].[uspInsertorUpdateDepartment]
#dept_name varchar(64),
#jax_dept_id char(32)
AS
BEGIN
SET NOCOUNT ON;
merge afm.[jax_dept] as target
using (select #dept_name as dept_name, #jax_dept_id as jax_dept_id) as source
on source.jax_dept_id = target.jax_dept_id
when matched then
update target
SET dept_name = #dept_name
when not matched then
insert (dept_name, jax_dept_id)
values (#dept_name,#jax_dept_id)
;
END
Use table type as a parameter for the SP, instead of passing individually. We can
use looping functoid if we use User Defined Table value as a parameter.
CREATE TYPE dbo.SampleType AS TABLE
(
dept_name varchar(64) not null,
jax_dept_id char(32) not null
)
---
Create PROCEDURE [dbo].[uspInsertorUpdateDepartment]
#TVP dbo.SampleType READONLY
AS
BEGIN
SET NOCOUNT ON;
--your insert or update query
For more infor on how to use table value parameter check out this link:-
https://learn.microsoft.com/en-us/sql/relational-databases/tables/use-table-valued-parameters-database-engine
Any ideas/suggestions appreciated....
I've been asked to come up with a simple way to import new data we receive from an outside vendor (text files). We get several text files and each needs to be imported into its own table. Some tables have to have the current/existing data moved into a table called TABLENAME_Previous (to work with various existing reports), then have the current table emptied out and the new data imported into it. Also, any data now in the "previous" table has to be appended to an archive table.
Here's an example:
customer.txt comes in from vendor....
First we move the contents of customers_previous to customers_arch
Next we move the contents of customers to customers_previous
Finally we import the new customers.txt file into the table customers
Has anyone ever written a SQL routine to do this, or knows where to find one, that wouldn't be too painful to modify?
Thanks
you may try something like this:
To copy your previous data to Archive
Insert into customers_arch select * from customers_previous
To Copy your Customer Data to Previous:
truncate table customers_previous;
insert into customers_previous select * from customers
Then to Load you text file use Bulk Insert to load your customer table after clearing it.
truncate table customers;
bulk insert customers
from 'd:\yourfolder\customers.txt'
WITH
(
FIELDTERMINATOR =',',
ROWTERMINATOR ='\n'
);
UPDATE:
Ok, Brian, to answer your other question, How to run it for multiple files saved in your WeeklyTable.
Suppose your WeeklyTable is like this:
Declare #WeeklyTable TABLE(ID int Identity(1,1), [FileName] varchar(50))
insert into #WeeklyTable Values
('Customers'),('Orders'), ('Order_Details')
You can create a dynamic query to run your script for each file.
Declare #Template varchar(max)
Set #Template = '
-- Start of [[FILENAME]] --------------------
Insert into [FILENAME]_arch select * from [FILENAME]_previous
GO
truncate table [FILENAME]_previous;
insert into [FILENAME]_previous select * from [FILENAME]
GO
truncate table [FILENAME];
bulk insert [FILENAME]
from ''d:\yourfolder\[FILENAME].txt''
WITH
(
FIELDTERMINATOR ='','',
ROWTERMINATOR =''\n''
);
'
Declare #s varchar(max)
Declare #FileName varchar(50)
Declare #ID int =0
Select TOP 1 #ID=ID, #FileName=[FileName] From #WeeklyTable Where ID>#ID order by ID
While ##ROWCOUNT>0 Begin
Set #s = REPLACE(#Template, '[FILENAME]', #FileName)
Print #s
-- EXEC(#s) -- Uncomment to EXEC the script.
Select TOP 1 #ID=ID, #FileName=[FileName] From #WeeklyTable Where ID>#ID order by ID
End
I am new to SQL, but am trying to learn its logic, I am assuming bulk insert will insert on all rows in this case a blob. (pdf file) below is my code but what I am trying to accomplish is, inserting a pdf file that I have put on SQL server in a row that has a matching Primary Key that I specify. So far I am missing the where clause to specify the PK
Declare #sql varchar(max)
Declare #filePath varchar(max)
Set #filePath = 'C:\iphone.pdf'
Set #sql='INSERT INTO HDData.dbo.PurchasedCellPhoneInfo(Receipt) SELECT * FROM OPENROWSET(BULK '''+ #filePath+''', SINGLE_BLOB) AS BLOB'
exec(#sql)
May I use an update t-SQL query instead of insert? and how would I drop the where to specify a specific row I want to insert this blob in?
Any help would be appreciated.
I also have tried this, following #misterPositive's suggestion for update query:
Declare #criteria varchar(50)
SET #criteria ='352014075399147'
UPDATE HDData.dbo.PurchasedCellPhoneInfo SET Receipt =
(SELECT Receipt FROM OPENROWSET (BULK 'C:\352014075399147.pdf, SINGLE_BLOB') a)
WHERE(IMEI = #criteria)
i do recieve this message:
Either a format file or one of the three options SINGLE_BLOB, SINGLE_CLOB, or SINGLE_NCLOB must be specified. i like this update query as it seems to fit what im trying to do.
You can do this to UPDATE:
UPDATE MyTable
SET blobField =
(SELECT BulkColumn FROM OPENROWSET (BULK 'C:\Test\Test1.pdf', SINGLE_BLOB) a)
WHERE (CriteriaField = #criteria)
Here is another way for PK
CREATE VIEW [dbo].[VWWorkDataLoad]
AS
SELECT RecordLine
FROM [dbo].[WorkDataLoad];
Now BULK INSERT should then look like:
BULK INSERT [dbo].[VWWorkDataLoad] FROM 'D:\NPfiles\TS082114.trn'
WITH (FIRSTROW = 2,FIELDTERMINATOR = ',' , ROWTERMINATOR = '\n');
If you want to insert new records then you could have an identity column for your PK and not have to worry about it. I have also seen functions used when a table is designed without identity on PK. Something like GetTableNamePK() in the select list.
If you want to update an existing record then you will want a where clause as you mentioned. This worked for me in testing:
Update TestBlob Set BinaryData = (SELECT * FROM OPENROWSET(BULK 'c:\temp\test.pdf', SINGLE_BLOB) AS BLOB)
Where ID = 2
If you do not want to use the Identity or Function this worked where ID is a primary key and I want to insert the BLOB with PK of 3:
INSERT INTO TestBlob2 (ID, BinaryData) SELECT 3, * FROM OPENROWSET(BULK 'c:\temp\test.pdf', SINGLE_BLOB) AS BLOB
Before you say me,
I put retainsameconnection, I had put.
and I have delay validation on ALL my components too
after in others query where I use this dynamic temp table
this fails (sometimes) saying temp table does not exist.
How can I solve it?
This is my query..
declare #usuario varchar(100)
declare #contra varchar(100)
declare #servidor varchar(100)
declare #based varchar(100)
declare #nombretabla varchar(100)
set #usuario ='pruebas'
set #contra ='123'
set #servidor ='1.6.7.9'
set #based ='op'
set #nombretabla='tablaBC245B7A910D4B488CC9EFF0EFD3C177'
/*this was a dynamic name got with this query in other execute sql task
declare #tabla varchar(50)
set #tabla=NEWID ()
set #tabla =REPLACE (#tabla ,'-','')
select 'tabla'+#tabla nombre
*/
declare #cadena varchar(500)
set #cadena ='SERVER='+#servidor+';DataBasE='+#based +
';uid='+#usuario +';pwd='+#contra+';'
set #cadena =''''+#cadena +''''
while OBJECT_ID('tempdb..##envases'+#nombretabla) IS NULL
begin
exec('
SELECT * into ##envases'+#nombretabla+'
FROM OPENROWSET(
''SQLNCLI'',
'+#cadena+',
''
select pro=p.pro,envase=e.pro from procorp p
join procorp e on p.envase=e.envase and e.esenvase=1
union all select pro=-1,envase=-1
'')
'
)
end
IF OBJECT_ID('tempdb..#primera') IS NOT NULL DROP TABLE #primera
IF OBJECT_ID('tempdb..#end') IS NOT NULL DROP TABLE #end
create table #primera(
pro int,
envase int
)
insert into #primera
exec('select * from ##envases'+#nombretabla+'')
select fila='select pro='+CONVERT(varchar(10),pro)+
', envase='+CONVERT(varchar(10),envase)+' union all '
into #end
from #primera
--select * from #end
--drop table #end
--drop table #primera
alter table #end
add
uno int not null default(1),
id int identity
declare #maximo int
set #maximo =(select MAX(id) from #end )
update #end set fila=fila+'terminado' where id=#maximo
update #end set fila=replace(fila,'union all terminado','') where id=#maximo
select uno, ' begin try drop table #tmpenvase end try begin catch end catch
select * into #tmpenvase from ( '+
convert(varchar(max),STUFF(
(select '', convert(varchar(max),fila) from #end order by id
for xml path('') )
,1,0,'') )+' ) q 'cadena
from #end e
group by uno
I would use standard SSIS objects for this.
I cant really follow what the end objective of your script is, but generally:
Connection strings can be managed using Configuration files, Variables and other means.
Within Data Flow Tasks, OLE DB Source Transformations can run SELECT statements, OLE DB Destinations can deliver data, OLE DB Commands can run UPDATE statements.
I would reimagine your requirements in terms of a series of tasks, rather than trying to force everything into a single T-SQL script.
Global temporary tables are visible to any user and any connection after they are created, and are deleted when all users that are referencing the table disconnect from the instance of SQL Server.
According to that paragraph of SQL Server Books-Online; as soon as SQL Server don't found any connection to a global temporary tables those temporary tables will deleted, and when you are create a local temporary table with EXEC command SQL Server will delete that local temporary table this will also applied to a global temporary table but after a checking for all users connections and so on. I think sometimes SQL Server found your global temporary table alone before the your next statement commits.
I suggest you to use local table variables and use INSERT EXEC to fill them.
And also there are some other ways too.