As part of a SSIS package I have a SQL table containing the staging table name and corresponding 'real' table name. The staging table names will change based on the date but there is a previous step that works out what the Real_Table is.
How do I loop through each one in SQL and insert all the data from the staging tables (columns are identical in both) into the real table and update the flag from 0 to 1 to mark it was done. This is my table:
Staging_Table Real_Table UpdateFlag
Customers_01012018 Customers 0
Order_01012018 Order 0
Suppliers_02022018 Suppliers 0
You can use while loop to load the data into Real tables,
DECLARE #total INT
DECLARE #start INT
DECLARE #query NVARCHAR(MAX)
DECLARE #staging_table NVARCHAR(MAX)
DECLARE #real_table NVARCHAR(MAX)
SET #start = 1
SET #total = (SELECT COUNT(*) FROM Stg_tables)
WHILE(#start <= #total)
BEGIN
SELECT TOP 1 #staging_table = Staging_Table, #real_table = Real_Table FROM Stg_tables WHERE UpdateFlag = 0
SET #query = 'INSERT INTO ' + #real_table + ' SELECT * FROM ' + #staging_table
EXEC(#query)
UPDATE Stg_tables SET UpdateFlag = 1 WHERE Staging_Table = #staging_table AND Real_Table = #real_table
SET #start = #start + 1
END
An overview of how to go about this is below. Of course make sure to match the correct configurations to your environment and set the metadata appropriately where applicable.
Create an Execute SQL Task that selects the staging and real table
names. Choose the "Full result set" ResultSet. On the Result Set
page, add an object variable and set the Result Name to 0 to use the
immediate results.
Add a Foreach Loop that is of the Foreach ADO Enumerator type. Use the object variable from the Execute SQL Task as the ADO Object Source Variable. On the Variable Mappings page, add a string variable at index 1 and 0. One of these will hold the staging table name and the other will hold the real table name. These will align with the order they were selected in the Execute SQL Task, so if you selected the staging table first there use a the variable that will hold this name at index 0.
Create another string variable that contains an expression selecting the necessary columns that will be loaded from the staging table with the variable holding this table name concatenated within in. An Example of this follows with the #[User::StagingTableVariable] variable representing the variable holding the staging table name.
Within the Foreach Loop, add Data Flow Task then add an OLE DB Source inside this. This will use the variable with the SQL selecting from the staging table, but to correctly set the metadata choose the SQL Command option and use a SQL statement that selects the same columns from an existing staging table name. Once this is set, change the ValidateExternalMetadata property to false, choose "SQL command from variable" for the Data Access Mode, and pick the variable holding the SQL statement that uses the staging table name.
Add an OLE DB Destination and connect the source to this. Like before, use an existing real table and map the columns. After this, again set ValidateExternalMetadata to false, change Data Access Mode to the use a table or view name from variable (I would recommend the fast load option), and add the variable holding the real table name.
After the Data Flow Task add another Execute SQL Task that's linked to the DFT. Create a string variable with an update statement for the mapping table where the table names originate from. Set the SQLSourceType value to variable and select this variable for the SourceVariable property. If you're using a text column in the WHERE clause of the update statement, make sure that the expression contains single quotes (') as a typical SQL update statement would.
Example OLE DB Source Variable Expression:
"SELECT ColA, ColB, ColC from YourSchema." + #[User::StagingTableVariable]
Example Variable Expression Update Command:
"UPDATE YourSchema.MappingTable SET UpdateFlag = 1 where Real_Table = '" + #[User::RealTableVariable] + "'"
Related
I'm trying to replace gibberish data on my development server (recovering it from the production server).
I first tried using SSMS's "import data" task, but encountered an error saying: "Cannot truncate a table with a foreign key constraint" therefore I resolved to do the following:
Create a new temp copy of the table on the development server (to avoid accessing it from the production database).
Write an update script.
This is my script:
DECLARE #IdToCopy INT;
DECLARE #cnt INT;
SET #IdToCopy = 1
WHILE #IdToCopy <= 55
BEGIN
UPDATE DocumentTypes
SET Name = DocumentTypesTemp.Name
FROM DocumentTypesTemp
WHERE DocumentTypesTemp.DocumentTypeId = #IdToCopy;
SET #IdToCopy += 1;
END;
I expected it to update the table to have the same values, but the only value was that of the last row from the copying table.
I expected it to update the table to have the same values, but the
only value was that of the last row from the copying table.
In each iteration of while loop, your entire DocumentTypes table is getting updated. This is the reason, you see only last record name appearing in the entire table, you have to join the source and destination table based on the id.
UPDATE dest
SET dest.NAME = src.NAME
FROM documenttypes dest
INNER JOIN documenttypestemp src
ON src.documenttypeid = dest.documenttypeid
Above query will update the name from destination table to source table with matching documenttypeid in a single query.
Note: You don't need a while loop to update the name from source to destination.
I fixed my script using the following:
DECLARE #IdToCopy INT;
DECLARE #cnt INT;
DECLARE #NameToCopy varchar(256);
SET #IdToCopy=1
SELECT #cnt=COUNT(*) FROM DocumentTypesTemp;
WHILE #IdToCopy <= #cnt
BEGIN
SELECT #NameToCopy=Name
FROM DocumentTypesTemp
WHERE DocumentTypesTemp.DocumentTypeId=#IdToCopy;
UPDATE DocumentTypes
SET Name=#NameToCopy
WHERE DocumentTypes.DocumentTypeId=#IdToCopy;
SET #IdToCopy += 1;
END;
I am creating an SSIS package to transfer tables between databases on different servers. The timestamp on the source database is in UTC and I would like to convert it to my local time using [CallConnectedTime] AT TIME ZONE 'UTC' AT TIME ZONE 'Pacific Standard Time' AS CallConnectedTime.
How is it possible to do this transformation in the Transfer SQL Objects Task? I thought about using a Data Flow Task but then I need to create one for each table I am bringing over.
The Transfer SQL Server Objects Task does not allow you to define any data transformations.
If you don't want to go the route of creating multiple Data Flow Tasks, you could create an ADO recordset of the table names after the Transfer task runs, then create a Foreach Loop to iterate over the ADO recordset. To do this you might use the following query in an Execute SQL Task, with ResultSet set to Full result set:
SELECT Name
FROM sys.tables
WHERE type_desc = 'USER_TABLE'
Map the result set to an Object variable type. Then you can create the Foreach Loop with an ADO enumerator, point it at your Object variable, and then create another variable to hold the value of the Name field.
Using the variable that now holds the Name field, create another Execute SQL Task inside your Foreach Loop. This Execute SQL task will build and execute a dynamic SQL statement that will UPDATE the table by setting the CallConnectedTime.
The Execute SQL Task inside your loop would then look something like this:
DECLARE #query NVARCHAR(MAX)
DECLARE #Table VARCHAR(1000) = ?
SET #query = N'
UPDATE ' + #Table + N'
SET CallConnectedTime = CallConnectedTime AT TIME ZONE ''UTC'' AT TIME ZONE ''Pacific Standard Time'''
EXEC(#query)
And under Parameter Mapping on the left pane of the editor, you'd add your table name variable as Parameter Name 0 like this:
Your Control Flow should then look something like this when you're done with the above steps (ignore the error icons).
I am very new to SSIS and I need to write a package that will move data from transactional databases to a master database. We have a transactional database per plant and the schema for all of these is identical. I need to go through each table in each database and copy all the data that hasn't been marked as exported to its corresponding table in the master database. After the records are successfully copied to the master database they should be marked as exported in the transactional database.
So far I've gotten my SSIS package to where I can iterate through the plant databases and read from one of the tables. I'm currently storing the resuls from that table into a variable. I accomplished the iteration part by using an expression in the For Each Loop Container's Connection Manager that sets the Initial Catalog to the current database name in the loop.
However, I'm not sure how to proceed after that. Here's a picture of my package's current state:
I've tried creating another Execute SQL Task that takes the results from Get New Apples and copies them to the master database. However, from what I've googled so far there doesn't seem to be an easy way to accomplish this.
A different approach I've tried is to create an OLE DB Source using the same connection manager as the For Each Loop Container. When I do that I get an error saying that the Apple table is not a valid object(My query being select * from Apple where exported = 0;).
Any suggestions as to how I can read a result set from a variable or get the OLE DB Source to work with the aforementioned Connection Manager would be very helpful.
I'm also open to alternate methods to accomplishing this. Like I said, I'm new to SSIS and am still feeling my way around it.
Originally I tried to make this as a stored procedure but it started to grow unmanagable and ugly very quickly:
SELECT *
INTO #tempapple
FROM (SELECT *
FROM [Plant1].[dbo].[Apple]
WHERE exported = 0
UNION
SELECT *
FROM [Plant2].[dbo].[Apple]
WHERE exported = 0) AS x;
INSERT INTO [Master].[dbo].[Apple]
SELECT id,
NAME,
description,
active,
plant
FROM #tempapple
WHERE id NOT IN (SELECT id
FROM [Master].[dbo].[Apple]);
UPDATE [Plant1].[dbo].[Apple]
SET exported = 1
WHERE id IN (SELECT id
FROM #tempapple);
UPDATE [Plant2].[dbo].[Apple]
SET exported = 1
WHERE id IN (SELECT id
FROM #tempapple);
DROP TABLE #tempapple;
I've got to make a few assumptions here:
The variable is type 'Object'
the foreach loop is on an ADO.Object enumerator setting the db name to a variable
insert an expression before the dataflow
in the expression set a new variable type string to "Select * from " + [dbname] + ".[schema].[tablename] where exported = 0"
4a. Note that dbname comes the enumerable set in #2
In your dataflow, set your source to variable and use that variable in #4.
This should get your data at least loaded.
You have options, for updated the isExported column in the source.
I'm writing this directly so you may need to modify it slightly.
declare #dbname as varchar(100) -- dbname
declare #SQL varchar(max)
declare db_cursor cursor for
[ this is where you insert your code for getting DBnames]
OPEN db_cursor
fetch next from db_cursor into #dbname
while ##fetch_status = 0
BEGIN
set #SQL = "Select * into #temptable from " + #dbname + ".[dbo].[Apple] where exported = 0
INSERT INTO [Master].[dbo].[Apple]
SELECT id,
NAME,
description,
active,
plant
FROM #tempapple
-- no where clause needed
UPDATE " + #dbname + ".[dbo].[Apple]
SET exported = 1
from " + #dbname + ".[dbo].[Apple] a
join #temptable tt on a.id=tt.id
DROP TABLE #tempapple; "
exec(#sql);
fetch next from db_cursor into #dbname
END
close db_cursor
deallocate db_cursor
I've decided to settle for a mix of my two approaches. The SSIS package remains mostly the same with the logic to iterate through each plant database. Within the loop I now have several Execute SQL Tasks to import data from the various tables. The logic for the import apples task looks something like this:
SELECT *
INTO #tempapple
FROM (SELECT *
FROM apple
WHERE exported = 0);
INSERT INTO [Master].[dbo].[apple]
SELECT id,
NAME,
description,
active,
plant
FROM #tempapple
WHERE id NOT IN (SELECT id
FROM [Master].[dbo].[apple]);
UPDATE apple
SET exported = 1
WHERE id IN (SELECT id
FROM #tempapple);
DROP TABLE #tempapple;
This allows me to not have reduntant SQL since each task will be executed once per plant database.
we have an system that creates and table in a database on our production server for each day/shift. I would like to somehow grab the data from that server and move it to our archive server and if the data is more than x days old remove it off the production server.
On the production server, the database is called "transformations" and the tables are named "yyyy-mm-dd_shift_table". I would like to move this into a database on another server running SQL 2012 into a database "Archive" with the same name. Each table contains about 30k records for the day.
The way i see it would be something like:
Get list of tables on Production Server
If table exists in Archive server, look for any changes (only really relevant for the current table) and sync changes
If table doesn't exist in Archive Server, create table and syn changes.
If date on table is greater and X days, delete table from archive server
Ideally i would like to have this as a procedure in SQL that can run either daily/hourly ect.
Suggestions on how to attack this would be great.
EDIT: Happy to do a select on all matching tables in the database and write them into a single table on my database.
A lot of digging today and i have come up with the flowing, This will load all the data from the remote server and insert it into the table on the local server. This requires a Linked server on your archive server which you can use to query the remote server. I'm sure you could reverse this and push the data but i didn't want to chew up cycles on the production server.
-- Set up the variables
--Tracer for the loop
DECLARE #i int
--Variable to hold the SQL queries
DECLARE #SQLCode nvarchar(300)
--Variable to hold the number of rows to process
DECLARE #numrows int
--Table to hold the SQL queries with and index for looping
DECLARE #SQLQueries TABLE (
idx smallint Primary Key IDENTITY(1,1)
, SQLCode nvarchar(300)
)
--Set up a table with the SQL queries that will need to be run on the remote server. This section creates an INSERT statment
--which is returning all the records in the remote table that do not exist in the local table.
INSERT INTO #SQLQueries
select 'INSERT INTO Local_Table_Name
SELECT S.* FROM [Remote_ServerName].[Transformations].[dbo].[' + name + '] AS S
LEFT JOIN Local_Table_Name AS T ON (T.Link_Field = S.Link_Field)
WHERE T.Link_Field IS Null'+
CHAR(13) + CHAR(10) + CHAR(13) + CHAR(10)
from [Remote_ServerName].[Transformations].sys.sysobjects
where type = 'U' AND name Like '%_Table_Suffix'
--Set up the loop to process all the tables
SET #i = 1
--Set up the number of rows in the resultant table
SET #numrows = (SELECT COUNT(*) FROM #SQLQueries)
--Only process if there are rows in the database
IF #numrows > 0
--Loop while there are still records to go through
WHILE (#i <= (SELECT MAX(idx) FROM #SQLQueries))
BEGIN
--Load the Code to run into a variable
SET #SQLCode = (SELECT SQLCode FROM #SQLQueries WHERE idx = #i);
--Execute the code
EXEC (#SQLCode)
--Increase the counter
SET #i = #i + 1
END
Initial ran over 45 tables inserted about 1.2 million records took 2.5 min. After that each run took about 1.5 min which only inserted about 50-100 records
I actually created a solution for this and have it posted on GitHub. It uses a library called EzAPI and will sync all the tables and columns from one server to another.
You're welcome to use it, but the basic process works by first checking the metadata between the databases and generating any changed objects. After making the necessary modifications to the destination server, it will generate one SSIS package per object and then execute the package. Can you choose to remove or keep the packages after they are generated.
https://github.com/thevinnie/SyncDatabases
I have a scenario wherein an SP is implemented with Dynamic Query and i need to remove to this implementation due to performance factor. This is an import functionality and we initially insert our data from excel to a staging table and from there we validate our data keeping the data into an another temp table. And then inserting the data into physical tables if all the validation passes.
Staging table can have four category of data(FParty, SParty, TParty, Owner) which is being passed from excel to staging one at a time. And the physical table for these category contains different number of columns. Hence at runtime only we will be able to know the category of data and from then we have to create temp table accordingly to validate the data.
As of now we are using dynamic query to create temp table at run time depending on the category. Procedure is as follow:
CREATE procedure [dbo].[GetData_Into_Temptbl] (
,#CategoryType varchar(50) -- FParty, SParty, TParty, Owner)
BEGIN
declare Category cursor for
select Fields from dbo.StagingTable where CategoryName= #CategoryType
Open Category
Fetch Next from Category into #Field
while ##Fetch_status = 0
begin
set #FieldsToCreatetempTable = # FieldsToCreatetempTable + ',' + #Field
Fetch Next from Category into #Field
end
close Category
deallocate Category
set #tblTemp = 'insert into #TempTableData ('+#FieldsToCreatetempTable+')'
Exec(#tblTemp)
END
Above code is working fine but need to replace the dynamic nature of the procedure. Kindly suggest any concept.
A table/cursor to build a comma separated list of columns is overkill, like major overkill.
You could consider unwinding the logic into 4 simple stored procedures, which can be called based on some logic, such as the categoryType
IF #categoryType = 'FParty'
BEGIN
exec dbo.InsertFPartyTempData
END
ELSE IF #categoryType = 'SParty'
BEGIN
exec dbo.InsertSPartyTempData
END
... // etc