I want to export data from one table into a new one with a nightly job.
To prevent generate dublicates, I implemented a column named "ExportState" in the source table which is 0 for not exported and 1 for exported.
My problem is, that I want to export the data and then setting the State to 1. But I can not make a INSERT INTO ... SELECT and then UPDATE Statements because it is possible that additional Data would inserted to the source table while the export routine runs. So I would at the end UPDATE the ExportState to 1 on records which I never INSERTed to the destination table.
Do you have suggestions to the following solutions ?
A. INSERT INTO ... SELECT and UPDATE ExportState row by row
B. Take a Snaphot INSERT and UPDATE ExportState of the snapshoted Data
Which makes more sense ?
The second problem: The source and destination tables are on different SQL Servers and database instances. Ideas ?
I would create a stored procedure to perform the task.
Within the stored procedure create a table variable or temp table. Insert the data from the source table where ExportState = 0 into the temp table. (If you have a primary key on this table just store the primary key in your temp table.)
Perform your insert statement from source table to destination table.
Using your temp table, perform your update statement to set ExportState = 1 for each record in your temp table.
Wrap all of this within a transaction.
Sample Code:
BEGIN TRAN
DECLARE #Exported TABLE (PK INTEGER NOT NULL);
INSERT INTO #Exported (PK) SELECT PK FROM SourceTable WHERE ExportState = 0;
INSERT INTO #DestinationTable (Field Names)
SELECT FieldNames
FROM SourceTable s
INNER JOIN #Exported e
ON s.PK = e.PK
WHERE s.ExportStatus = 0;
UPDATE s SET ExportStatus=1
FROM SourceTable s
INNER JOIN #Exported e
on s.PK =e.PK;
COMMIT TRAN
Invoke the stored procedure from your nightly job.
To connect to databases on other SQL Servers, look into using Linked Servers. You should be able to configure one under the "Server Objects" folder in SSMS 2008. Here is a link to more info if you are interested...http://msdn.microsoft.com/en-us/library/ff772782.aspx
Related
I have two tables with the same field names and a stored procedure that updates table B with Table A's data by doing a delete from current table and insert into current table from another table that has update values in it:
delete from ac.Table1
insert into ac.Table1
select *
from dbo.OriginalTable
where dtcreate <getdate()-1
I had to recreate Table1 through GIS software which adds GlobalIDs and an Object ID field. The original order had Object ID at the end and the new table has it at the front. Will this impact executing the SQL statement above?
Yes it will. The order of the columns should match for each value to go in desired column
You can try
Insert into ac.Table1 (column1....columnN)
So I have an empty target table that I am wanting to copy all the records into from an existing Report table. However, the existing Report table does not have any primary keys, and my new target table does. I want to copy over all the records from the existing Report table that are not duplicates on "Report.field1" and "Report.field2" and they are not NULL in either one as well.
Is there a quick and dirt way to do that? Like:
INSERT INTO target REPORT
ON CONFLICT SKIP
Please see:
How to avoid duplicate SQL data while executing the INSERT queries without source database
IF not exists(select * from Report where field1 = #field1 and field2 = #field2) and #field1 is not null and #field2 is not null
INSERT INTO Report (..., ..., ...)
VALUES (..., ..., ...)
I have a unique requirement - I have a data list which is in excel format and I import this data into SQL 2008 R2., once every year, using SQL's import functionality. In the table "Patient_Info", i have a primary key set on the column "MemberID" and when i import the data without any duplicates, all is well.
But some times, when i get this data, some of the patient's info gets repeated with updated address / telephone , etc., with the same MemberID and since I set this as primary key, this record gets left out without importing into the database and thus, i dont have an updated record for that patient.
EDIT
I am not sure how to achieve this, to update some of the rows which might have existing memberIDs and any pointer to this is greatly appreciated.
examples below:
List 1:
List 2:
This is not a terribly unique requirement.
One acceptable pattern you can use to resolve this problem would be to import your data into "staging" table. The staging table would have the same structure as the target table to which you're importing, but it would be a heap - it would not have a primary key.
Once the data is imported, you would then use queries to consolidate newer data records with older data records by MemberID.
Once you've consolidated all same MemberID records, there will be no duplicate MemberID values, and you can then insert all the staging table records into the target table.
EDIT
As #Panagiotis Kanavos suggests, you can use a SQL MERGE statement to both insert new records and update existing records from your staging table to the target table.
Assume that the Staging table is named Patient_Info_Stage, the target table is named Patient_Info, and that these tables have similar schemas. Also assume that field MemberId is the primary key of table Patient_Info.
The following MERGE statement will merge the staging table data into the target table:
BEGIN TRAN;
MERGE Patient_Info WITH (SERIALIZABLE) AS Target
USING Patient_Info_Stage AS Source
ON Target.MemberId = Source.MemberId
WHEN MATCHED THEN UPDATE
SET Target.FirstName = Source.FirstName
,Target.LastName = Source.LastName
,Target.Address = Source.Address
,Target.PhoneNumber = Source.PhoneNumber
WHEN NOT MATCHED THEN INSERT (
MemberID
,FirstName
,LastName
,Address
,PhoneNumber
) Values (
Source.MemberId
,Source.FirstName
,Source.LastName
,Source.Address
,Source.PhoneNumber
);
COMMIT TRAN;
*NOTE: The T-SQL MERGE operation is not atomic, and it is possible to get into a race condition with it. To insure it will work properly, do these things:
Ensure that your SQL Server is up-to-date with service packs and patches (current rev of SQL Server 2008 R2 is SP3, version 10.50.6000.34).
Wrap your MERGE in a transaction (BEGIN TRAN;, COMMIT TRAN;)
Use SERIALIZABLE hint to help prevent a potential race condition with the T-SQL MERGE statement.
How to do regular updates to a database table in SSIS. The table has foreign key constraints.
I have a package running every week, and I have to update the data in the table from a flat file. Most of the contents are the same with update values and other new rows.
UPDATE : My data file contains updated contents ( some rows missing, some rows added, some modified ). The data file does not have the Primary keys ( I create the primary keys when I first bulk insert the data from the data file ), on subsequent SSIS package runs, I need to update the table with new data file contents.
e.g.
table
---------------------------------------------
1 Mango $0.99
2 Apple $0.59
3 Orange $0.33
data file
---------------------------------------------
Mango 0.79
Kiwi 0.45
Banana 0.54
How would I update the table with data from the file. The table has foreign key constraints with other tables.
another approach, to load massive group data instead of dealing row by row:
On database
create an staging table (e.g. StagingTable [name], [price])
Create a procedure (you may need to change the objects names, and add
transaction control and error handling etc just a draft):
create procedure spLoadData
as
begin
update DestinationTable
set DestinationTable.Price = StagingTable.Price
from DestinationTable
join StagingTable
on DestinationTable.Name = StagingTable.Name
insert into DestinationTable
(Name, Price)
select Name, Price
from StagingTable
where not exists (select 1
from DestinationTable
where DestinationTable.name = StagingTable.Name)
end
On SSIS
Execute SQL Task with (truncate [staging_table_name])
Data Flow task transferring from your Flat File to the Staging Table
Execute SQL Task calling the procedure you created (spLoadData).
Following are the few thoughts/steps:
Create a Flat File Connection manger.
Take Data flow task.
Create Flat File Source with connection manager just created.
Take lookup transformation(s) as many as you need to get FK values based on your source file values.
Take a lookup transformation after all above lookups, to get all values from Destination table.
Keep Conditional split and compare source values and destination values.
If all columns matched then UPDATE, else INSERT.
Map above conditional split results accordingly to OLEDB Destnation/OLEDB Command.
Give a try and let me know the results/comments.
I need to merge data from 2 tables into a third (all having the same schema) and generate a mapping of old identity values to new ones. The obvious approach is to loop through the source tables using a cursor, inserting the old and new identity values along the way. Is there a better (possibly set-oriented) way to do this?
UPDATE: One additional bit of info: the destination table already has data.
Create your mapping table with an IDENTITY column for the new ID. Insert from your source tables into this table, creating your mapping.
SET IDENTITY_INSERT ON for your target table.
Insert into the target table from your source tables joined to the mapping table, then SET IDENTITY_INSERT OFF.
I created a mapping table based on the OUTPUT clause of the MERGE statement. No IDENTITY_INSERT required.
In the example below, there is RecordImportQueue and RecordDataImportQueue, and RecordDataImportQueue.RecordID is a FK to RecordImportQueue.RecordID. The data in these staging tables needs to go to Record and RecordData, and FK must be preserved.
RecordImportQueue to Record is done using a MERGE statement, producing a mapping table from its OUTPUT, and RecordDataImportQueue goes to RecordData using an INSERT from a SELECT of the source table joined to the mapping table.
DECLARE #MappingTable table ([NewRecordID] [bigint],[OldRecordID] [bigint])
MERGE [dbo].[Record] AS target
USING (SELECT [InstanceID]
,RecordID AS RecordID_Original
,[Status]
FROM [RecordImportQueue]
) AS source
ON (target.RecordID = NULL) -- can never match as RecordID is IDENTITY NOT NULL.
WHEN NOT MATCHED THEN
INSERT ([InstanceID],[Status])
VALUES (source.[InstanceID],source.[Status])
OUTPUT inserted.RecordID, source.RecordID_Original INTO #MappingTable;
After that, you can insert the records in a referencing table as folows:
INSERT INTO [dbo].[RecordData]
([InstanceID]
,[RecordID]
,[Status])
SELECT [InstanceID]
,mt.NewRecordID -- the new RecordID from the mappingtable
,[Status]
FROM [dbo].[RecordDataImportQueue] AS rdiq
JOIN #MappingTable AS mt
ON rdiq.RecordID = mt.OldRecordID
Although long after the original post, I hope this can help other people, and I'm curious for any feedback.
I think I would temporarily add an extra column to the new table to hold the old ID. Once your inserts are complete, you can extract the mapping into another table and drop the column.