I use TOS to transfer a SQL Server table to another SQL Server. That works more or less. But I have one issue with truncating the table. In the properties for the output table I define "Truncate Table" for the table action and "Insert" for the data action. At the second run I get a lot of duplicate key errors. If I run the "TRUNCATE TABLE" manually in the SQL Server Management Studio, the job works fine.
Are there any known issues with truncate table? Talens Version is 5.3.2
Thanks in advance
I mimicked the scenario and it works fine in Talend Platform for data Management version 5.6.1. I cannot test it on the TOS, but perhaps you can upgrade to the newest TOS version and try again. To be thorough I tried it using separate connection components and built-in connections. The only difference is that using separate connection object requires a commit object.
The workaround I recommend is this:
create a proc to truncate your table and call it from a tMSSqlSP component
connect this to your original subjob which transfers the data between the two tables using an OnSubJobOK flow.
In your tMSSqlOutput component (which performs the truncate/insert) in for Action on Table use Default (so it will not truncate the table)
for Action on data use Insert
I tried this method and it works. This workaround will save you the time and frustration of dealing with the TOS issue.
Related
I am trying to create an ADF activity that can truncate the target snowflake table before a for each loop copies the blob files to it. I can't use the pre copy because it will clean up the table in each iteration. Interestingly when i use a lookup to truncate the table it throws an error hat the ODBC query is not valid however it does the job at database level and truncates the table. Has anyone encountered a similar error.
Yes we had this same issue with ADF using the ODBC connector. It seems to be a known bug. The same thing happens for any UPDATE/INSERT/DELETE statement.
Solutions we used:
Use the newer Native Linked Service Connector. Recommended.
Do your truncate in a procedure and use a Lookup activity to just call the procedure
Prior to the native connector, in some places we would follow up the Truncate activity with a "On Completion" dependency, and have that do a lookup to validate that the table is actually empty so that it could throw an error and stop the pipeline if it didn't work. This was not ideal because it requires an extra ping to the database, and some wacky ADF flow management.
I need a bit advice how to solve the following task:
I got a source system based on IBM DB2 (IBMDA400) which has a lot of tables that changes rapidly and daily in structure. I must load specified tables from the DB2 into a MSSQL 2008 R2 Server. Therefore i thought using SSIS is the best choice.
My first attempt was just to add both datasources, drop all tables in MSSQL and recreate them with a "Select * Into #Table From #Table". But I was not able to get this working because I could not connect both OLEDB Connections. I also tried this with an Openrowset statement but the SQL Server does not allow that for security reasons and I am not allowed to change that.
My second try was to manually read the tables from the source and drop and recreate the tables with a for each loop and then load the data via the Data Flow Task. But I got stuck on getting the meta data from the Execute SQL Task... so i dont got the column names and types.
I can not believe that this is too hard to archieve. Why is there no "create table if not exist" checkbox on the Data Flow Task?
Of course i searched for the problem here before but could not find a solution.
Thanks in advance,
Pad
This is the solution i got at the end:
Create a File/Table which is used for selection of the source tables.
Important: Create a linked Server on your SQL Instance or a working Connectionstring for the OPENROWSET (i was not able to do so - i choosed the linked server)
Query source File/Table
Build a loop through the resultset
Use Variables and Script Task to build your query
Drop the destination table
Build another Querystring with INSERT INTO TABLE FROM OPENROWSET (or if you used linked Server OPENQUERY)
Execute this Statement
Done.
As i said above i am not quite happy with this but for now it should be ok. I will update this if i got another solution.
I need to create a trigger on one server to fire when a certain table is changed, after which I have to do some calculation with the data so I can import (it's no just copying) it into another table on a different server.
Also I only want to import the new data that hasn't been imported through a earlier trigger.
How would I go about this?
Create Linked Server to your target server, then use MERGE statement to perform action depending on existence of record in destination table.
I would refrain from creating triggers referring remote servers though, it might have performance and reliability implications. Consider using Service Broker if you want to achieve small latency and still make the update reliable.
You can create a linked server between two SQL Servers to send data from one to the other.
To create a linked server you need to use the system stored procedure sp_addlinkedserver, you can also do it through SQL Server Management Studio I believe.
Here is an example you can try:
EXECUTE sp_addlinkedserver #server=N'serverip/hostname', #provider=N'SQLNCLI'
You can view if your linked server has been created by querying sys.servers
You can query the linked server database with the following syntax:
SELECT x
FROM [linkedservername].[database].[schema].[table]
More information: http://msdn.microsoft.com/en-us/library/ff772782.aspx
For the trigger updating only data which hasn't been handled before. There are many ways you can do this. If your source table has an 'update date' type column, you can do it based on that. Alternatively, if your table has an identity column and you want to copy data incrementally you can store in a table the 'last id' which is copied over each time, then on the next run of the trigger you can tell it to start from that id+1 that way rows are only transferred once.
I am not at all familiar with Oracle so bear with me!
I am using version Oracle 10G with the web front end called Enterprise Manager. I have been given some CSV files to import however when I use the Load Data from User Files option I think I can set everything up but when the job runs it complains that there are unique constraints, I guess because there is duplicate data trying to be inserted.
How can I get the insert to create a new primary key similar to a MSSQL auto inc number?
Oracle does not have an analog to the MSSQL auto incrementing field. The feature has to be simulated via triggers and Oracle sequences. Some options here are to:
create a trigger to populate the columns you want auto incremented from a sequence
delete the offending duplicate keys in the table
change the values in your CSV file.
You might look at this related SO question.
There is no autoinc type in Oracle. You have to use a sequence.
By using a before insert trigger, you could get something similar to what you get by using an autoinc in SQL Server.
You can see here how to do it.
Is there a way to replicate a sql server database but not push out deletes to the subscribers?
You don't mention which version of SQL Server you're running, but Andy Warren wrote an article on configuring INSERT, UPDATE, and DELETE behaviour in SQL Server 2005. You can configure this through the GUI, using his instructions:
http://www.sqlservercentral.com/articles/Replication/3202/
It's tempting to 'intervene' in a normal replication and 'disarm' the subscriber's side delete stored procedures, but this leaves no option to recover from replication failure. If the replication tries to recover, a reinitialize may be needed and this will drop any 'stale' data that the replication agent considers deleted.
An alternative is to use a normal replication, and use a script that generates insert and update triggers on all tables in the subscriber database, that insert/update that data into yet a third database. This way the third DB will collect all the data that ever existed, the second DB can re-initialize it's subscription if it needs to (when you do, just remember that bulk inserts don't call the insert trigger and check for new data and add it to the third DB), and the first DB doesn't have to perform the extra work that the triggers are.
Do this....Drop the article. Create a new storedprocedure in the corresponding database that mimicks the system store procedure (sp_del...) and contains the same parameter but does nothing. Add the article again...and set the delete store procedure under the article's properties to the new delete stored procedure that you created....
Or you can select Do not replicate Delete Statements....I think that works but i haven't tried it.