I have a strange problem when extracting rows from our MS CRM. Let me explain the flow:
I have a SSIS package that extracts data from CRM and loads it to another database. From this database the data is transformed and loaded to a data warehouse.
If this SSIS package runs at the same time as CRM is inserting/updating many rows i get a strange result: The data that is extracted (using a normal OLD DB Source) in SSIS contains duplicate rows (the ModifiedOn date is different though).
If I run the exact same query tha extracts the data from CRM manually in SSMS i get no duplicates.
So.... it seems like the connection from SSIS somehow makes reads of rows that are updated - both before and after the update (dirty reads of some kind).
Anybody experienced this before?
Thx a lot
/Nicolaj
Related
We have a SSIS package which is loading data from Oracle DB table to SQL Server table the data is very huge so whenever the job is running its taking long time to insert the records. Below are the scenarios for our requirement,
There is no last modified column in source table so we cannot able to find which record updated and which one is newly inserted. Hence we are truncating destination table every time and load all the data from source table every time.
We have tried lookup transformation but no luck.
So can anyone please help me for better solution to create a correct package by loading data with less time.
I have created an SSIS package for my database running on SQL Server 2014.
This SSIS package extracts data from an Excel workbook (residing in a shared folder on my Windows Server) and feeds a table in my database. I have tested the package (through a SQL Job) and it works fine.
The logic behind the creation of this package is to pull data on a daily basis from that Excel workbook (the latter being updated on a daily basis). It does have a date column; so, data updated are tagged with the relevant date.
Since this SQL job will be executed on a daily basis, the table will be filled with duplicates (that is, data from Day 1 will be repeated on Day 2 and so on).
How can I deal this issue? I was thinking on adding a T-SQL flow to my SSIS package to delete the contents of the Table before it pulls the data from the Excel Workbook. I guess this would work but I am on the look out for a more elegant solution.
Any help would be appreciated.
i would like to ask maybe basic question. My SSIS uses SQL query, result is exported to *.csv file. Everything works fine for months (sql select, sql records/results are in csv export) - but yesterday i have issue: SSIS ran without error, but in *.csv export one column was without any records(column header is in csv file, but records not) - in another columns there were records (records are from same table as missing column:(). when i ran SSIS manually again (cca 4 min after), data were in column. SSIS is deployed and run on same sever as database. my question is - if there is no error in SSIS Execution report, data flow task ran correctly, now query returns all data withou editing .... is there any chance to find what caused data loss? maybe on server log?
thank you
I am new to SSIS. I am using SSIS 2012 to transfer data from Excel(COZYROC Excel Source Plus component) to SQL server database(OLE DB Destination). My requirement is whenever columns in the excel are not matched with mapped columns or any columns are missing, I should log the error message in the database.
Please help to resolve this problem.
I don't believe that is possible.
SSIS (and SSRS and other applications) require a 'contract' between source and destination such that any changes to the source will throw a mapping error and force the developer to re-map the data flow.
SSIS is not capable of a scenario such as 'if the source columns change, pump the rest and log the changes to another path'.
This is also an example of why Excel makes a terrible data source for a normalized database/ETL project, because users can easily change the Excel doc in such a way that would break data mapping.
Good luck.
I'm working on an application that requires a lot of data. This data is stored in SAP (some big enterprise planning tool) and needs to be loaded in an Oracle database. The data I'm talking about is 15.000+ rows long and each row has 21 columns.
Every time an interaction is made with SAP (4 times a day), those 15.000 rows are exported and have to be loaded in the Oracle database. I'll try to explain what I do now to achieve my goal:
Export data from SAP to a CSV file
Remove all rows in the Oracle database
Load the exported CSV file and import this into the Oracle database
What you can conclude from this is that the data has to be updated in the Oracle database if there is a change in the row. This process takes about 1 minute.
Now I'm wondering if it would be faster to check each row in the Oracle database for changes in the CSV file. The reason why I ask this before trying it first is because it requires a lot of coding to do what my question is about. Maybe someone has done something similar before and can guide me with the best solution.
All the comments helped me reduce the time. First Truncate, then insert all rows with the Oracle DataAccess library instead of OleDb.