I am trying to copy data from views on a trusted SQL Server 2012 to tables on a local instance of SQL Server on a scheduled transfer. What would be the best practice for this situation?
Here are the options I have come up with so far:
Write an executable program in C# or VB to delete existing local table, query the data from remote database and then write results to tables in the local database. The executable would run on a scheduled task.
Use BCP to copy data to a file and then upload into local table.
Use SSIS
Note: The connection between local and remote SQL Server is very slow.
Since the transfers are scheduled, so I suppose you want this data to be up-to-date.
My recommendation would be to use SSIS and schedule it using SQL Agent. If you wrote a C# program, I think the best outcome you will gain is a program imitating SSIS. Moreover, SSIS will be a very easy to amend the workflow anytime.
Either way, to make such program/package up-to-date, you will have to answer an important question: Is the source table updatable or is it like a log (inserts only)?
This question is so important because it will determine how you will fetch the new updates from the source table. For example, if the table represents logs, you will most probably use the Primary Key to detect new records, if not, you might want to seek a column representing update date/time. If you have the authority to alter the source table, you might want to add timestamp column which represent the row version (timestamp differs than datetime)
For building an SSIS package, it will mainly contain the following components:
Execute SQL Task to get the maximum value from source table.
Execute SQL Task to get the last value where it should start from at the destination table. You can get this value either by selecting the maximum value from the destination table or if the table is pretty large you can store that value in another table (configuration table for example).
Data Flow which moves the data from source table starting after the value fetched in step 2 to the value fetched in step 1.
Execute SQL Task for updating the new maximum value back to the configuration table if you chose this technique.
BCP can be used to export the data compress and transfer over network which can be then imported into local instance of SQL.
Also with BCP data exports can be contained with smaller batches of data for easier management of data.
https://msdn.microsoft.com/en-us/library/ms191232.aspx
https://technet.microsoft.com/en-us/library/ms190923(v=sql.105).aspx
Related
I am relatively new to SSIS and have to come up with a SSIS package for work such that certain tables must be dynamically moved from one SQL server database to another SQL server database. I have the following constraints that need to be met:
Source table names and destination table names may differ so direct copying of table does not work with transfer SQL server object task.
Only certain columns may be transferred from source table to destination table.
This package needs to run every 5 minutes so it has to be relatively fast.
The transfer must be dynamic such that if there are new source tables, the package need not be reconfigured with hard coded values.
I have the following ideas for now:
Use transfer SQL Server object task but I'm not sure if the above requirements can be met, especially selective transfer of tables and dynamic mapping of columns.
Use SQLBulkCopy in a script component to perform migration.
I would appreciate if anyone could give some direction as to how I can go about meeting the requirements and if my existing ideas are possible.
I have a simple ETL job copying data from MS SQL to DB2 using DataStage. I need to update a column in MS SQL, "SenttoDB2" once I have successfully copied the data to DB2.
I figured that I just need to create another stage after DB2 and pass the "key" from the source in the update SQL to update the column. Is this correct or am I missing a step somewhere?
You could add an after stage update sql to the source DB2 stage. The sql will get written when the data is pulled, but it will be rolled back on job failure.
If the timing is exceptionally important then you will need to create a second job that updates the source table after the job completes.
We have around 5000 tables in Oracle and the same 5000 tables exist in SQL server. Each table's columns vary frequently but at any point in time source and destination columns will always be the same. Creating 5000 Data flow tasks is a big pain. Further there's a need to map every time a table definition changes, such as when a column is added or removed.
Tried the SSMA (SQL Server Migration Assistance for Oracle ) but it is very slow for transferring huge amount of data then moved to SSIS
I have followed the below approach in SSIS:
I have created a staging table where it will have a table name, source
query (oracle), Target Query (SQL server) used that table in Execute
SQL task and stored the result set as the full result set
created for each loop container off that execute SQL task result set
and with the object and 3 variables table name, source query and
destination query
In the data flow task source I have chosen OLE DB source for oracle
connection and choose data access mode as an SQL command from a
variable (passed source query from loop mapping variable)
In the data flow task destination I have chosen OLE DB source for SQL
connection and choose data access mode as an SQL command from a
variable (passed Target query from loop mapping variable)
And looping it for all the 5000 tables..it is not working can you please guide us how I need to create it for 5000 tables dynamically from oracle to SQL server using SSIS. any sample code/help would be greatly appreciated. Thanks in advance
Using SSIS, when thinking about dynamic source or destination you have to take into consideration that the only case you can do that is when metadata is well defined at run-time. In your case:
Each table columns vary frequently but at any point of time source destination columns will always same.
You have to think about build packages programatically rather than looping over tables.
Yes, you can use loops in case you can classify tables into groups based on their metadata (columns names, data types ...). Then you can create a package for each group.
If you are familiar with C# you can dynamically import tables without the need of SSIS. You can refer to the following project to learn more about reading from oracle and import to SQL using C#:
Github - SchemaMapper
I will provide some links that you can refer to for more information about creating packages programatically and dynamic columns mapping:
How to manage SSIS script component output columns and its properties programmatically
How to Map Input and Output Columns dynamically in SSIS?
Implementing Foreach Looping Logic in SSIS
First thing first. I'm totally new to SSIS and trying to figure out its potential when it comes to ETL and eventually go to SSAS. I have the following scenario:
I have an Intersystems Database which I can connect via ADO .NET
I want to take data from this db and make inserts into MS SQL through incremental loads
My proposed solution/target is:
Have table in the MS SQL that stores the last pointer read or date/time snapshot. (irrevevant at this stage). Let's keep it simple and say we are going to use the record ID that exists in the Intersystems Database
Get the pointer from this table and use it as a parameter through ODBC to read the source database and then make inserts into the target MS SQL db
Update the pointer with the last record read so that next time we continue from there. (I don't want to get into the complications of updates/deletes. let's keep it simple)
Progress so far:
I have succeed to make a connection with MS SQL to read the pointer from there and place it in a variable
I have managed to use the [Execute SQL task] using parameters to read data from Intersystems Db and I'm placing that into a variable using FullResultSet
I have managed to use the [ForEach Loop Container] using the [Foreach ADO Enumerator] to go through each record and each field (yeeeey!)
Now. I can use a [Script task] that makes inserts into the MS SQL database using VB.NET code (theoretically) and then update the counter with the last record read from the source database. I have spent endless hours looking for solutions using ODBC parameters and the above is the only way forward I could see working.
My question is this:
Is this the only way and best practise? Isn't there some easy way that I can plug this resultset into some dataflow components which does the inserts and updates the record pointer for me??
Please assume that I do not have rights access to write into Intersystems Db and thus I cannot make any changes there to the tables structures. But I can only read data so that I can place it into MS SQL.
Over to you guys (or gals?)
I would suggest using a dataflow to improve your design for both efficiency (bulk loading vs row by row in script) and ease of use (no need for scripting).
You should use an execute SQL to get your pointer and save it into a variable.
You should build a sql variable using dynamic sql and above variable.
Make a data connection in manager to Source
Add a dataflow and go into it
Add a source manager and select your source from popup
Choose sql from variable and choose your variable
At this point you should have all the data you want and you can continue to transform or directly load to your target.
Edit: Record Pointer part
Add a multicast (this makes as many copies as you want)
Add an Aggregate Object and max(whatever your pointer is)
OleDBSQL Object (Allows live SQL and used mainly for updates
9a. UPDATE "YourPointerTable" SET "PointerField in DB" = ? (? is actually what you need to enter.
9b. Map to whatever you named in step 8.
This will allow you to handle insert/updates
From Multicast flow a new stream to a lookup object and map your key to the key of destination table
Specify no matches to redirect to no match output
Your matches map to an UPDATE
Your no matches map to an Insert
I'm using SSIS to connect to a SQL Server Database and pull a table into another SQL Server Database. Using Visual Studio 2013 to manage the SSIS scripts.
One of the tables I need to pull is huge so I'd like to just pull data that is greater than a date Range. ie: Data from Jan 1 2016 and newer.
How do i do this via SSIS? I feel like there should be somewhere to add a 'Where' clause or the equivelent to that.
Is it easier to link the two databases, although for security reasons I'm not sure if that is an option.
Any insight would be great.
Thanks!
Try it
Use a Data Flow Task
Configure the Source as OLE DB Source to data Access mode as SQL Command or SQL Command from variable (you can parametrize both)
Add an OLE DB destination, configure Data access mode to Table or
view - Fast load and Rows per batch and Maximum insert commit size to
100000 (you need test to fits to your need)
You can do this without a script. Create a data flow and set up your source via sql command or create a stored procedure to handle your filtering. Here is an example using a sql command
Then create a destination and map the source columns to your destination.
Another option is to set the data access mode to a variable. With this you can build the sql query via variables and make the date filter dynamic with a variable expressions.
I typically used stored procedures and pass in parameters for filtering but if you are just starting with ssis I would try the other two options first.