I am trying come up with a way to pull the tables out of an Access database, automate the creation of those same tables in a SQL 2008 DB, and move the data to the new tables. This process will happen on a regular basis and there may be different tables each time.
I would like to do this totally in SSIS.
C# SQL CLR objects are an option.
The main issue I have been running into is how to get the Access table's schema and then convert that to a SQL script that I can run via SSIS.
Any ideas?
TIA
J
SSIS cannot adapt to new tables at runtime. (You can change connections, move a source to a table with a different name, but the same schema) So, it's not really easy to do what I think you are saying: Upsize an arbitrary set of tables in an Access DB to SQL (mirroring their structure and data, naming, etc), so that I can then write some straight SQL to transform the data into another SQL database or the same part of the database.
You can access the SSIS object model from C# and build a package (or modify a template package) programmatically and then execute it. This might offer the best bang for your buck, but the SSIS object model is kind of deep. The SSIS Team blog have finally started putting up examples (a year after I had to figure a lot of this out for myself)
There is always the upsizing wizard, and I'm sure there are some third party tools.
Related
I have more than 500s of tables in SQL server that I want to move to Dynamics 365. I am using SSIS so far. The problem with SSIS is the destination entity of dynamics CRM is to be specified along with mappings and hence it would be foolish to create separate data flows for entities for 100s of SQL server table sources. Is there any better way to accomplish this?
I am new to SSIS. I don't feel this is the correct approach. I am just simulating the import/export wizard of SQL server. Please let me know if there are better ways
It's amazing how often this gets asked!
SSIS cannot have dynamic dataflows because the buffer size (the pipeline) is calculated at design time (as opposed to execution time).
The only way you can re-use a dataflow is if all the source to target mappings are the same - Eg if you have 2 tables with exactly the same DDL structure.
One option (horrible IMO) is to concatenate all columns into a massive pipe-separated VARCHAR and then write this to your destination into a custom staging table with 2 columns eg (table_name, column_dump) & then "unpack" this in your target system via a post-Load SQL statement.
I'd bite the bullet, put on your headphones and start churning out the SSIS dataflows one by one - you'd be surprised how quick you can bang them out!
ETL works that way. You have to map source, destination & column mapping. If you want that to be dynamic that’s possible in Execute SQL task inside foreach loop container. Read more
But when we are using Kingswaysoft CRM destination connector - this is little tricky (may or may not be possible?) as this need very specific column mapping between source & destination.
That too when the source schema is from OLEDB, better to have separate Dataflow tasks for each table.
Hi I am using SSIS (MSSQL) to copy data between multiple tables. This has been working fine up until recently when the S.A.P. team keeps updating the schema of the tables without telling me.
I have multiple tables that they continue to add columns to; this in turn makes my SSIS job of copying the data across fail.
Is there a way in SSIS that I can look at the source table and adjust my destination table to reflect the changes on the fly?
I'm quite new at SSIS and don't mind running a script out of the GUI but wondered if this was an option within the GUI I'm already familiar with.
So in short, can I in SSIS allow for new columns being added to source tables and update my destination tables automatically to stop my jobs failing
(Oh and map source to destination tables automatically)?
You'll have to include the new columns in the data flow, i.e. source and destination (include and map them). So basically you CANNOT automate what you're looking for in SSIS. Hope it helps.
Look into BiML Script, which lets you create and execute SSIS packages dynamically based on the meta data available at run time.
I was wondering if anyone knew how to reverse engineer an Access Database. I would like to be able to generate the SQL code that is used to create the database tables and to insert all the records in the table. In other words, I would like to create whats is similar to a MySQL dump file.
Any ideas would be great
thanks
jason
There's nothing built into Access that will generate the DDL for your tables.
There are many third party tools however (ERWin, ERStudio, Visio, etc) that can generate the DDL for you.
I don't know anything that will generate the Insert scripts for you. Access does however have plenty of export/import options if you just want to create a copy of your data and then use that as an import source.
It should be pointed out that there's nothing stopping you from writing some VBA code to loop through the TableDefs and creating the DDL and insert scripts yourself.
One possible approach that may work, is to upsize your tables to SQL server, and then have SQL server generate scripts for you.
Unfortunately the resulting scripts would probably be only compatible with SQL server. So then you would have to run them on SQL server and then pull the data in tables down to access.
Access does support DDL, but unfortunately it does not have any tools are facilities built in to generate the scripts.
I'm wanting to take data from a SQL Server table and populate a Oracle table. Right now, my solution is to dump the data into a Excel table, write a macro to create a sql file that I can load into Oracle. The problem with this is I want to automate this process and I'm not sure I can automate this.
Is there an easy way to automate populating a Oracle table with data from a SQL Server table?
Thanks in advance
I suppose it depends on your definition of "easy".
The most robust approach would be to either use heterogeneous connectivity in Oracle to create a database link to the SQL Server database and then pull the data from SQL Server or to create a linked server in SQL Server that connects to Oracle and then push the data from SQL Server to Oracle.
Yes. Take a look at MS SQL's SSIS which stands for SQL Server Integration Services. SSIS allows all sorts of advanced capabilities, including automated with Sql Server Jobs, for moving data between disparate data sources. In your case, connecting to Oracle can be achieved a variety of ways.
There are three ways to automate this:
1) You can do as Paul suggested and created an SSIS package that will do this and it can be scheduled via SQL Agent,
2) If you don't want to deal with SSIS, you can download the free SQL# (SQLsharp) CLR Library from http://www.SQLsharp.com/ and use the DB_BulkCopy Stored Procedure to do this in a T-SQL Stored Proc which can also be scheduled via SQL Agent. [note: I am the author of SQL#]
3) You can also set up a Linked Server from SQL Server to Oracle, but this has the draw-back of being a potential security hole. Of course, you could use an Oracle Login that only has write-access to that single table (or something similar to that).
There are lots and lots of ways to do it. Which you choose depends on your requirements.
Using Excel is fine if it's a one time thing.
If it's a once-in-a-while thing, then you could write a simple .NET app that uses a single DataSet and multiple DataAdapters to do the data dump. C# code example here.
if it's a regular thing, then you could put the above in a Schtasks task, or you could use SSIS. I think SSIS is an extra-cost option.
if the requirement is for "online access", then a linked database is probably appropriate.
I would like to publish some data of a sql server 2k to msaccess databases.
I'd like to do that given a table supplying datatransformation info, for example :
tablenameOnServer | pathToPub
------------------------------------------------------
Clients | D:\Data\Pub1\ClientData1.mdb
Orders | D:\OtherData\Pub\Sales.mdb
The given mdb file should be created (or an empty one copied of course) and the table should be created each time the script runs.
I of course don't need a full blown solution, but some pointers as where to start are very welcome. I thought I'd use SSIS for this, but am new to it and I like to know where I start best in order to avoid too much loss of time :
Do I use SSIS with BIDS (vs2008), can I read data in a package and create tables on the fly?
Do I use C# and manipulate and create packages in code?
Or what do I do best? Is SSIS the obvious solotion anyway?
In any case : some pointers to get me started would be very welcome...
UPDATE : This question is about publishing data, so it can be shipped on CD for example. It's not about linking to an sql server.
The simplest solution is to simply link the sql server table in access. Then you can see the data in realtime.
You can create an Access database to do this fairly easily.
Here is a basic algorithm
Within access, create a linked table pointing to your table.
create an Access query to filter data according to your criteria called NewQuery
Use VBA to create new database NewDB
export your NewQuery with structure and columns to the NewDB
If you want to keep the SSIS packages, I would make a template SSIS package and then using the SSIS objects from C# create the packages. I've done this to start off packages which are later customized.
Alternatively, if you aren't going to keep the packages after they are generated and aren't going to use a lot of SSIS features (logging etc.), I would consider doing the whole thing from C# (or other language), because SSIS isn't a great tool for connection managers (source or destination) which change.
From C#, you can simply read the schema, create the table in Access with matching types and columns and populate it.