You'll have to forgive my ignorance on this - I'm generally a one server kind of guy!
I have a live and reporting server we'll call LIVE and REP.
I have a set of tables on LIVE, some data from which needs to be PIVOTted every night onto the REP server. This already happens through a SSIS package - trouble is - *the extract script is fixed but I need it to be dynamic.
ie. I need to have a new table on LIVE which defines what exactly is extracted to REP.
This means that I want to
DROP the extracted tables on REP every night
CREATE the new tables on REP as defined in the LIVE table
Build dynamic SQL to SELECT from the LIVE tables into the newly created REP tables
I'm concerned because I know SSIS can be funny about table definitions etc etc.
I'm not asking about how to build the dynamic SQL - I've already done that part, and it runs fine entirely on LIVE - but it needs to be cross-server.
Has anyone ever done this before? What is the best practice? Am I even allowed to EXEC dynamic SQL within a SSIS package, and can this be scheduled?
This might give you some ideas:
Create main workflow where you construct drop tables dynamically and create list of tables by creating a result set:
Setup a loop that iterates through each table in result set:
Then in your data flow task you can read from one connection / table and write to another:
Related
Desired result and why:
I have a lot of old Access databases that we are trying to get to SQL Server, and I'm essentially trying to make the Access DB the "middleman" so our old programs can still read/write to them but the information will also be saved in SQL Server. We need the middleman because of how interconnected these tables are through various programs we are rewriting in modern languages. Once we rewrite all of them we will cut the cord and live in SQL Server, but this will take a lot of time.
What I've tried:
We tried creating a linked table to SQL Server and renaming it so it would take the place of the original table. After doing this the table stopped receiving data so we quickly reverted back.
In order to investigate this I created Table B which is just another linked table to SQL Server, and then tried using the After Insert macro on Table A to send any new rows to the linked table but nothing happens. If I manually add a record to Table B it carries over to SQL Server just fine, but I can't get Table A to send data to Table B. I created Table C that is just a local access table and if I manually add a record to Table A it does show up in Table C. No errors at all, it just doesn't do what I need it to do.
I'm lost on how to accomplish this and open to any help or suggestions on how to move forward with this. One thing to note though, is that most of the access databases I have are not using forms at all which is I'm trying to take the macro route instead of any VBA. I need these to trigger without any interaction from the user.
You should use the tool dedicated to this task:
SQL Server Migration Assistant for Access (AccessToSQL)
Ok, there are from comments some new and signficant moving parts here.
For example, data is to be migrated to sql server. As noted, EVEN in access land, all and every table needs and should have a PK for the "basic" data base operations. While it is possible to do some work, and say some importing of data, the instant one wants some forms, VBA code and starts to build a working applcation? Then all tables should have a PK.
And of course if you moved the data to sql server, then it not going to make a lot of sense to have OTHER applcations attempt to modify the linked tables in access, since the data is not in Access anymore!!! Those other sources in theory should thus also hit sql server, and not attempt to use what amounts to a link on a linked table.
However, it does depend. For example, if you use vb.net code and say open a access database, you CAN in fact have that vb.net code open a access table, and in fact it can be a linked table. (however, it would make a WHOLE lot more sense for the vb.net code to open and hit sql server - introduction of a link on a link is going to be problematic.
However, in testing, I have found that say vb.net can open a access table, and even if it is a link, then access will translate though the jet engine (the access data engine), and you can do this.
However, data macros and table triggers on existing access tables? They might work on linked tables, but you of course need to ensure that the linked table does allow edits, and allows inserts. Only AFTER one has verified that you can click on a linked table to sql server - can edit, and then add should one mess around with data macros and triggers on say local tables.
it also depends on what the new software tools and platform is being used here.
But, from a basic database point of view - and general data mangement?
All code, and designs should assume, and be designed around the assumption that each row of data has a PK. This is not always possible, but is a RARE use case.
Practical data management - and use of a database should from both table designs, and from workflow designs, and from a developer point of view assume the concept of a PK row id. Without such assumptions, then you not in the software industry anymore - but in a hack field, and one that will result in great future difficulty when attempting to build work flows and build general information systems.
So, with above in mind: Your table B - it has to work as a valid sql server table.
The sql server table(s). They need a PK, and after linking to sql server, you can open up the linked table in access. Test if edits work, test if adding works, and even perhaps test if delete works. Only AFTER such time, do you now want to start testing any code or other operations from the Access client side.
Introduction of using a linked table from another application? That is a foggy area, but I can confirm for example that say .net oleDB provider will and can open a access database and use + consume even linked tables.
You also don't mention if you using sql logon, or windows auth for the sql server linked tables. But if you using sql logons, then when linking a table, you see this check box - and you want to ensure you selected this when linking the table(s) in question:
Note that you ONLY get this prompt on the first time create of the table link - additional use of the linked table manager (such as re-fresh links) does not offer this prompt. If you don't select the save password option, then you often see a sql logon prompt when you attempt to open a linked table in access.
Hi I am using SSIS (MSSQL) to copy data between multiple tables. This has been working fine up until recently when the S.A.P. team keeps updating the schema of the tables without telling me.
I have multiple tables that they continue to add columns to; this in turn makes my SSIS job of copying the data across fail.
Is there a way in SSIS that I can look at the source table and adjust my destination table to reflect the changes on the fly?
I'm quite new at SSIS and don't mind running a script out of the GUI but wondered if this was an option within the GUI I'm already familiar with.
So in short, can I in SSIS allow for new columns being added to source tables and update my destination tables automatically to stop my jobs failing
(Oh and map source to destination tables automatically)?
You'll have to include the new columns in the data flow, i.e. source and destination (include and map them). So basically you CANNOT automate what you're looking for in SSIS. Hope it helps.
Look into BiML Script, which lets you create and execute SSIS packages dynamically based on the meta data available at run time.
During our SQL Server database deployments, we create a temporary table which contains the new desired state of data for a particular table. We then merge the temp table into the target table (we actually use individual insert, update and delete statements, but that's probably not relevant). The inserts/updates/deletes performed are captured and written out to a log.
We would like to be able to report on what changes would be applied by a deployment, without actually applying them. This is currently done by rolling back the transaction at the end of the above process. This doesn't feel particularly great though.
Now what we are thinking of doing is, instead of performing the changes and rolling them back, we will generate a migration script for the table (generate some SQL code that performs the necessary inserts, updates and deletes). If we want to do the actual deployment, this code will be dynamically executed. If not, the code will just be printed to a log.
It shouldn't take long to put together some code which can generate migration scripts for two specified tables, but I first wanted to verify that there isn't already an existing tool which can do this?
Searching on Google, I can find lots of talk about migrating whole databases, but nothing about generating a data migration script to effectively merge one table into another.
So my question is, does anyone know of such a tool?
There are several data compare tools like:
SQL Data Compare from Red Gate
SQL Server Data Tools
dbForge Data Compare from Devart
Is that what you're looking for?
I'm currently trying to build a data flow in SSIS to select all records from a mapping table where an ID column exists in the related Item table. There are two complications:
The two tables are currently in different databases on different servers.
The databases are in Azure, for which I've read Linked Servers are not supported.
To be more clear, the job to migrate data from Staging environment to Production. I only want to push lookup records into prod if the associated Item IDs are in there. Here's some psudo-TSQL to give a clear goal of what I'm trying to achieve:
SELECT *
FROM [Staging_Server].[SourceDB].[dbo].[Lookup] L
WHERE L.[ID] IN (
SELECT P.[Item]
FROM [Production_Server].[TargetDB].[dbo].[Item] P
)
I haven't found a good way to create this in SSIS. I think I've created a work-around that involves sorting both tables and performing a merge join, but sorting both sides is an unnecessary hit on performance. I'm looking for a more direct and intuitive design for this seemingly simple data flow.
Doing this in a data flow, you'd have your Source query, sans filter, fed into a Lookup Component which is the subquery.
The challenge with this is SSIS is likely on-premises so that means you are going to pull all of your data out of Stage Azure to the server running SSIS and push it back to the Prod Azure instance.
That's a lot of network activity and as I'm reading the Azure pricing guide, I guess as long as you have the appropriate DTUs, you'd be fine. Back in the day, you were charged for Reads and not Writes so the idiom was to just push all your data to target server and then do the comparison there, much as ElendaDBA mentions. Only suggestion I'd make on the implementation is to avoid temporary tables or ad-hoc creation/destruction of them. Just implement as a physical table and truncate and reload prior to transmission to production.
You could create a temp table on staging server to copy production data into. Then you could create a query joining those two tables. After SSIS package runs, you could delete the temp table on staging server
What I need to do is some data between 2 databases. The source can be anything (comma separated file, xls file, any database, ...), the destination is MS SQL Server.
I do not need to sync all data, I just need to sync particular tables.
Example:
I need to sync accounting Software (runs on PostgreSQL) CUSTOMERS table with CRM (runs on SQL Server).
Some problems this tool should be able to face:
1) Accounting software customers table has 1 field that is not mapped in crm customers table. (In this way I want to map this extra field to the field CUSTOMERS_CUSTOM_DATA.EXTRA_FIELD)
2) Having some rules (like sync only customers whose code is between 10000 and 99999)
3) Allowing to perform some post insert tasks (for example I am using manually managed seuqences for the tanble IDs, so after inserting 10 records I need to add 10 to the sequence)
4) Having an exception handling mechanism so if something is wrong it can wither call a sql server stored procedure (that I already have and it will send an e-mail to me) or simply send a message to notify that something was wrong in the nightly sync.
5) Be easy to schedule when to perform data sync (hourly, daily, INCLUDING MANUAL)
6) Perform data conversion: if Surname field in source table is varchar(20) and in destination table is varchar(15) I want to explicitly say "perform a truncation".
7) Have different rules for insert or update. For example in the source e-mail field is not present, but I want to populate it in the destination I decide to perform this operation on insert only, not on update. (for example as I insert a new customer I want to populate the e-mail field concatenating name and surname, but then I want to let the users to modify it, this first insertion is just to simplify data entry, but then this particular case will be handled manually. So I want to say (on insert populate e-mail field, on update don't do anything with email field)
8) In case of delete in the source db don't delete on the destination but only change the varchar(10) STATUS to DELETED.
Note: I know that Integration Services will be perfect for this, but I must support the Express Edition, so SSIS is not an option.
I created a bunch of scripts and scheduled stored procedure that at present do what I need, but it is very hard to maintain and the total lack of a GUI makes the work much slower. I remember having seeing TALEND time ago, maybe that tool is also the answr I need, anyway I need to provide a quick answer to management, so I have now no time to investigate all the tools on the market, and I would prefer to have a suggestion from an expert.
I believe SQL Server Integration Services does all that, and I believe SQL Server Management Studio allows you to create and package your SSIS jobs so that they can be deployed elsewhere.
Finally I went for TALEND, I never really used SSIS, I just saw a live demo of it at a SQL Server conference. Anyway Talend is a free alternative (and quite rich) to SSIS, so it will suite the needs of all customers, including the ones (95%) that has SQL Server Express.