I'm modeling a new microservice architecture migrating some part of a monolithic software to microservices.
I'm adding a new PostgreSQL database and the idea is in the future use that database but for now I still need to keep updated the old SQL Server database and also synchronize the PostgreSQL database if something new appears in the old database.
I've searched for ETL tools but are meant to move data to a datawarehouse (that's not what I need). I just can't replicate the information because the DB model is not the same.
Basically I need a way to detect new rows inserted in the SQL Server database, transform that information and insert it in my PostgreSQL.
Any suggestions?
PostgreSQL's foreign data wrappers might be useful. My approach would be, to change the frontend to use PostgreSQL and let postgreSQL handle the split via it's various features (triggers, rules, ...)
Take a look at StreamSets Data Collector. It can detect changes in SQL Server and insert/update/delete to any DB that has a JDBC driver including Postgres. It is open source but you can buy support. You can also make field changes/additions/removals/renaming to the data stream so that the fields match the target table.
Related
I maintain a Windows based application backed by SQL Server DB so there is a set of SQL entities, like tables, views. With time I add new features and fix bugs so schema of the tables and views changes. Once I need to deploy a new version of the application I deploy the DB part by relying on DacPac/DacFx which automatically generates a difference between already deployed DB and the supplied DacPac so the already deployed DB is altered to match the DacPac's content. This way I don't have to write a code which compares 2 schemas and then generates a difference - DacFx does that for me.
That works well but now I need to expand the application so it also supports SQLite DB, I will for sure have to create a new application layer working with SQLite which is doable but one place I need help with is being able to create and maintain SQLite DB schema in the same way I do for SQL Server with DacPac/DacFx so a difference in schemas is computed and applied. While doing that I ideally want do write the SQL schema once so it could be applied to SQL Server as well as SQLite. Ideally, I need to generate SQLite schema based on the schema specific to SQL Server.
I looked into sqldiff which is capable of generating difference between 2 SQLIte DBs and thought I could:
use a technique from here to migrate SQL Server schema to SQLite
generate a temporary SQLite DB based on the generated above schema
compare the above deployed temporary DB to an existing SQLite DB by using the sqldiff and finally apply the difference to the target SQLite DB
but the sqldiff, as stated in the Limitations section:
The sqldiff utility is not designed to support schema migrations
In addition it has limitations around views:
The sqldiff.exe utility does not (currently) display differences in
TRIGGERs or VIEWs.
So I interpret that like that tool could probably be used for some migration cases but it is not really recommended.
How do you suggest generating and applying the schema differences?
I'm also interested to know how others solve the task of incrementally updating schema of their SQLite DB even if I take SQL Server completely out of equation and would instead maintain SQLite schema, in the source code, only. Does everyone create their own schema comparing tools instead of using something similar like DacFx in SQL Server world?
What kind of database the Sql Server Migration Assistant uses as an internal data repository and stores it in the source-metabase.mb file?
I guess that this it is one of standard tool that I could use to open and edit some entries (I need to autamatically add some custom scripts for tables with BLOBs data migration )
You could also just suggest the way how to check most popular database formats: SqlServer Compact, MySQL, Access..
it is one of standard tool that I could use to open and edit some entries
I would not count on it :) It was a proprietary metadata format that has nothing to do with DB products that SSMA supports. It can store metadata for representing Oracle and also SQL Server among others, obviously formats are not connected with file structure that actual DBs use. SSMA format has no open docs, also it may fail to synchronize your changes after manual intervention if you reverse engineer it (due to the fact it was designed just as the migration tool to target SQL Server and was supposed to mostly create new objects there based on source database counterparts).
Can you just write some stored procedures or triggers in your database? For most DBs metadata is exposed as special tables/views anyway. Probably you need to do it only for SQL Server as it's your target db after migration, right? Looking into ways to directly parse or manipulate files managed by "big" DB (like SQL Server or Oracle) doesn't seem to be a good idea for most scenarios. (except digital forensics, for example)
SQL Server metadata related views are here and functions are here. You may profile your SQL Server instance while connecting to it with SSMA just to get some feel what it does to extract metadata (object names, columns of tables, source of SPs etc.)
Data manipulation is pretty much clear from the DB side if you need it too.
We have an firebird database connected to our access control system and then a separate web app that I developed for our time and attendance using sql server 2005 as the data source.
I wanted to use entity framework to connect to the firebird database to access data like users, transactions, sites, etc. As this method is very complicated getting the connection using firebird .NET provider the other option I have is creating a sort of replication (Mirror) from the firebird database to sql server.
I have done this with a DTS previously (Selecting the data and then inserting it) and it worked fine but had many manual processes involved in getting the data and updates made it difficult.
Is there a simpler way to do this or any suggestions would be appreciated.
Unfortunately you need to track what to replicate at the data level. If you are only pushing it to the MS SQL database you could use a modified timestamp, or a record version field (create a generator, set a trigger to update the version field upon update) to reduce what you select. Another popular option is to update a field to current_transaction, but if you do a restore you will start counting from 0.
If you are sending data both ways it gets more complicated -- you need to have conflict resolution. You could look at something like the Microsoft Sync Framework which can use the methods above.
I have a very old system that contains three separate databases. The tables within these databases are organized very poorly and the system is inefficient because of it. Our new system requires the data from the old system to be transformed into one new database that has a different schema.
Question
What is the best approach to transform the data between the old system and the new? Also is there a method to synchronize the data so as people still use the old system the new system will be updated. This is my first time doing this, so please bare with my poorly worded question.
We are currently using MSSQL 2008 R2 and the new system is moving to MSSQL 2012.
To move the data I would suggest using SSIS. You can build queries that transform the data in the tables to the new tables. If the schema remains the same you could use SQL Server replication to synchronize them: http://msdn.microsoft.com/en-us/library/ms151198.aspx. However, this could lead to some performance issues in practice.
Can I store any custom tables in SharePoint's own database?
Is this supported behavior or not?
(I mean tables in MS SQL database, not SharePoint lists.)
If I can, how well does this play with backup/restore functionality?
What are possible caveats?
For anyone wondering why I'm asking: there's an app which is bound to SharePoint server and needs to store some purely relational internal information that doesn't make sense apart from that SharePoint instance. I would like to narrow down data storage to one place but I'm not sure if SharePoint likes its database being used for other purposes.
I'm using SharePoint 2007.
Is it possible? Sure. Should you? Nope.
The SharePoint content/configuration databases are subject to change with any update Microsoft releases, and any changes you make will very likely be destroyed, and if your farm depends on them, be left non-functional.
If you want to store purely relational data in a set of tables, just create another database. There's nothing stopping you from using the same SQL Server instance that houses your SharePoint content and/or configuration databases to store other relational databases as well.
Not a good idea: Support for changes to the databases used by Windows Sharepoint Services
...
Making any modification to the database schema
Adding tables to any of the databases
...
If an unsupported database modification is discovered during a support call, the customer must perform one of the following procedures at a minimum:
Perform a database restoration from the last known good backup that did not include the database modifications
Roll back all the database modifications
It is even worse than the above. It is likely that future upgrades will notice your changes to the content database schema and refuse to upgrade the database period.