I have a project where there will be a master source database, client windows service, client application, client database.
We need to have a client database because there are times when the client won't be able to reach out to the master database source due to connectivity issues.
I was hoping to get some of your expertise on what would the best/most efficient way to sync certain tables from client database to master database and other tables from master database to client database in close to real-time (within minutes). I would also need to keep track of what was synced in the master database so that I can use it in a dashboard.
There could be up til 10,000 + clients trying to pull this information all at once.
Any suggestions would be helpful.
You may be interested in a scheme based on Apache Kafka technologies.
Apache Kafka has the ability to create connectors.
The architectural scheme will look like this:
Local DB - Connector - Apache Kafka - Connector - Server DB.
Connectors support different databases.
You can use connectors to connect to the database itself or to separate tables.
Your scheme is similar to the ETL scheme based on Apache Kafka technologies.
You can also develop an application that will track what was synced in the main database using Apache Kafka Streams.
The other way, if you don't use Apache Kafka, is to master - master replication.
Related
I have some tables from three databases that I want to copy their data to another database in an automated way and these data are quite large. My servers are running on AWS. What is the simplest and most reliable way to do so?
Edit
I want them to stay on-sync (automation process as DevOps engineer)
The databases are all MySQL and all moved between AWS EC2. The data is in range between 100GiB and 200GiB
Currently, Maxwell to take the data from the tables then moved to Kafka and then a script written in Java to feed the other database.
I believe you can use AWS Database Migration Service (DMS) to replicate tables from each source into a single target. You would have a single target endpoint and three source endpoints. You would have three replication tasks that would take data from each source and put it into your target. DMS can keep data in sync via ongoing replication. Be sure to read up on the documentation before proceeding as it isn't the most intuitive service to use, but it should be able to do what you are asking.
https://docs.aws.amazon.com/dms/latest/userguide/Welcome.html
Is there any way to send the entities from Orion Context Broker to SQL Server by using docker compose file?
Right now, Orion subscribes to the entities and Quantum Leap notifies the Crate DB when there is a change for those entities. However, Crate DB is not what I want and this is the reason: https://community.powerbi.com/t5/Desktop/Direct-Query-for-PostgreSQL/m-p/776979/highlight/true#M374297
So basically my question is, if there is any way to replace the Crate DB with any of these supported ones: https://learn.microsoft.com/en-us/power-bi/desktop-directquery-data-sources.
(I randomly chose the SQL Server but any other is fine)
I don't know Quantum Leap or CrateDB but, basically, you could create a Context Consumer to receive notifications from Orion (a subscription would be created for that) so each notification is persisted in your target DB (in this case SQL Server).
In fact, maybe you can take advantage of Cygnus instead of developing that piece of software from scratch. Cygnus is extensible and there is documentation about how to extend it, so new sinks can be developed. You need to develop the sink for SQL Server. Note that there already sinks for other SQL-based databases (MySQL and Postgresql in particular) so they could be a good starting point.
If at the end you develop a new sink, it would be a great contribution to Cygnus catalog of sinks. Please, don't hesitate to send as the pull request in that
I am developing an web application in which i need to maintain the website in the local servers itself with the database in the computer itself , the local database will change periodically.There is a central database through which i have to access all the data in all the remaining DB's .
The problem is that even when internet connection is disabled, the local server will update the local database but when when it regains the internet connection it has to update the central database with the local modified data.
The tables( i mean the database schema, table names, attributes all) in all the DB's is same.The data should be appended if added any new ,should be deleted if any deleted and should be modified if any.
I am using MySQL server as DB, Apache Tomcat as server and using JSP, Servlets for business logic.
Please visit http://dev.mysql.com/doc/refman/5.1/en/replication-howto.html
Mysql replication might do the job but there are a few things that you have to consider, like:
the amount of data that has to be synchronized
the OS used on master and slave servers
because of the internet connection issue - why you disable internet connection? one option might be a scheduled job (crontab)
I have a WinForms business application that connects to a SQL Server on a server within the business network. Recently we have added an ASP.NET web site so some of the information within the system can be accessed from the Internet. This is hosted on the same server as the SQL Server.
Due to the bandwidth available to the business network from the Internet we want to host the web site with a provider but it needs access to the SQL Server database.
95% of data changes are made by the business using the WinForms application. The web site is essentially a read only view of the data but it is possible to add some data to the system which accounts for the other 5%.
Is replication the best way to achieve the desired result e.g. SQL Server within the business network remains the master database as most changes are made to this and then replicate this to the off site server? If so which type of replication would be the most suitable and would this support replicating the little data entered from the ASP.NET web site back to the main server?
The SQL Server is currently 2005 but can be upgraded as required for any replication requirements.
Are there other solutions to this problem?
Yes, since the web application is causing 5% (max) transaction; you can separate it.
I mean, you can have a different DB which is a carbon copy of the master one and have web application point to this DB.
You can setup a bi-directional transaction replication. So that, transaction made to the master DB will get replicated as well as transaction made to the secondary DB will be replicated as well.
No need of upgrading; as SQL Server 2005 supports replication.
For further information check MSDN on replication here: Bidirectional Transactional Replication
In a Nutshell, here are the steps you would do:
Take a full backup pf the master DB
Restore the DB to newly created DB server
Configure trans replication between them.
For better performance, you can also have the primary DB mirrored onto someother DB server.
My team and I are developing a program for a bank and need to synchronize data with an existing system that is deployed on the internal network of the bank's clients. Any update made to the data in our system should be replicated to their system and vice-versa. The database used by our system is Oracle 11g and the existing system used Oracle 10g. We had initially considered using web services for this but the developers of the existing system don't want to build the web services. So we are left with the option of using database links to synchronize data.
What are the pros and cons of using database links to replicate data?
Since it's a banking application handling monetary transactions, how do we encrypt the data sent across the networks?
You wouldn't use database links to replicate data-- at least not alone. You would want to configure either Oracle Streams, Oracle GoldenGate, or Advanced Replication to handle your bidirectional replication. Streams would be the preferred choice since you're dealing with Oracle to Oracle replication since it's a newer and more efficient architecture than Advanced Replication. Both options are included with the enterprise edition license. GoldenGate is a product that Oracle purchased relatively recently that would be more appropriate if you were trying to replicate data from Oracle to non-Oracle databases and requires additional licensing.
If you want to encrypt the data in transit between the two databases, you would need to license the Oracle Advanced Security option on both databases. That is an extra cost option on top of the enterprise edition license. The alternative would be to configure a VPN or secure port forwarding between the servers.