SETUP
I have three instances running of my app deployed in three separate geographical locations running locally (since Internet connections are not reliable).
I have one master instance of the app running on DigitalOcean.
I would like to sync the local databases with the master database daily.
MY CURRENT APPROACH
I have a cron job scheduled to pull the data from the local databases and upload them into a database running on a DigitalOcean VPS. My concern is that the id columns of the three local dbs will conflict resulting in an incorrect merge in the online master database.
I am running Rails 4.1 with Ruby 2.0 using Postgres as my DB.
I am open to any solutions that come up with a relatively simple way of keeping the databases in sync.
Thank you
Simplest solution would be to have all your unique autonumbering one column ID keys to consist of 2 different columns. An "ServerID" and an auto numbering ID. It makes your design more complicated but you never have to worry of non unique keys.
Related
I have some tables from three databases that I want to copy their data to another database in an automated way and these data are quite large. My servers are running on AWS. What is the simplest and most reliable way to do so?
Edit
I want them to stay on-sync (automation process as DevOps engineer)
The databases are all MySQL and all moved between AWS EC2. The data is in range between 100GiB and 200GiB
Currently, Maxwell to take the data from the tables then moved to Kafka and then a script written in Java to feed the other database.
I believe you can use AWS Database Migration Service (DMS) to replicate tables from each source into a single target. You would have a single target endpoint and three source endpoints. You would have three replication tasks that would take data from each source and put it into your target. DMS can keep data in sync via ongoing replication. Be sure to read up on the documentation before proceeding as it isn't the most intuitive service to use, but it should be able to do what you are asking.
https://docs.aws.amazon.com/dms/latest/userguide/Welcome.html
I have two AWS RDS Postgres Instances. Sometimes I create new instances for applications that are (very) vaguely related to other applications. Which always leads me to the question; should I just create a new database in an existing instance or keep things separate and create a new instance instead?
I would recommend that you use the same database server (Amazon RDS instance).
You can logically separate the data via either:
CREATE DATABASE: Full logical separation. You login to one database and never see the other one. OR
CREATE SCHEMA: Data is kept separate, but can be referenced from the other. Quite common for staging areas, such as doing ETL in a Staging Schema, then publishing to a Production Schema.
From your description, I'd say that CREATE DATABASE would be appropriate.
The benefit is that you only need to manage one database and there is little impact on cost unless you need to increase the size of the database instance to handle the higher load (but it would still be cheaper than running two separate databases).
Just keep an eye on the CloudWatch metrics to be sure that the database is handling the increased load correctly.
Normally, the biggest reason for using a different server is because they are owned/managed by different teams. However, in your situation the same team seems to 'own' both data stores, so that wouldn't be an issue.
Based on our requirement we need to have one database per account as we cannot have a single database. So we were thinking about multiple database on the SQL Server and based on my research we can have appx 32,000 database on single database instance but the resource limit will be reached far sooner.
How many is too many databases on SQL Server?
Another point was to use no-sql database like MongoDB. As far as I know we can create database per account with different database name.
I wanted to check reference to resources used by MongoDB, do I need more ram, processor and lots of servers for the architecture.
We are looking to have about 10,000 databases max on a server. Can that work on the a single mongodb server?
If you do have similar architecture currently running, please do share.
I am assinged for the task of Continuous deployment from development server to production server.
In my development server all the database objects will be created under the 'DBO' Schema. But in Production server based on every Tenants company list differenet SCHEMAS will be there.
for E.g in my development server if a tablename is created like
dbo.ABC
dbo.XYZ
And while i creating a tenant(Omkar---db) (Sarkur,Mathur--- schemas), the database objects will be like
Sarkur.ABC, sarkur.XYZ
Mathur.ABC, Mathur.XYZ
Now, i have to compare these two databases to check whether any changes in structure of the database objects, addition / deletion of database objects. If so that changes has tobe synchronized in the production database.
If anyone know that how to compare these two different schemas object, pls let me know..
1 option that I know is looking suitable
Flyway :
It is Easy to setup, simple to master. Flyway let's you regain control of your database migrations with pleasure and plain sql.
Solves only one problem and solves it well. Flyway migrates your database, so you don't have to worry about it anymore.
Made for continuous delivery. Let Flyway migrate your database on application startup. Releases have never been this easy.
Big Plus It's Open Source framework!
http://flywaydb.org/
In a cloud hosting environment (amazon, rackspace,) you can create multiple instances. Let's say I have a database server (mysql,) and other persistent data.
If I create more instances, what happens to the data ? Ex.
1 Instance -> user table (in a db)
I make another 3 instances
4 Instances -> each one has it's one user table
Errors: if someone adds data to the table on instance 3 how does instance nr 4 see it ? If I merge the instances back to one, which instance data does it keep ?
Thank you
I would suggest having one (or more) dedicated database servers that all the instances connect to. If you are using Amazon Web Services check out their RDS service ( http://aws.amazon.com/rds/ )
That way you don't need to worry about replication - if you do want each server running it's own db instance you'll have to look into replication - for MySQL this is a good guide: http://dev.mysql.com/doc/refman/5.0/en/replication.html
I would strongly recommend the former solution for the database. Replication is tricky to get right and can be a nightmare to maintain
If you are using static data eg images I would recommend using amazon's S3 service for uploading to ( http://aws.amazon.com/s3/ ) - that way all your servers are getting their data from a single point instead of having to replicate over servers, which is always going to end up a less scalable solution