I have 5 exact same Postgres database at 5 locations. I would like to upload (real time or every night) records from each database to a 6th Remote Master Database of same structure (one way upload only, no need to sync back).
in this way I will have their own respective data on 5 database and all the records from 5 databases in 6th Database.
How can I achieve this with Postgres? I have some clues about ETL but no idea on how to implement the same.
Any Help on this will be highly appreciated, Thank you.
Related
I was reviewing the oracle setup at our place . Have few doubts on best practices. Using oracle 12 c and planning to move to 19 by EOY. Open to answers on both versions if there is a difference.
In a microservice architecture we have 10 apps each having 10 tables they interact with. Is it better to have 10 different DB or 10 different users/schemas in same DB.
All the tables in total have 31Tb of data. It's mentioned that oracle 12c Bigfile can only grow till 32Tb. Is that a true limitation of oracle that it can't grow further?
For tablespace currently , all the objects are saved in 1 tablespace and indexes are saved in 2nd tablespace. Is that a good strategy. Is there something better like having different tablespace for different users linked to different microservices or things like save clob objects in 1 tablespace and rest in another tablespace.
Is there an out of box purge or archive solution. I have seen row level archival option which basically turns a flag on or off. Ideally would like a functionality where every weekend a year old data gets purged or archived automatically.
There is a table with orders which need to be fulfilled in next 3 months and then once they are past the delivery date they remain in the table for select and no updates happen on them. I was thinking to apply partitions to such a table. There is 100 Gb of data each in such tables. Will applying partitions help? What kind of strategy will work best for this usecase.
I have a database which contains data for events. For every event database size increases by 1 GB.
I have planned to create a separate database for archive. After completing each event, I plan to move that data to the archive database using a stored procedure.
I have already added many indexes on database to improve speed.
So is this good technique or is there any better idea to improve the speed on database ?
Thanks in advance.
I have two databases - a CRM database (Microsoft Dynamics crm) and a company database.
These two databases are different.
How to copy the company database (all objects) into CRM database every 5 seconds?
Thanks
The cheapest way to do this (and one of the easiest) is to use a method called log shipping. This can (on a schedule even every 5 minutes or so) copy the log file to another machine and from the shipped log file restore to the target data base. Please ignore geniuses that will claim it can be done every minute because it takes a little while to close the log backup file, move it and reapply but a 5-10 minute window is achievable.
You can also use mirroring,transactional replication, and other High Availability solutions but there is no easy way to keep two machines in sync.
Do you need to duplicate the data? Can;t you query the source system directly if they're on the same server?
Else this might point you in the right direction: Keep two databases synchronized with timestamp / rowversion
I need advise on copying daily data to another server.
Just to give you an image of the situation, I will explain a little. there are workstations posting transactions to 2 database servers (DB1 and DB2). These db servers hosted on 2 separate physical servers and are linked. Daily transactions are 50.000 for now but will increase soon. There might be days some workstations down (operational but cannot post data) and transactions posted after a few days.
So, what I do is I run a query on those 2 linked servers. The daily query output contains ~50.000 records with minimum 15 minutes fetching time as linked servers have performance problems.I will create a SP and schedule it to run 2AM in the morning.
My concern starts from here, the output will be copied across to another data warehouse (DW). This is our client's special land, I do not know much about. This DW will be linked onto these db servers to make it possible to send the data (produced by my stored procedure) across.
Now, what would you do to copy the data across:
Create a dummy table on DB1 to copy stored procedure output on the same server so make sure it is available and we do not need to rerun stored procedure again. Then client retrieves it later.
Use "select into" statement to copy the content to remote DW table. I do not know what happens with this one during fetching and sending data across to DW. Remember it takes ~15 mins to fetch the data by my stored procedure.
post the data (retrieved by stored procedure) with xml file through ftp.
Please tell me if there is a way of setting an alert or notification on jobs.
I just want to take precautions so it will be easier to track when something goes wrong.
Any advice is appreciated very much. Thank you. Oz.
When it comes to coping data in SQL Server you need to look at High Availability Solutions, depending on the version and edition of your SQL Server you will have different options.
http://msdn.microsoft.com/en-us/library/ms190202(v=sql.105).aspx
If you need just to move data for specific tables you can have options like SSIS job or SQL Server Replication.
If you are looking to have all tables in a given databases copied to another server you should use Log Shipping. Which allows you to copy entire content of source database to another location. Because this is done of smaller interval the your load will be distributed over larger period of time instead of having large transaction running at once.
Another great alternative is SQL Server Replication. This option will capture transaction on the source and push them to the target. This model requires publisher (source), distributor (can be source or another db) and subscriber (target).
Also you can create SSIS job that runs on frequent basis and just moves specified amount of data.
My current project requires immediate replication between 5 databases. Servers are physically distributed across the globe. Currently we use Redis with one master server installed on another 6th server. All database writes on any of 5 servers are performed to this 6th master server, and other 5 are slaves of it. This approach has a lot of flaws and i'm looking for solution to replace it. Any suggestions please?