We have a MongoDB that have heavy writes from a device. We want to regularly (maybe on schedule) read data from that MongoDB and transfer it to a SQL Server. Then that SQL server is where we would do all our analysis from.
What would be the best way to do the transfer and load from MongoDB to SQL Server?
Has anyone tried Apache Spark to do this?
Ended up using a 3rd party utility called Studio 3T. It has functionality for MongoDB to SQL Migration and a task can be created for it and can be scheduled. Also, the utility has more functionality other than migration which I find useful.
This utility was chosen due to limited budget. We just had to pay for ultimate license.
Related
I have two remote databases, Oracle 19c and SQL Server 19. In my redhat linux, I have to read SQL Server table and write it to Oracle.
I wrote a Python code to read data from SQL Server and insert it to Oracle database. But, I think it is so slow. Since, the table has about 16 million records.
I search a lot and see this post:
https://stackoverflow.com/a/66550879/6640504
It said that Oracle Transparent Gateway makes it possible to run a procedure for migrating data from extra database to Oracle in little time.
I can access to Oracle database using terminal and don't have any graphical accesses.
Would you please guide me if Oracle Transparent Gateway is the best way to migrate data from SQL Server to Oracle database, how to install Oracle Transparent Gateway and use it? If not, what is the best way?
Any help is really appreciated.
One solution would be to work from SQL Developer.
You can download SQL Developer for free from https://www.oracle.com/tools/downloads/sqldev-downloads.html
SQL Developer provides a "migration workbench" that supports SQL Server.
The data migration can be done online or offline.
Online causes the table data to be moved by SQL Developer when you have completed the necessary information in the wizard; Offline causes SQL Developer to generate scripts after you have completed the necessary information in the wizard, and you must later run those scripts if you want to move the data. (Online moves are convenient for moving small data sets; offline moves are useful for moving large volumes of data.)
Could anyone advise me how can sync data between SQL Server on premise to AWS using CDC or change tracking. I'm not sure,can I use SSIS for integration?
Thank you
The canonical AWS method for SQL Server-to-cloud CDC is Database Migration Services, it works with both self-managed SQL Server instances and RDS instances of SQL Server. Be aware it has some limitations and prerequisites for your SQL Server, so do an assessment up front that your scenario is applicable.
If you can't meet the prerequisites or constraints above, you can also consider using the popular open source tool Debezium, especially if you have to have a lot of custom hooks before or after the CDC, since it posts all the changes to a Kafka topic.
SSIS is an acceptable tool if you're only doing batch-level updates and don't need near-realtime replication and you want to fully control the synchronization, but there is a lot of overhead to developing and maintaining those types of packages. I wouldn't recommend it.
I need a means to create a SQL Server Express partial replica of a SQL Server 2014 server database and then synchronize the two, triggered by a C# command. Eventually, there could be as many as a hundred partial replicas on various clients. Anyone know how I can do this and where I should look to learn how to do it? Thanks!
SymmetricDS is a product that will synchronize SQL Server 2014 with SQL Server Express either in a partial sync scenario or full sync scenario. You configure which tables you need to synchronize. It will by default synch in near real time based on your configuration or you can schedule a variety of jobs to meet your needs. To support your C# use case you could look into the JMX and/or REST API's provided as well.
SymmetricDS Open Source
SymmetricDS Professional
Which software can you recommend to sync data from a FoxPro source to a SQL Server destination?
Here are a couple links with information that might be of help:
Migrating From Visual FoxPro
VFP conversion to support SQL server Backend
At my company we have data being written to foxpro tables daily, and are synced to SQL Server nightly using SSIS, so for the actual conversion I recommend that.
Getting the foxpro data to your central location could be accomplished by transferring the foxpro files via FTP (we've done this for years), or you could set up some sort of Web Service.
I don't think you'll find a tool to do this automatically and on a scheduled basis especially if they are geographically remote and the sync has to happen over the public internet. I'd be looking at writing a web service of some sort, which will sit on a server on the VFP side with methods to expose the VFP data, and another service on the SQL Server side to access that web service periodically and perform CRUD operations.
I'm managing a web app featuring countless ETL (Extract, Transform & Load) processes feeding a datawarehouse (using SSIS dtsx packages + *.sql files).
At the moment, everything is governed by several SQL Agent processes monitoring "scheduling queue" tables.
I'm trying to figure out a way to develop a scheduler or scheduling "framework" of some sort that needs to do a subset of what SQL Server Agent does (executing dtsx packages, executing SQL) but with a configuration UI based on ASP.NET, as users need to be able to create & modify schedules, manually launch processes and monitor custom logs.
Configuration info, activity, execution logs, schedules, jobs, job steps and everything else needed that I'm missing should be persisted to DB.
I'm guessing a Windows Service would be flexible but development costs might skyrocket.
I appreciate any sort of input, particularly some clues on SQL Server Agent's internal workflow. I DID try searching for info, but it being PROPRIETARY software, I found zilch.
(maybe I should specify I'm more than familiar with SQL Server and C# based .NET development, I just have no idea where to start from..)
The SQL server agent database is called 'msdb'. You can attach profiler to msdb and create some scheduled jobs to see how it works.
The gist of the answer is one should use SQL Server Service Broker, coupled with SQL Server Service Broker External Activator from the Feature Pack