We have a requirement where we need to migrate MySQL data which is running on AWS-RDS services to Snowflake. Any pointers/docs/references which can guide us would help.
The idea is to create a "data lake" in Snowflake.
MySQL instance running on AWS needs to be migrated to Snowflake-data lake.
The data needs to be migrated as "semi-structured" data.
Regards,
Somen Swain
Related
Need inputs on the below requirement where there is a need to migrate the data from databases like
SQL Server, Teradata, Oracle-DB to AWS-S3 first to create a data lake.
Specifically looking for suggestions on :
--> Which services of AWS can be used to achieve this, as in to migrate the data models along with the data.
--> Any tools which can help expedite this migration process.
Regards,
Somen Swain
I am new to Azure and have no prior experience or knowledge regarding working with Azure data warehouse systems (now Azure Synapse Analytics Framework)
I have access to a "read only" data warehouse (not in Azure) that looks like this:
I want to replicate this data warehouse as it is on Azure cloud. Can anyone point me to the right direction (video tutorials or documentation) and the number of steps involved in this process? There are around 40 databases in this warehouse. And what if I wanted to replicated only specific ones?
We can't do that you only have the read only permisson. No matter which data warehouse, we all need the server admin or database owner permission to do the database replicate.
You can easily get this from the all documents relate to the database backup/migrate/replicate, for example: https://learn.microsoft.com/en-us/sql/t-sql/statements/backup-transact-sql?view=sql-server-ver15#permissions,
If you have enough permission then you can to that. But for Azure SQL datawarehouse, now we called SQL pool (formerly SQL DW), we can't replicate other from on-premise datawarehouse to Azure directly.
The official document provide a way import the data into to Azure SQL pool((formerly SQL DW)):
Once your dedicated SQL pool is created, you can import big data with
simple PolyBase T-SQL queries, and then use the power of the
distributed query engine to run high-performance analytics.
You also could use other ETL tool to achieve the data migration from on-premise datawarehouse to Azure. For example using Data Factory, combine these two tutorials:
Copy data to and from SQL Server by using Azure Data Factory
Copy and transform data in Azure Synapse Analytics by using Azure
Data Factory
I have a SQL Server 2012 hosted on a standalone machine. I want to migrate it to my AWS Redshift (already existing data warehouse).
My question is wether it is possible via AWS Data migration service ?
I am also open to other efficient methods for migration. Currently I am doing the following steps
taking a backup of the SQL server DB in the standalone server.
uploading it to AWS-S3.
Droping and restoring the Db from S3 in AWS-RDS (Sql-server)
I would like this data to be present in my data warehouse i.e AWS-Redshift
Thanks for the help in advance !
There are 2 types of migration within DMS
"one off" data migration, where the data is copied using sql
statements
"continuous replication", where the "change dta capture" system on
the source is used to capture and process just the updates.
SQL server can be used as a source for both of these types however there are caveats and limitations that should be read and understood thoroughly.
https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Source.SQLServer.html
So long as you follow the instructions and meet the limitiations that are documented then it will work great.
I'm researching the differences between AWS and Azure for my company. We going to make an web-based application. Which is going to be across 3 regions, each region needs to have a MS SQL database.
But I can't figure how to do the following with AWS: the databases need to sync between each region (2 way). So the data stays the same on every Database.
Why we want this? For example a customer* from Eu adds a record to the database. Now this database needs to sync with the other regions. Resulting that a customer form the region US can see the added records. (*Customers can add products to the database)
Do you guys have any idea how we can achieve this?
it's a requirement to use Ms SQL.
If you are using SQL on EC2 instances then the only way to achieve multi-region, multi-master for MS SQL Server is to use Peer-to-Peer Transactional Replication, however it doesn't protect against individual row conflicts.
https://technet.microsoft.com/en-us/library/ms151196.aspx
This isn't a feature of AWS RDS for MS SQL, however there is another product for multi-region replication that's available on the AWS marketplace, but it only works for read replicas.
http://cloudbasic.net/aws/rds/alwayson/
At present AWS doesn't support read replicas for SQL server RDS databases.
However replication between AWS RDS sql server databases can be done using DMS (database migration service). Refer below link for more details
https://aws.amazon.com/blogs/database/introducing-ongoing-replication-from-amazon-rds-for-sql-server-using-aws-database-migration-service/
I have created an application on Bluemix. I need to copy my database on Bluemix that can be accessed from my adapter. Can anyone give me detailed steps on how to proceed?
First thing: if your database is reachable through the Internet and you only need to connect to it from the application, please note that a cf application on Bluemix can access the public network and so it is already able to connect to your DB in this scenario.
Assuming that you have a requirement for migrating the DB on Bluemix, you didn't specify which kind of database you want to migrate, here are the main (not all) possibilities you currently have:
RDBMS:
PostgreSQL by Compose (you need an account on compose.io)
SQL Database (DB2, only Premium plan available)
ClearDB (MySQL)
ElephantSQL (this is basically a PostgreSQL as a Service - that is you have to work on the db via API)
you could use the RDBS capability of dashDB
No-SQL:
Cloudant (documental)
Redis by Compose (ultra fast key-value db. You need an account on compose.io)
MongoDB by Compose (you need an account on compose.io)
IBM Graph (graph No-SQL db)
I suggest you to take a look at the Bluemix Catalog (subcategory Data and Analytics) and to refer to the Docs as well.
You can create dashDB service on your bluemix, and copy / upload your data to Bluemix dashDB database, using dashDB VCAP Credentials to connect to it from your adapter, or you can bind your dashDB service to you application on Bluemix.