I am new to Azure and have no prior experience or knowledge regarding working with Azure data warehouse systems (now Azure Synapse Analytics Framework)
I have access to a "read only" data warehouse (not in Azure) that looks like this:
I want to replicate this data warehouse as it is on Azure cloud. Can anyone point me to the right direction (video tutorials or documentation) and the number of steps involved in this process? There are around 40 databases in this warehouse. And what if I wanted to replicated only specific ones?
We can't do that you only have the read only permisson. No matter which data warehouse, we all need the server admin or database owner permission to do the database replicate.
You can easily get this from the all documents relate to the database backup/migrate/replicate, for example: https://learn.microsoft.com/en-us/sql/t-sql/statements/backup-transact-sql?view=sql-server-ver15#permissions,
If you have enough permission then you can to that. But for Azure SQL datawarehouse, now we called SQL pool (formerly SQL DW), we can't replicate other from on-premise datawarehouse to Azure directly.
The official document provide a way import the data into to Azure SQL pool((formerly SQL DW)):
Once your dedicated SQL pool is created, you can import big data with
simple PolyBase T-SQL queries, and then use the power of the
distributed query engine to run high-performance analytics.
You also could use other ETL tool to achieve the data migration from on-premise datawarehouse to Azure. For example using Data Factory, combine these two tutorials:
Copy data to and from SQL Server by using Azure Data Factory
Copy and transform data in Azure Synapse Analytics by using Azure
Data Factory
Related
What's the backend database query of this Microsoft Dataverse Analytics dashboard?
I'm trying to workaround Dataverse analytics by accessing the transactional database behind that dashboard, I'm interested in getting Daily Active Users (DAU) shown above but via a SQL query and reading directly from the backend database.
It appears that the DB is this https://learn.microsoft.com/en-us/dynamics365/customer-engagement/web-api/entitytypes?view=dynamics-ce-odata-9 but I have not been able to comprehend the data model and I'm unable to find the tables to get DAU. Any thoughts?
Thanks
Basically you have to do everything what is MS doing in behind the scenes. CRM online is SaaS model and we don’t have access to Azure SQL server directly. But what you can do is, one of these options:
Use “Data export service” to replicate the data to your own Azure SQL server, then build Power BI on your own from the data
You can use REST Web API to pull the data and visualize (May not be so much flexible)
Based on your need and urgency, you may wait or use preview version of TDS endpoint, for read-only direct SQL access. Read more
We have two Azure PaaS SQL Servers, one in our production resource group and another in our QA resource group.
I'm looking for the best way to 'copy' the data from an Azure SQL DB in the production resource group to an Azure SQL DB in the QA resource group. As part of this process the data must undergo some simple transformations to anonymise the data before it lands in the QA DB.
We have deletion locks in place on the resource groups so deleting and recreating databases is an option I'd rather avoid.
The 'copying' can take place on a weekly schedule and it would be acceptable to drop any existing data in the target Azure SQL DB each time the process is run.
What are the most suitable options in Azure to accomplish this?
Go with Azure Data Factory. You can use it for pretty much anything that SSIS was used for onsite, and more.
How to update or delete data in azure sql DB using azure stream analytics
Currently, Azure Stream Analytics (ASA) only supports inserting (appending) rows to SQL outputs (Azure SQL Databases, and Azure Synapse Analytics).
You should consider to use workarounds to enable UPDATE, UPSERT, or MERGE on SQL databases, with Azure Functions as the intermediary layer.
You can find more information about such workarounds in this MS article.
Firstly, we need to know what is Azure Stream Analytics.
An Azure Stream Analytics job consists of an input, query, and an output. Stream Analytics ingests data from Azure Event Hubs, Azure IoT Hub, or Azure Blob Storage. The query, which is based on SQL query language, can be used to easily filter, sort, aggregate, and join streaming data over a period of time. You can also extend this SQL language with JavaScript and C# user defined functions (UDFs). You can easily adjust the event ordering options and duration of time windows when preforming aggregation operations through simple language constructs and/or configurations.
Azure Stream Analytics now natively supports Azure SQL Database as a source of reference data input. Developers can author a query to extract the dataset from Azure SQL Database, and configure a refresh interval for scenarios that require slowly changing reference datasets.
That means that you can not insert or update data in azure sql DB using Azure Stream Analytics.
Azure Stream Analytics is not a database manage tool.
Hope this helps.
Question: Data stored/generated in on-premises SQL server needs to be logged to app insights. This data represents both the temporary data that is passed to stored procedures and also concrete data that is stored in tables. I am asking this question to figure out if there is any direct way to do this. I have come up with the following options till now, in case there is no direct solution please let me know which is the most suited option for the task.
Options:
Log that data into a temporary table and have a windows service/ azure web job that picks up that data in batches and does the logging.
Use a SQL CLR stored procedure that logs the data directly to app insights using Azure App Insights DLL
Use Azure Data Factory to export the data from on-premise SQL server to some Azure based storage and then to App Insights
I'm researching the differences between AWS and Azure for my company. We going to make an web-based application. Which is going to be across 3 regions, each region needs to have a MS SQL database.
But I can't figure how to do the following with AWS: the databases need to sync between each region (2 way). So the data stays the same on every Database.
Why we want this? For example a customer* from Eu adds a record to the database. Now this database needs to sync with the other regions. Resulting that a customer form the region US can see the added records. (*Customers can add products to the database)
Do you guys have any idea how we can achieve this?
it's a requirement to use Ms SQL.
If you are using SQL on EC2 instances then the only way to achieve multi-region, multi-master for MS SQL Server is to use Peer-to-Peer Transactional Replication, however it doesn't protect against individual row conflicts.
https://technet.microsoft.com/en-us/library/ms151196.aspx
This isn't a feature of AWS RDS for MS SQL, however there is another product for multi-region replication that's available on the AWS marketplace, but it only works for read replicas.
http://cloudbasic.net/aws/rds/alwayson/
At present AWS doesn't support read replicas for SQL server RDS databases.
However replication between AWS RDS sql server databases can be done using DMS (database migration service). Refer below link for more details
https://aws.amazon.com/blogs/database/introducing-ongoing-replication-from-amazon-rds-for-sql-server-using-aws-database-migration-service/