I typically use pyodbc when running jupyter notebooks from my machine, but this does not work on Azure ML. My assumption is that this is being caused by Azure ML not knowing if I'm on my company's network as I typically need a VPN to the server if I'm not in office. The only solutions I can find online involve copying the data over on Azure Data Factory however I need to avoid this if possible as there are many tables I will need to experiment with, but nothing is intended to be long term and I'm unsure what I will even end up using.
Ideally there is a way to make pyodbc work but any other suggestions are welcome. I have researched integration runtimes but was unsure if that would solve my problem here.
The only solutions I can find online involve copying the data over on
Azure Data Factory however I need to avoid this if possible as there
are many tables I will need to experiment with, but nothing is
intended to be long term and I’m unsure what I will even end up using.
Ideally there is a way to make pyodbc work but any other suggestions
Unfortunately, the on-Prem SQL Server is not supported as a Data Source in Azure ML.
Only the Data sources available below are supported:-
Approach1)
You can copy your data from the on-premises SQL database to Azure SQL via copy tool in Azure Data factory and connect to Azure SQL via Azure Machine learning by directly connecting to it via Datasource like below:-
You can also use Self-hosted integration run time to connect to your SQL server on-prem in your data factory:-
Click on Option 2 to download the Integration runtime and set it in your local machine with the Registration keys mentioned above:-
Approach2)
If there’s a large data You can automate your entire copy process from the on-prem SQL server to Azure SQL by using the Azure DevOps pipeline.
References:-
https://learn.microsoft.com/en-us/answers/questions/775844/unable-to-connect-sql-server-to-azure-ml-pipeline By Ramr-msft
How To: Azure Data Factory CI/CD with Azure DevOps pipelines — The YAML WAY! | by Raghavendra Bharadwaj | Servian
I am trying to configure log shipping from on premises server to azure Iaas SQL VM. Please let me know the method and limitations.
Mainly i wanted to know how we will configure network shared path between on prem , azure and how backup and restore will happen
So for the second question.
No different as you would backup and restore things between 2 VM's in your Datacenter. Don't see a point to expand on this one.
As for the first question. You would want to create a Site-to-Site VPN between Azure and your own Datacenter, after that you would treat you Azure VM as a VM in your Datacenter.
As for the setup guide: there are a lot of guides running around in the internet, I'll provide a link to the Azure documentation
, where you can look up both ARM and ASM ways to do that.
I have come to learn that we can push the source data file from our local machine to Azure VM to load the database of VM's SQL Server. I have activated my Azure trial account and have setup SQL Server in a VM. Now, I am really clueless how to load data from my local machine.
How it can be achieved by using SSIS flow? Trying to search over the internet, please share any references.
Thanks
You need to be able to connect from your local machine to the Azure VM. One way of doing this is give your VM a public IP and configure the right firewall rules. An VPN or other tunneling can also work. Your solution also depends on where your SSIS server lives.
I did it by following the steps mentioned in Connect to a SQL Server Virtual Machine on Azure (Resource Manager)
The solution consists of an SQL Database and desktop application working hard with that DB. DB Deployed as Azure SQL Database, application running on Azure Virtual Machine with Windows Server onboard. What is the deployment guidelines for this two services in Azure? I want to achieve max performance. Should they both be located in same resource group?
You will need to get your app and the database located as close as possible so yes the same resource group.
Make sure you do not have Mars enabled (MultipleActiveResultSets in connection string).
Make sure you do use connection pooling.
Read this to understand what you are dealing with:
https://www.simple-talk.com/sql/learn-sql-server/improving-the-quality-of-sql-server-database-connections-in-the-cloud/
Ed
I have a simple ms sql 2012 express database & a simple VB Express 2010 application as its front end.
Can I simply host my MS SQL database on Azure & update the application to interrogate the hosted database? This would mean, the application would still be local, but the DB would be hosted. My offices are distributed & if we could do this it would save quite a bit of hassle.
Do I need to use a virtual machine, cloud services or straight hosting? Its difficult for me to understand the subtleties each service.
I've had a look at the azure services information and the documentation is pitched a little high for me. I just wanted to know if this is relatively simple thing to do & if anyone has has any pointers to How To's or Noob guides to azure.
It's absolutely possible and in fact, super easy to set up.
Check out this simple step-by-step instruction on how to set up SQL Server DB on Windows Azure - https://www.windowsazure.com/en-us/develop/net/common-tasks/sql-azure-management/
You can even connect to it from your SQL Server Management Studio just like you would connect to a local database or a database on your network.
Ideally - yes, you can change the connection string used by the VB desktop app to reference the Windows Azure SQL Database instance instead of the local database. That said, SQL Database is not 100% compatible with SQL Server. Be sure to check out the differences at http://msdn.microsoft.com/en-us/library/windowsazure/ff394115.aspx
Can I simply host my MS SQL database on Azure & update the application
to interrogate the hosted database?
Yes, your SQL Azure database is accessible via a connection string, but you have to make sure that you create a firewall rule in SQL Azure to allow your locally hosted or remotely hosted applications to access the database. Everything is turned off by default.
Do I need to use a virtual machine, cloud services or straight
hosting?
I'm not exactly sure what you're asking here, but a VM may be overkill, as that gives you basically a place to host a VM in Azure. You control the O/S, patching, etc. A cloud service (a web or worker role) is a little less maintenance heavy as the O/S, patches, updates, etc. are maintained by Azure and you just worry about your application. Straight hosting - are you talking about hosting on 1&1 or GoDaddy or something? If so, that's really no different than your VB app running locally and accessing your Azure database. For this option, just make sure you have your firewall rules set properly, otherwise you won't be able to access the database.
Hopefully this helps. Good luck!
Azure has some specific requirements that need to be met (like every table has to have a Clustered Index), and it does not support some functionality that is available on "normal" SQL Server (like replication, FILESTREAM data, or data compression).
Other than that, as long as your app meets the requirements, there should not be a reason for it not to work on Azure. The majority of the functionality not supported by Azure is most likely out6 of the scope of what a "noob" would implement, so you probably are safe.
However, Azure might be overly expensive for your needs. It might be cheaper to go for hosting the database somewhere. This would mean you would only have to change the connection string of the application to point to the new database server (though there could be additional limitations imposed by your hosting provider).
Good luck with whichever option you end up taking!