My client has his data stored on SQL Server hosted on an on-premise network. I established a VPN connection from Google to the network, but I don't know how to follow from here. My final goal is to process his data using cloud functions. Any suggestions?
PS: I read that Shared VPC can be used to accomplish this, but I don't have a proper organization for this purpose :/
Edit: I followed the suggestions on the comments but now I'm missing to extract the data since pyodbc is not pre-installed on Cloud Functions. Any ideas oh how to query an on-prem database on SQL Server through Cloud Functions?
Related
I typically use pyodbc when running jupyter notebooks from my machine, but this does not work on Azure ML. My assumption is that this is being caused by Azure ML not knowing if I'm on my company's network as I typically need a VPN to the server if I'm not in office. The only solutions I can find online involve copying the data over on Azure Data Factory however I need to avoid this if possible as there are many tables I will need to experiment with, but nothing is intended to be long term and I'm unsure what I will even end up using.
Ideally there is a way to make pyodbc work but any other suggestions are welcome. I have researched integration runtimes but was unsure if that would solve my problem here.
The only solutions I can find online involve copying the data over on
Azure Data Factory however I need to avoid this if possible as there
are many tables I will need to experiment with, but nothing is
intended to be long term and I’m unsure what I will even end up using.
Ideally there is a way to make pyodbc work but any other suggestions
Unfortunately, the on-Prem SQL Server is not supported as a Data Source in Azure ML.
Only the Data sources available below are supported:-
Approach1)
You can copy your data from the on-premises SQL database to Azure SQL via copy tool in Azure Data factory and connect to Azure SQL via Azure Machine learning by directly connecting to it via Datasource like below:-
You can also use Self-hosted integration run time to connect to your SQL server on-prem in your data factory:-
Click on Option 2 to download the Integration runtime and set it in your local machine with the Registration keys mentioned above:-
Approach2)
If there’s a large data You can automate your entire copy process from the on-prem SQL server to Azure SQL by using the Azure DevOps pipeline.
References:-
https://learn.microsoft.com/en-us/answers/questions/775844/unable-to-connect-sql-server-to-azure-ml-pipeline By Ramr-msft
How To: Azure Data Factory CI/CD with Azure DevOps pipelines — The YAML WAY! | by Raghavendra Bharadwaj | Servian
We are soon to migrate our back-end from an Amazon SQL Server RDS to another identically set up RDS server. We intend that our systems will not be interrupted because they connect using an AWS alias that we will swap to point to the new RDS server.
While we believe all our systems use the alias and not direct server addressing, it is possible that we have missed the change in some application code.
Can anyone advise a method we can use to identify applications that are connecting without the alias?
Is there anything to indicate the connection method from with SQL Server Profiler?
Is it possible to log what applications use the alias from AWS?
My question is as follows: I have local servers with SQL Server database and I have a server in google cloud also with SQL Server database.
I link these servers through the Linked Server functionality of SQL Server itself, where one server can see the other.
This link is OK and working.
My big question is regarding security, when making this link between my local servers and the google-cloud server, how vulnerable am I to the network?
What is my security risk in doing this?
The Linked Servers function uses a driver named OLE DB to connect. How vulnerable you're really depends on the version of said driver, if you're using the last version it supports TLS 1.2 which ensures that the information is private and secured.
The most notable risk resides on how secure your Google Cloud SQL account, besides that the GCP is a secure environment for your DB, as mentioned in the FAQ:
Cloud SQL customer data is encrypted when stored in database tables, temporary files, and backups. External connections can be encrypted by using SSL, or by using the Cloud SQL Proxy.
Google encrypts and authenticates all data in transit at one or more network layers when data moves outside physical boundaries not controlled by Google or on behalf of Google.
I have an on premise oracle database. Can I use anything on AWS e.g. API Gateway to query the database and expose the results via API? I know I could do API Gateway -> Lambda -> Oracle DB where the code in the Lambda function would query the database (assuming query takes less than 5 mins). Are there any other easy options that would be serverless and with minimal amount of code?
Basically I would like to find the simplest way to create an API layer over the top of an existing on premise oracle database so that applications (hosted on AWS) can access this data without connecting directly to the database. Does AWS provide anything out of the box?
There does not seem to be an out of the box way provided by AWS to connect API Gateway to your on premise Oracle DB. So basically the way you provided (API Gateway->Lambda->Oracle) should be the way to go.
Now the question is if you want to connect to your Oracle directly or if your want to create a replication of your database in RDS and create a synch mechanism between RDS and your on premise Oracle DB to keep the DB highly responsive and available (in case of network failure between AWS and your local network). I think that depends on how you access your DB on premise.
If your won't create an replica in RDS you should at least use a VPN connection to your local network to keep data transfer from on premise Oracle to AWS RDS secure.
Yes it is possible to use AWS Lambda and expose the API through API
Gateway. But that is the easy part.
The tough part is to get your On-Premise database connected to AWS
infrastructure. If you have an on premise database, and you are
working in a large enterprise, you will need to get through a lot of
approvals to setup a VPN or a AWS Direct Connect.
The ideal solution is to use AWS Direct Connect to extend your
corporate infrastructure to connect to AWS and then use Lambda to
connect to the DB.
Also there is no out of the box solution in AWS to connect to
OracleDB. At the most, you can wrap all business logic in Stored
Procedures, and execute them in the lambda function. You can always
use the JDBC from Lambda to connect and query your database.
Try this from AWS Marketplace https://aws.amazon.com/marketplace/pp/B01MU8W71L
Is this possible through some tcp redirection? If so, how?
Edit: I'd like to connect to on-premises SQL from a worker role to do some custom ATS-SQL synchronization.
About 18 months ago, Clemens Vasters posted an article about building a Port Bridge over the Service Bus, which demonstrates how to connect to an on-premises SQL Server database. No use of Azure Connect. The article is here.
This is just a sample, and not necessarily considered production-ready, but should hopefully get you going in the right direction.
You can try out the new Service Bus EAI & EDI Labs Release. It provides connectivity to various on-premise systems including SQL Server. The connectivity is provided using Service Bus Relay, and may be exactly what you are asking for. In the process, we can do transformations of the data on the cloud etc.
It is not in production yet, but we would like to get feedback on this.
http://blogs.msdn.com/b/windowsazure/archive/2011/12/16/announcing-the-service-bus-eai-amp-edi-labs-release.aspx
Thanks,
Sameer [MSFT]
sameerch # microsft.com
Have you looked at the Sync Framework to synchronise your data?
Sync Framework: SQL Server to SQL Azure Synchronization
Much as I love the idea of routing raw TCP traffic through the service bus the performance is going to be pants (sorry Clemens...)
Have you taken a look at WIndows Azure COnnect
http://www.windowsazure.com/en-us/home/tour/virtual-network/
This will basically give you a VPN like connection back to your on-premise SQL Server. If you can make it work (i.e. can install the agent locally) then this is probably the best approach.
If all you need is data sync, have you looked at SQL Azure data sync?
http://social.technet.microsoft.com/wiki/contents/articles/sql-azure-data-sync-overview.aspx