Basically, I need to work on streaming data from a SQL Server table using Node and I was wondering if there is a server somewhere that would allow for some test data to work with.
You can use Ragic or Amazon.com's simpledb or Cloud SQL from Google or Dedicated DB Servers or follow some more at cloudboost
Related
I typically use pyodbc when running jupyter notebooks from my machine, but this does not work on Azure ML. My assumption is that this is being caused by Azure ML not knowing if I'm on my company's network as I typically need a VPN to the server if I'm not in office. The only solutions I can find online involve copying the data over on Azure Data Factory however I need to avoid this if possible as there are many tables I will need to experiment with, but nothing is intended to be long term and I'm unsure what I will even end up using.
Ideally there is a way to make pyodbc work but any other suggestions are welcome. I have researched integration runtimes but was unsure if that would solve my problem here.
The only solutions I can find online involve copying the data over on
Azure Data Factory however I need to avoid this if possible as there
are many tables I will need to experiment with, but nothing is
intended to be long term and I’m unsure what I will even end up using.
Ideally there is a way to make pyodbc work but any other suggestions
Unfortunately, the on-Prem SQL Server is not supported as a Data Source in Azure ML.
Only the Data sources available below are supported:-
Approach1)
You can copy your data from the on-premises SQL database to Azure SQL via copy tool in Azure Data factory and connect to Azure SQL via Azure Machine learning by directly connecting to it via Datasource like below:-
You can also use Self-hosted integration run time to connect to your SQL server on-prem in your data factory:-
Click on Option 2 to download the Integration runtime and set it in your local machine with the Registration keys mentioned above:-
Approach2)
If there’s a large data You can automate your entire copy process from the on-prem SQL server to Azure SQL by using the Azure DevOps pipeline.
References:-
https://learn.microsoft.com/en-us/answers/questions/775844/unable-to-connect-sql-server-to-azure-ml-pipeline By Ramr-msft
How To: Azure Data Factory CI/CD with Azure DevOps pipelines — The YAML WAY! | by Raghavendra Bharadwaj | Servian
Is there a way by which we can establish a connection from AWS to sql server and pull the data. I am aware of the method of using cdata connector with glue jobs and it looks promising but I want to explore options here. The idea is to pull the data from sql server to s3 bucket.
You can directly use from_options method of GLUE to pull data from below data stores
s3, mysql, postgresql, redshift, sqlserver, oracle, and dynamodb
and dump wherever you required.
More Information on
https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-crawler-pyspark-extensions-dynamic-frame-reader.html
My client has his data stored on SQL Server hosted on an on-premise network. I established a VPN connection from Google to the network, but I don't know how to follow from here. My final goal is to process his data using cloud functions. Any suggestions?
PS: I read that Shared VPC can be used to accomplish this, but I don't have a proper organization for this purpose :/
Edit: I followed the suggestions on the comments but now I'm missing to extract the data since pyodbc is not pre-installed on Cloud Functions. Any ideas oh how to query an on-prem database on SQL Server through Cloud Functions?
We found in documentation the following:
AWS Glue can connect to the following data stores by using the JDBC protocol:
• Amazon Redshift
• Amazon Relational Database Service (MySQL, PostgreSQL, Aurora, and MariaDB)
• Publicly accessible (Amazon Redshift, MySQL, PostgreSQL, Aurora, and MariaDB) databases
Is it possible to make a JDBC connection with SQL Server for data stores? I'm trying create to Crawler with data store in SQL Server.
Should I create new instance of SQL Server on RDS?
Thanks
It would be possible if the correct JDBC driver was integrated into AWS Glue but it is not. One of the downsides of a serverless environment is you can't add drivers to the server.
AWS reps have informed me that at present, you cannot connect to a database outside an Amazon VPC. This is obviously frustrating. I believe they are putting it on the roadmap.
If you are able to set up an RDS instance with a database they didn't explicitly name, you should try setting up a Glue job to connect to it. If it fails at first because it lacks the nece, I would imagine you should be able to connect to it by supplying the JDBC driver
You can connect to SQL Server using JDBC, here is a article on how to do it.
https://www.progress.com/tutorials/jdbc/accessing-data-using-jdbc-on-aws-glue
Although it's for Salesforce, you can use the similar steps for SQL Server too. Just replace Salesforce JDBC driver with SQL Server JDBC driver.
I developed an app with vb.net to update a SQL Server database.
The app is connected with SQL Server in my computer and seems working very well.
But my target is to put this database in the common mass storage to be updated with this app from 20 people (20 client PC).
My questions are:
how to do to install the database in the common mass storage?
should the SQL Server Express be installed in the 20 client computers?
how to connect the app to the database (located in the common server) using ADO.net?
Thanks in advance
It really depends. If everybody is suppose to use the same data I would look at putting up a sql server that everyone can access on the network. If the data needs to be used from computers outside of a local network I would look at getting a database from a web host. For these instances you would have to update the server to your connection string to the url of the new sql server. If everyone is using there own data it is ok to use there own sql express instance.