I am looking for any connector or any other ways that i can connect and get data from snowflake to IoT Sitewise.
The ways of connecting to Snowflake are all documented here: https://docs.snowflake.com/en/user-guide/conns-drivers.html
Related
My client has his data stored on SQL Server hosted on an on-premise network. I established a VPN connection from Google to the network, but I don't know how to follow from here. My final goal is to process his data using cloud functions. Any suggestions?
PS: I read that Shared VPC can be used to accomplish this, but I don't have a proper organization for this purpose :/
Edit: I followed the suggestions on the comments but now I'm missing to extract the data since pyodbc is not pre-installed on Cloud Functions. Any ideas oh how to query an on-prem database on SQL Server through Cloud Functions?
Is there a way to connect to a sample hadoop DB online. I'm trying to test the kerberos connection but I need to test on an actual Hadoop database. I just need to read from the database, is there such service?
Thanks!
We found in documentation the following:
AWS Glue can connect to the following data stores by using the JDBC protocol:
• Amazon Redshift
• Amazon Relational Database Service (MySQL, PostgreSQL, Aurora, and MariaDB)
• Publicly accessible (Amazon Redshift, MySQL, PostgreSQL, Aurora, and MariaDB) databases
Is it possible to make a JDBC connection with SQL Server for data stores? I'm trying create to Crawler with data store in SQL Server.
Should I create new instance of SQL Server on RDS?
Thanks
It would be possible if the correct JDBC driver was integrated into AWS Glue but it is not. One of the downsides of a serverless environment is you can't add drivers to the server.
AWS reps have informed me that at present, you cannot connect to a database outside an Amazon VPC. This is obviously frustrating. I believe they are putting it on the roadmap.
If you are able to set up an RDS instance with a database they didn't explicitly name, you should try setting up a Glue job to connect to it. If it fails at first because it lacks the nece, I would imagine you should be able to connect to it by supplying the JDBC driver
You can connect to SQL Server using JDBC, here is a article on how to do it.
https://www.progress.com/tutorials/jdbc/accessing-data-using-jdbc-on-aws-glue
Although it's for Salesforce, you can use the similar steps for SQL Server too. Just replace Salesforce JDBC driver with SQL Server JDBC driver.
I have an on premise oracle database. Can I use anything on AWS e.g. API Gateway to query the database and expose the results via API? I know I could do API Gateway -> Lambda -> Oracle DB where the code in the Lambda function would query the database (assuming query takes less than 5 mins). Are there any other easy options that would be serverless and with minimal amount of code?
Basically I would like to find the simplest way to create an API layer over the top of an existing on premise oracle database so that applications (hosted on AWS) can access this data without connecting directly to the database. Does AWS provide anything out of the box?
There does not seem to be an out of the box way provided by AWS to connect API Gateway to your on premise Oracle DB. So basically the way you provided (API Gateway->Lambda->Oracle) should be the way to go.
Now the question is if you want to connect to your Oracle directly or if your want to create a replication of your database in RDS and create a synch mechanism between RDS and your on premise Oracle DB to keep the DB highly responsive and available (in case of network failure between AWS and your local network). I think that depends on how you access your DB on premise.
If your won't create an replica in RDS you should at least use a VPN connection to your local network to keep data transfer from on premise Oracle to AWS RDS secure.
Yes it is possible to use AWS Lambda and expose the API through API
Gateway. But that is the easy part.
The tough part is to get your On-Premise database connected to AWS
infrastructure. If you have an on premise database, and you are
working in a large enterprise, you will need to get through a lot of
approvals to setup a VPN or a AWS Direct Connect.
The ideal solution is to use AWS Direct Connect to extend your
corporate infrastructure to connect to AWS and then use Lambda to
connect to the DB.
Also there is no out of the box solution in AWS to connect to
OracleDB. At the most, you can wrap all business logic in Stored
Procedures, and execute them in the lambda function. You can always
use the JDBC from Lambda to connect and query your database.
Try this from AWS Marketplace https://aws.amazon.com/marketplace/pp/B01MU8W71L
Is this possible through some tcp redirection? If so, how?
Edit: I'd like to connect to on-premises SQL from a worker role to do some custom ATS-SQL synchronization.
About 18 months ago, Clemens Vasters posted an article about building a Port Bridge over the Service Bus, which demonstrates how to connect to an on-premises SQL Server database. No use of Azure Connect. The article is here.
This is just a sample, and not necessarily considered production-ready, but should hopefully get you going in the right direction.
You can try out the new Service Bus EAI & EDI Labs Release. It provides connectivity to various on-premise systems including SQL Server. The connectivity is provided using Service Bus Relay, and may be exactly what you are asking for. In the process, we can do transformations of the data on the cloud etc.
It is not in production yet, but we would like to get feedback on this.
http://blogs.msdn.com/b/windowsazure/archive/2011/12/16/announcing-the-service-bus-eai-amp-edi-labs-release.aspx
Thanks,
Sameer [MSFT]
sameerch # microsft.com
Have you looked at the Sync Framework to synchronise your data?
Sync Framework: SQL Server to SQL Azure Synchronization
Much as I love the idea of routing raw TCP traffic through the service bus the performance is going to be pants (sorry Clemens...)
Have you taken a look at WIndows Azure COnnect
http://www.windowsazure.com/en-us/home/tour/virtual-network/
This will basically give you a VPN like connection back to your on-premise SQL Server. If you can make it work (i.e. can install the agent locally) then this is probably the best approach.
If all you need is data sync, have you looked at SQL Azure data sync?
http://social.technet.microsoft.com/wiki/contents/articles/sql-azure-data-sync-overview.aspx