I'm trying to use AWS QuickSight to analyse some data that is being stored in SQL Server on an Azure SQL server.
According to QuickSight, it can connect to an SQL Server, but whenever I try to validate the connection, the process hangs for about a minute then comes back with 'Cannot open server "..." requested by the login. The login failed.'
I initially suspected that this was an issue with the firewall on the MS SQL server on Azure. Accordingly, I looked up the IP regions here
The region I am using is US West (Oregon) (us-west-2), so I white-listed the IP range 54.70.204.128 to 54.70.204.159 - but I am still getting exactly the same 1 minute (or so) pause, before the error comes back in Quick Sight.
The exact error is:
Your database generated a SQL exception. This can be caused by query timeouts, resource constraints, unexpected DDL alterations before or during a query, and other database errors. Check your database settings and your query, and try again.
If I click "Show Details" then I get a further message saying:
Error details
region: us-west-2
timestamp: XXX
requestId: XXX
sourceErrorCode:40532
sourceErrorMessage: Cannot open server "..." requested by the login. The login failed. ClientConnectionId:*
sourceErrorState: S0001
sourceException: com.microsoft.sqlserver.jdbc.SQLServerException
sourceType: SQLSERVER
Obviously some of the above has been redacted.
I cannot believe that QuickSight cannot connect to an Azure MS SQL database, so I'm wondering if anyone else had had this problem, and what their solution was?
I myself had this issue and it seems many others did. However, as noted above, there is little to no documentation that provides the steps to connect Quicksight and Azure Sql Server.
The issues for myself were primarily in the details that I gave Quicksight. Most connections with Azure sql server database connect seamlessly with your basic information:
Server, Port, Database Name, Username, Password
However, AWS Quicksight was trying to connect to my Azure Sql server with JDBC authentification.
JDBC Authentification requires your Username input to be = "username"#"servername"
Example of Correct Connection:
Server: "servername".database.windows.net
Port: 1433
Database Name: "databasename"
Username: "username"#"servername"
Password: "password"
Lastly, I turned off the SSL checkbox. It did not work with SSL connection.
Please see this document: Relational Data Sources
You can use any of the following relational data stores as data sources for Amazon QuickSight:
Amazon Athena
Amazon Aurora
Amazon Redshift
Amazon Redshift Spectrum
Amazon S3
Amazon S3 Analytics
Apache Spark 2.0 or later
MariaDB 10.0 or later
Microsoft SQL Server 2012 or later
MySQL 5.1 or later
PostgreSQL 9.3.1 or later
Presto 0.167 or later
Snowflake
Teradata 14.0 or later
Note
You can access additional data sources not listed here by linking or importing them through supported data sources.
You can retrieve data from tables and materialized views in PostgreSQL instances, and from tables in all other database instances.
Amazon Redshift clusters, Amazon Athena databases, and Amazon Relational Database Service (RDS) instances must be in AWS. Other database instances must be in one of the following environments to be accessible from Amazon QuickSight:
Amazon EC2
On your local network
In a data center or some other internet-accessible environment
AWS QuickSight document doesn't say it support Azure SQL database, just said the supproted other database environments.
And others have asked this problems in AWS Discussion Forums, no one or AWS QuickSight official given the answer.
Reference: Can Quicksight connect to Azure SQL Database?
What we can guess that it doesn't support Azure SQL for now.
Hope this helps.
According to any Googling I have done, and the responses posted here, it appears that while there is no specific statement from AWS or Azure saying the two cannot be connected, equally there is no response to say that they can. Interestingly, nobody has responded to say that they have already got it working. My feeling at the moment is that it cannot work.
While Azure SQL is not explicitly listed as one of the QuickSight data sources, you can still use it as a data source. TLS/SSL is also supported now and I have tested it personally.
You just need to make sure to use the "username"#"servername" format for the Username, as mentioned by Scotty Smith.
Related
My client has his data stored on SQL Server hosted on an on-premise network. I established a VPN connection from Google to the network, but I don't know how to follow from here. My final goal is to process his data using cloud functions. Any suggestions?
PS: I read that Shared VPC can be used to accomplish this, but I don't have a proper organization for this purpose :/
Edit: I followed the suggestions on the comments but now I'm missing to extract the data since pyodbc is not pre-installed on Cloud Functions. Any ideas oh how to query an on-prem database on SQL Server through Cloud Functions?
My question is as follows: I have local servers with SQL Server database and I have a server in google cloud also with SQL Server database.
I link these servers through the Linked Server functionality of SQL Server itself, where one server can see the other.
This link is OK and working.
My big question is regarding security, when making this link between my local servers and the google-cloud server, how vulnerable am I to the network?
What is my security risk in doing this?
The Linked Servers function uses a driver named OLE DB to connect. How vulnerable you're really depends on the version of said driver, if you're using the last version it supports TLS 1.2 which ensures that the information is private and secured.
The most notable risk resides on how secure your Google Cloud SQL account, besides that the GCP is a secure environment for your DB, as mentioned in the FAQ:
Cloud SQL customer data is encrypted when stored in database tables, temporary files, and backups. External connections can be encrypted by using SSL, or by using the Cloud SQL Proxy.
Google encrypts and authenticates all data in transit at one or more network layers when data moves outside physical boundaries not controlled by Google or on behalf of Google.
I'm trying to build a webservice that talks to a SQL database hosted on a server in our internal network. The service is hosted by Azure as a Web App. Is there a good way of doing this? Do I have to use Azure Sql databases, and if I do, is there a way to have the Azure database act as a proxy for our internal database?
There are already rules permitting connections to the ports on our database server, so I don't think that's the problem. I see a lot of questions regarding connecting to Azure hosted sql databases, but nothing about connecting Azure web apps to other kinds of databases.
The error occurs when I try to call a stored procedure (via generated entity framework code) and is as follows:
Error occurred: System.Data.Entity.Core.EntityException: The underlying provider failed on Open. ---> System.Data.SqlClient.SqlException: A network-related or instance-specific error occurred while establishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections.
Our database is configured to allow remote connections, so what I'm guessing the Web App is having difficulty connecting to our vpn.
Please let me know if you need any additional information.
Thanks,
Josh
You can leverage Azure Hybrid Connections which is a feature of App service. Within App Service, Hybrid Connections can be used to access application resources in other networks. It provides access from your app to an application endpoint and uses Azure Relay service to connect to on-premise.
Check out the below link for more details :
https://learn.microsoft.com/en-us/azure/app-service/app-service-hybrid-connections
First option is to look on azure app service hybrid connection but for you to do that you should have Windows server 2012 or above.
Azure App Service Hybrid
Azure App Service hybrid connection is good if you are pulling small amount of data.
If you are pulling large amount of data or your SQL server version is below server 2012 you have two options:
Azure Site to Site VPN
Azure SQL Data Sync
Azure SQL DB Sync is a feature that available on Azure SQL database. You can create a Azure SQL database on azure and sync your on-premise SQL database or SQL database table to Azure SQL database and you can connect your application to Azure SQL database instead of connecting to on-premise database server. This will increase your performance of your application.
We ended up adding the application to an Azure Virtual network that allowed connections to our on-prem servers. The remaining difficulties were due the wrong port numbers being open.
What was very helpful in debugging this was the Kudu console in Azure, under Advanced tools -> console. There you can run commands from the machine hosting your application like ping, or the below:
sqlcmd -S tcp:servername,1433 -U Username -d databasename -P password -q "SELECT * FROM tablename"
We found in documentation the following:
AWS Glue can connect to the following data stores by using the JDBC protocol:
• Amazon Redshift
• Amazon Relational Database Service (MySQL, PostgreSQL, Aurora, and MariaDB)
• Publicly accessible (Amazon Redshift, MySQL, PostgreSQL, Aurora, and MariaDB) databases
Is it possible to make a JDBC connection with SQL Server for data stores? I'm trying create to Crawler with data store in SQL Server.
Should I create new instance of SQL Server on RDS?
Thanks
It would be possible if the correct JDBC driver was integrated into AWS Glue but it is not. One of the downsides of a serverless environment is you can't add drivers to the server.
AWS reps have informed me that at present, you cannot connect to a database outside an Amazon VPC. This is obviously frustrating. I believe they are putting it on the roadmap.
If you are able to set up an RDS instance with a database they didn't explicitly name, you should try setting up a Glue job to connect to it. If it fails at first because it lacks the nece, I would imagine you should be able to connect to it by supplying the JDBC driver
You can connect to SQL Server using JDBC, here is a article on how to do it.
https://www.progress.com/tutorials/jdbc/accessing-data-using-jdbc-on-aws-glue
Although it's for Salesforce, you can use the similar steps for SQL Server too. Just replace Salesforce JDBC driver with SQL Server JDBC driver.
I'm struggling with Microsoft SQL Data Sync Agent, which I need to setup the windows Azure platform.
In the ms sql data sync agent preview, the "Ping Sync Service" is successfully executed.
My windows firewall is allowing ingoing via port 1433.
The problem is, when I want to "Register":
First, I'm not sure that I enter the correct "Server" and "Database" names.
Second, I'm not sure whether I should use "SQL" or "Windows" Authentication
Third, how do I ensure that the Client Sync Agent has appropriate rights to the database?
Among alot of other trials, my best guess is:
Server " .\SQLEXPRESS"
Database "SiteSqlServer"
but still I get the following 2 error messages:
1) Unable to connect to the Database. PLease check if service account has connectivity, credentials and permissions to connect. Server: "XXX\SQLEXPRESS". Database: "SiteSqlServer".
2) Connection faild. Please check database credentials along with permissions for Microsoft SQL Data Sync Windows service logon account to connect to database with specified credentials.
Other reasons for failure:
Windows Azure SQL databases have to be added on the SQL Data Sync portal.
SQL Server databases lower than SQL Server 2005 SP2 are not supported on the SQL Data Sync.
I'm using: dotnetnuke CMS system and Microsoft webmatrix.
Any idea on how to proceed from here is warmly welcomed!
Thanks!
It seems you are trying to sync SQL Azure with SQL Express which is not supported and not tested. That's why when you try as described above you get that error. Based on my experiences there are issues using with SQL Express and SQL DataSync and SQL Express is not listed helow:
http://msdn.microsoft.com/en-us/library/windowsazure/hh456371.aspx
There are a few articles on internet where a few folks who went ahead and hacked the system to work with some vers'on of SQL Express so you are your own if you wish to try further. An article I found is here: