I have the following usage pattern that I'm wondering if there's a known way to deal with it.
Let's say I have a website where a user can build a query to run it against the remote database. The remote database is secure and the user will not have access to it. Therefore, the query, what will be something like: SELECT * FROM myTable will be sent to our web server, and our web server will query the remote DB on another server, receive the results and pass them back in the HTTP response. So, the flow is:
Location1 (Europe): User/browser submits HTTP POST containing the SQL Query.
Location2 (US): HTTP Server receives request, runs SQL against database:
Location3 (Asia): Database runs query, returns data
Location2 (US): HTTP Server receives SQL resultset back. Sends response.
Location1 (Europe): User/browser receives the data back in the rendered webpage.
Supposing that I don't have control of the three locations, we can see that there may be a lot of data transfer latency if the size of the resultset is large. I was wondering if there is any way to do something like the following instead, and if so how it could be done:
Location1 (Europe): User/browser submits HTTP POST containing the SQL Query.
Location2 (US): HTTP Server receives request, sends back QueryID immediately, runs SQL against database, asynchronously.
Location3 (Asia) Database runs query
Location1 (Europe): User/browser receives response from database. (How? It cannot pull directly from DB)
To summarize, if we imagine the resultset is 50MB in size, in the first case, the 50MB would go from:
Asia (DB) -> US (Server) -> Europe (Client)
and in the second case it would go from:
Asia (DB) -> Europe (Client)
You can decouple authentication with authorization to allow more flexible connections between all three entities: Browser, HTTP server, and DB.
To make your second example work you could do:
The HTTP server (US) submits asynchroneously the query to the DB (Asia) and requests a auth token for it.
The HTTP server (US) sends the auth token back to the browser (Europe), while the query is now running.
The browser (Europe) now initiates a second HTTP call against the DB (Asia) using the auth token, and maybe the queryID as well.
The DB will probably need to implement a simple token auth protocol. It should:
Authenticate the incoming auth token.
Retrieve the session.
Start streaming the query result set back to the caller.
For the DB server, there are plenty of out-of-the-box slim docker images you can spin in seconds that implement authorization server and that can listen to the browser using nginx.
As you can see the architecture can be worked out. However, the DB server in Asia will need to be revamped to implement some kind of token authorization. The simplest and widespread strategy is to use OAuth2, that is all the rage nowadays.
Building on #TheImpalers answer:
How about add another table to your remote DB that is just for retrieving query result?
When client asks the backend service for database query, the backend service will generate a UUID or other secure token and tell the DB to run the query and store it under the given UUID. The backend service also returns the UUID to the client who can then retrieve the associated data from the DB directly.
TLDR:
Europe (Client) -> US (Server) -> Asia (Server) -> Asia (DB)
Open a HTTP server in Asia (if not don't have access to same DC/server - rent a different one), then re-direct request from HTTP US -> HTTP Asia, which will connect to local DB & stream the response.
Redirect can either be a public one (302) or a private proxying over VPN if you care about latency & have such possibility.
Frontend talking to DB directly is not a very good pattern, because you can't do any middleware operations that you'll need in a long term (breaking changes, analytics, authorization, redirects, rate-limiting, scalability...)
If your SQL is very heavy & you can't do sync requests with long-lasting TCP connections, set up streaming over websocket server (also in Asia).
Related
Our Application is fetching data From Snowflake privatelink account using JDBC queries , the app is running behind a restricted firewall and proxy , When we run SnowCD it shows many URL are blocked , but if we pass proxy information in snowcd then it succesfully pass all test.
Now when we run our app to connect snowflake and execute queries , those queries which returns small data executes but those who returns large data (3000 rows+) goes in waiting , and after long wait timeout error comes.
Same queries works when data is small.
net.snowflake.client.jdbc.SnowflakeChunkDownloader : Timeout waiting for the download of #chunk0
From this stackoverflow discussion I came to know, that when snowflake JDBC execute a small resultset the response comes directly , if its a large resultset a separate request goes to Internal stage (aws s3) and that url is different than snowflake account url , and if proxy is there this might create problem. Private link don't need proxy parameters but STAGE url's need proxy.
But when i am trying proxy properties in JDBC Url and Tomcat level as well there is no difference, it's not working.
I didn't found any proper Snowflake Documentation to explain this Large ResultSet vs Small Result Set behavior.
I created a simple website with a login page that requires to authenticate only with an email address.
I did not store any login logs nor I enabled any logging functionalities on IIS.
Now, people are asking for the list of emails who logged in the past two days.
I used EntityFramework to connect to the database. This is the authentication method:
public static bool Authenticate(string email)
{
using (var db = new DatabaseEntities())
{
var user = db.Users.Where(x => x.Email.ToLower() == email.ToLower()).FirstOrDefault();
if (user != null)
return true;
}
return false;
}
I have tired this in SQL but nothing relevant will show:
SELECT t.[text]
FROM sys.dm_exec_cached_plans AS p
CROSS APPLY sys.dm_exec_sql_text(p.plan_handle) AS t
Is there a way to access the logs in SQL for every time this line of code was executed?
var user = db.Users.Where(x => x.Email.ToLower() == email.ToLower()).FirstOrDefault();
Perhaps there is a way to access the logs for every post request sent to the IIS server?
If you didn't already enable a logging feature in IIS or SQL Server then there is nothing you can do retroactively unfortunately.
Moving forward you can pretty easily enable out of the box features of SQL Server to log User connections via either SQL Server Audit or a Logon Trigger to store the log to a table.
This article lists a few other methodologies (in addition to what I mentioned above) such as using a Trace.
Unfortunately, the queries sent by the application to the database cannot be viewed in IIS. These query records can only be viewed through SQL Server.
The Logging module in IIS records the communication between the client and the server, which includes the URL status code and time.
Even the Fail Request Tracing module cannot capture SQL Server query records, but only the communication records with SQL Server, such as request response time and success and failure.
Is there a way to filter SQL profiler data to show only data from current user/session?
I tried using the LoginName or SessionLoginName filters, but the problem is that most of the calls are made by the application's web service and I see no indication who called this service.
SQL Server does not have the context of the end client when multiple tiers are involved so there is no trace column you can filter on to identify requests originating from a specific end client session. The easiest method is to trace in an isolated test environment with a single client.
If the web service has an end client session context identifier, the service could specify the client session id as the Application Name in the connection string so that you can filter on a specific client session. However, that should generally be done only in a test environment since a separate connection pool is created for each unique connection string.
I am new to nodejs and I would like to figure out how I could have event listeners from SQL Server trigger a Nodejs function on my server.
For instance, I would like my Nodejs server to receive notifications when there is any change (insert/update) for any given table in my database, which would then trigger an update of the UI on the client-side.
I'm not altogether familiar with SQL Server, but I don't think you'll want to be using your DB server to POSTs to a webhook endpoint anyhow.
You're better off handling this functionality in your application code rather than on your database server.
The general idea is that it's your application that's going to be operating on the database anyhow, so you should have your application send out the the updates to your UI. A high-level step-by-step would look like so:
Client sends request to server
Server receives request and updates database
Upon a successful update of the database, your server sends a response to the client
The client receives the response from the server and adjusts the UI accordingly
I am using google cloud messaging in my web based android application. I want to send a message to all of my android apps through gcm (one by one, not simultaneously). Commonly, my web server sends request to gcm with data and then gcm sends that data to particular app. So if my database contains records of 10 apps then my web server will request gcm 10 times. Is there a way that my web server gives access of database table to gcm. Then gcm using that database table send messages to apps one by one. So my web server does not need to request the gcm server 10 times. Is it possible?
Thanks in advance for your kind reply!
There is no way Google can access your database, but you can send multicast messages to up to 1000 recipients using the registration_ids parameter instead of to in you HTTP request.
See also https://developers.google.com/cloud-messaging/server-ref#downstream
Upd.: you can also subscribe all your clients to a single topic and then send to that topic.
https://developers.google.com/cloud-messaging/topic-messaging