How do I detect database connection issues from Elixir Ecto? - database

I am writing an Elixir project that connects to a Postgres database via Ecto. The database server is on a different server from the application itself, and is more subject to outages that would not affect the Elixir project than if they were running on the same hardware.
When the application starts normally, the database connection seems to be made automatically, and everything works fine. But if there is a connection error, Ecto simply spews any errors into the log.
What I would like to do is detect the current connection status, and report that information via a simple Plug route to an external load balancer, such that traffic can be routed to a separate server with an active connection.
The trouble is, I'm unsure how to determine if Ecto has a viable connection to the database, apart from listening to the log, which doesn't then report that the database connection has been restored.
What can I call to determine if an Ecto connection is live and usable, preferably without making no-op queries against that connection?

Ecto simply spews any errors into the log.
This is the default strategy with enabled backoff for the connection. You can disable the backoff by setting backoff_type to :stop in repo options and deal with errors by yourself.
What can I call to determine if an Ecto connection is live and usable, preferably without making no-op queries against that connection?
I think you can check if the connection "usable" only by using it :)
try do
Ecto.Adapters.SQL.query(MyApp.Repo, "SELECT 1")
rescue
e in DBConnection.ConnectionError -> :down
end
BTW, the module mentioned above is the default pool module for the connection you can rely on its internals(don't) and do the checkout manually. Own pool can be implemented instead of this one :)

To detect whether a connection to the database is live and usable, you can try connecting to it directly outside of Ecto using :gen_tcp.connect/3. The code could look something like this:
case :gen_tcp.connect('127.0.0.1', 5432, []) do
{:ok, _} ->
# code for handling ok scenario
{:error, error} ->
# code for handling error scenario. Typically, you'll get the econnrefused error when the server is down
end
Note that Ecto gets the details of the server to connect to through the Mix config you set for it. If you want to change the server it connects to, you'll have to run Application.put_env/4 and then restart the Ecto repository for Ecto to recognize the new config. For example:
Application.put_env(:my_app, MyApp.Repo, [adapter: Ecto.Adapters.Postgres, username: "postgres", password: "postgres",database: "new_db", hostname: "different_host.com", pool_size: 20])
Supervisor.stop(MyApp.Repo)

Related

Azure Function Database Connection

I have a Python package that I am able to run successfully on an Azure Data Science Virtual Machine. However, when I push it to Azure as a Function, I cannot successfully make a database connection. I was getting an error that the ODBC Driver 13 for SQL Server was not supported, so I changed the driver to ODBC Driver 17 for SQL Server and now I am NOT getting an error, but no data is being returned for a query that I know should return data.
Is there any other reason that data would not be returned? Firewall issues? do I need to add a binding? Do I need to separate out the connection string to feed each part (e.g., Driver, UID, PWD) into pyodbc.connect() separately? Right now I am feeding it in like this:
setting = os.environ("CONNECTIONSTRING")
conn = pyodbc.connect(setting)
This query works fine returning data when I run it on the VM using this code, just not as a Function.
(Note, this is different from my previous post regarding reading the Azure App Setting. That problem has been solved).
There are many parts where this could be breaking.
I'd suggest start by having a Profiler or Extended Events trace on your SQL Server to verify whether a connection is even being established. If not then you need to work through the the various points of connectivity to find out where it breaks. The identity, firewall, NSGs etc might all come into play here.
Once you see a connection then you can play with permissions to ensure that your query then returns your data.
Without a full picture of your infrastructure and settings it is hard to pin it down further.
Turns out it was not a database connectivity issue like I thought it was; it was a code error.

What does `test-on-borrow` do?

What does it do? How does it work? Why am I supposed to test the database connection before "borrowing it from the pool"?
I was not able to find any related information as to why I should be using it. Just how to use it. And it baffles me.
Can anyone provide some meaningful definition and possibly resources to find out more?
"test-on-borrow" indicates that a connection from the pool has to be validated usually by a simple SQL validation query defined in "validationQuery". These two properties are commonly used in conjunction to make sure that the current connections in the pool are not stale (no longer connected to the DB actively as a result of a DB restart, or timeouts enforced by the DB, or whatever other reason that might cause stale connections). By testing the connections on borrow, the application can automatically reconnect to the DB using new connections (and dropping the invalid ones) without a manual restart of the app and thus preventing DB connection errors in the app.
You can find more information on jdbc connection pool attributes here:
https://tomcat.apache.org/tomcat-8.0-doc/jdbc-pool.html#Common_Attributes

Block all connections to a database db2

I'm working on logs. I want to reproduce a log in which the application fails to connect to the server.
Currently the commands I'm using are
db2 force applications all
This closes all the connections and then one by one I deactivate each database using
db2 deactivate db "database_name"
What happens is that it temporary blocks the connections and after a minute my application is able to create the connection again, due to which I am not able to regenerate the log. Any Ideas how can I do this?
What you are looking for is QUIESCE.
By default users can connect to a database. It becomes active and internal in-memory data structures are initialized. When the last connection closes, the database becomes inactive. Activating a database puts and leaves them initialized and "ready to use".
Quiescing the database puts them into an administrative state. Regular users cannot connect. You can quiesce a single database or the entire instance. See the docs for some options to manage access to quiesced instances. The following forces all users off the current database and keeps them away:
db2 quiesce db immediate
If you want to produce a connection error for an app, there are other options. Have you ever tried to connect to a non-estisting port, Db2 not listening on it? Or revoke connect privilege for that user trying to connect.
There are several testing strategies that can be used, they involve disrupting the network connection between client and server:
Alter the IP routing table on the client to route the DB2 server address to a non-existent subnet
Use the connection via a proxy software that can be turned off, there is a special proxy ToxiProxy, which was designed for the purpose of testing network disruptions
Pull the Ethernet cable from the client machine, observe then plug it back in (I've done this)
This has the advantage of not disabling the DB2 server for other testing in progress.

Automatic failover with SQL mirroring and connection strings

I have 3 servers set up for SQL mirroring and automatic failover using a witness server. This works as expected.
Now my application that connects to the database, seems to have a problem when a failover occurs - I need to manually intervene and change connection strings for it to connect again.
The best solution I've found so far involves using Failover Partner parameter of the connection string, however it's neither intuitive nor complete: Data Source="Mirror";Failover Partner="Principal" found here.
From the example in the blog above (scenario #3) when the first failover occurs, and principal (failover partner) is unavailable, data source is used instead (which is the new principal). If it fails again (and I only tried within a limited period), it then comes up with an error message. This happens because the connection string is cached, so until this is refreshed, it will keep coming out with an error (it seems connection string refreshes ~5 mins after it encounters an error). If after failover I swap data source and failover partner, I will have one more silent failover again.
Is there a way to achieve fully automatic failover for applications that use mirroring databases too (without ever seeing the error)?
I can see potential workarounds using custom scripts that would poll currently active database node name and adjust connection string accordingly, however it seems like an overkill at the moment.
Read the blog post here
http://blogs.msdn.com/b/spike/archive/2010/12/15/running-a-database-mirror-setup-with-the-sqlbrowser-service-off-may-produce-unexpected-results.aspx
It explains what is happening, the failover partner is actually being read from the sql server not from your config. Run the query in that post to find out what is actually being used as the failover server. It will probably be a machine name that is not discoverable from where your client is running.
You can clear the application pool in the case a failover has happened. Not very nice I know ;-)
// ClearAllPools resets (or empties) the connection pool.
// If there are connections in use at the time of the call,
// they are marked appropriately and will be discarded
// (instead of being returned to the pool) when Close is called on them.
System.Data.SqlClient.SqlConnection.ClearAllPools();
We use it when we change an underlying server via SQL Server alias, to enforce a "refresh" of the server name.
http://msdn.microsoft.com/en-us/library/system.data.sqlclient.sqlconnection.clearallpools.aspx
The solution is to turn connection pooling off Pooling="false"
Whilst this has minimal impact on small applications, I haven't tested it with applications that receive hundreds of requests per minute (or more) and not sure what the implications are. Anyone care to comment?
Try this connectionString:
connectionString="Data Source=[MSSQLPrincipalServerIP,MSSQLPORT];Failover Partner=[MSSQLMirrorServerIP,MSSQLPORT];Initial Catalog=DatabaseName;Persist Security Info=True;User Id=userName; Password=userPassword.; Connection Timeout=15;"
If you are using .net development, you can try to use ObjAdoDBLib or PigSQLSrvLib and PigSQLSrvCoreLib, and the code will become simple.
Example code:
New object
ObjAdoDBLib
Me.ConnSQLSrv = New ConnSQLSrv(Me.DBSrv, Me.MirrDBSrv, Me.CurrDB, Me.DBUser, Me.DBPwd, Me.ProviderSQLSrv)
PigSQLSrvLib or PigSQLSrvCoreLib
Me.ConnSQLSrv = New ConnSQLSrv(Me.DBSrv, Me.MirrDBSrv, Me.CurrDB, Me.DBUser, Me.DBPwd)
Execute this method to automatically connect to the online database after the mirror database fails over.
Me.ConnSQLSrv.OpenOrKeepActive
For more information, see the relevant links.
https://www.nuget.org/packages/ObjAdoDBLib/
https://www.nuget.org/packages/PigSQLSrvLib/
https://www.nuget.org/packages/PigSQLSrvCoreLib/

Can jdbc connections be recovered?

Can jdbc connections which are closed due to database un-availability be recovered.
To give back ground I get following errors in sequence. It doesn't look to be manual re-start. The reason for my question is that I am told that the app behaved correctly without
the re-start. So if the connection was lost, can it be recovered, after a DB re-start.
java.sql.SQLException: ORA-12537: TNS:connection closed
java.sql.SQLRecoverableException: ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
IBM AIX RISC System/6000 Error: 2: No such file or directory
java.sql.SQLRecoverableException: ORA-01033: ORACLE initialization or shutdown in progress
No. The connection is "dead". Create a new connection.
A good approach is to use a connection pool, which will test if the connection is still OK before giving it to you, and automatically create a new connection if needed.
There are several open source connection pools to use. I've used Apache's JDCP, and it worked for me.
Edited:
Given that you want to wait until the database comes back up if it's down (interesting idea), you could implement a custom version of getConnection() that "waits a while and tries again" if the database doesn't respond.
p.s. I like this idea!
The connection cannot be recovered. What can be done is to failover the connection to another database instance. RAC and data guard installations support this configuration.
This is no problem for read-only transactions. However for transactions that execute DML this can be a problem, especially if the last call to the DB was a commit. In case of a commit the client cannot tell if the commit call completed or not. When did the DB fail; before executing the commit, or after executing the commit (but not sending back the acknowledgment to the client). Only the application has this logic and can do the right thing. If the application after failing over does not verify the state of the last transaction, duplicate transactions are possible. This is a known problem and most of us experienced it buying tickets or similar web transactions.

Resources