I get this error every so often when utilizing sqlx with pgx, and I believe it's a configuration error on my end and a db concept I'm not grasping:
error: 'write tcp [redacted-ip]:[redacted-port]->[redacted-ip]:[redacted-port]: write: connection timed out
This occurs when attempting to read from the database. I init sqlx in the startup phase:
package main
import (
_ "github.com/jackc/pgx/stdlib"
"github.com/jmoiron/sqlx"
)
//NewDB attempts to connect to the DB
func NewDB(connectionString string) (*sqlx.DB, error) {
db, err := sqlx.Connect("pgx", connectionString)
if err != nil {
return nil, err
}
return db, nil
}
Any structs responsible for interacting with the database have access to this pointer. The majority of them utilize Select or Get, and I understand those return connections to the pool on their own. There are two functions that use Exec, and they only return the result and error at the end of the function.
Other Notes
My Postgres db supports 100 max_connections
I only showed a few active connections at the time of this error
I am not using SetMaxIdleConnections or SetMaxOpenConnections
Refreshing the page and triggering the request again always works
Any tips on what might be happening here?
EDIT: I did not mention this server is on compose.io, which in turn is hosted on AWS. Is it possible AWS turns these connections into zombies because they've been open for so long and the timeout occurs after unsuccessfully trying them one by one?
With the help of some rough calculations, I've set the maximum lifetime of these connections to 10 minutes. I inserted this code into the init function in the question above to limit the number of open connections, idle connections, and to limit to the life of the connection to 30s.
db.SetConnMaxLifetime(time.Duration(30) * time.Second)
db.SetMaxOpenConns(20)
db.SetMaxIdleConns(20)
Hopefully this helps someone else.
SELECT * FROM pg_stat_activity; is great for nailing down connections as well.
Related
I am using the mongo Db package for golang to find a document. When I run multiple readMongo functions in go routines, the time taken to run the readMongo function increases. Is this a I/O limit of my machine? The documents i'm reading are less then 0.5MB.
func main(){
go readMongo()
go readMongo()
go readMongo()
}
func go readMongo(){
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
client, err := mongo.Connect(ctx, options.Client().ApplyURI("mongodb://localhost:27017"))
if err != nil{
//handle err
}
t1:=time.Now()
collection := client.Database("Data").Collection("myCollection")
if err := collection.FindOne(ctx,bson.M{"_id": "myKey"}).Decode(&data); err!= nil{}
t2:=time.Now()
fmt.Println(t2.sub(t1).Milliseconds())
}
First: connect to the db once, and use the same client for all the goroutines. The mongodb client manages the connection pooling.
With increased concurrency, the performance of individual reads will decrease mainly because of the network bandwidth. All parallel reads will share the same connection to the database, causing slowdowns. In your example, you are reading through multiple connections in parallel, causing the reads to share the network bandwidth. In real-life production loads, the database traffic is usually random and this slowdown for each individual goroutine is not so predictable.
Am getting this Operational Error, periodically probably when the application is not active or idle for long hours. On refreshing the page it will vanish. Am using mssql pyodbc connection string ( "mssql+pyodbc:///?odbc_connect= ...") in Formhandlers and DbAuth of gramex
How Can I keep the connection alive in gramex?
Screenshot of error
Add pool_pre_ping and pool_recycle parameters.
pool_pre_ping will normally emit SQL equivalent to “SELECT 1” each time a connection is checked out from the pool; if an error is raised that is detected as a “disconnect” situation, the connection will be immediately recycled. Read more
pool_recycle prevents the pool from using a particular connection that has passed a certain age. Read more
eg: engine = create_engine(connection_string, encoding='utf-8', pool_pre_ping=True, pool_recycle=3600)
Alternatively, you can add these parameters for FormHandler in gramex.yaml. This is required only for the first FormHandler with the connection string.
kwargs:
url: ...
table: ...
pool_pre_ping: True
pool_recycle: 60
When I connect to database (using standard go sql library) using VPN and VPN interface goes down, there's a 75 seconds timeout when I try to do SQL query, no matter if the interface goes up meanwhile. I'd like to decrease this timeout to some reasonable time, so my application won't be frozen for 75 seconds in such case.
db, err := sql.Open(driverName, dataSourceName)
Is it possible to set it somehow via db variable?
The database/sql package doesn't provide a general way to timeout a call to database/sql.Open. However, individual drivers provide this functionality via the DSN (dataSourceName) connection strings.
https://github.com/lib/pq
sql.Open("postgres", "user=user dbname=dbname connect_timeout=5")
https://github.com/go-sql-driver/mysql
sql.Open("mysql", "user:password#/dbname?timeout=5s")
https://github.com/denisenkom/go-mssqldb
sql.Open("sqlserver", "sqlserver://username:password#host/instance?dial+timeout=5")
etc ...
Starting with Go 1.8, the sql.DB abstraction now accepts context.Context, which can be used to time out connections faster.
func (c *Client) DoLookup(ctx context.Context, id int) (string, error) {
var name string
// create a child context with a timeout
newCtx, cancel := context.WithTimeout(ctx, time.Second)
// release resources used in `newCtx` if
// the DB operation finishes faster than the timeout
defer cancel()
row := c.db.QueryRowContext(newCtx, "SELECT name FROM items WHERE id = ?", id)
err := row.Scan(&name)
if err != nil {
return "", err
}
return name, nil
}
If your DoLookup function doesn't yet take a context.Context (and it really should!) you can create a parent one by calling context.TODO().
Developing with VS 2013 ASP.NET MVC 5 Web Project and Separate Azure hosted SQL Server database.
At the bottom is all my error information from Visual Studio 2013. I've narrowed down the problem and found a link to the Microsoft Description of the problem without a solution. I'm Developing with Database First and Entity Framework 6. ASP.NET 4 MVC & Razor. I connect to a SQL Azure database - I think this is whats falling over i've checked the logs for Azure website etc already
I have delimited text files (that were uploaded to APP_DATA) that I load into a DataTable then use SQL-Bulk Copy to dump content into Azure Database. All works 100% fine so long as my files are only containing a few hundred records. But I need to insert 20MB files with approx 200,000 rows. When I try the big files I get an Error at the point ASP.NET is performing the Bulk Copy. No matter what I set for batch size etc it bails around the 4000 row mark every-time. I've exhausted all options and at my whits end, I even tried Scaling up the Azure database to Business from FREE web. I tried scaling up the website too. Here is the code :
public void BatchBulkCopy(DataTable dataTable, string DestinationTbl, int batchSize,int identity)
{
try {
// Set the timeout.
System.Diagnostics.Debug.WriteLine("Start SQL Bulk Copy");
using (SqlBulkCopy sbc = new SqlBulkCopy("Server=tcp:eumtj4loxy.database.windows.net,1433;Database=AscWaterDB;User ID=HIDDEN#HIDDEN;Password=XXXXXXX;Trusted_Connection=False;Encrypt=True;Connection Timeout=900;", SqlBulkCopyOptions.TableLock))
{
sbc.DestinationTableName = DestinationTbl;
sbc.BulkCopyTimeout = 0;
// Number of records to be processed in one go
sbc.BatchSize = 1000;
// Add your column mappings here
sbc.ColumnMappings.Add("D2001_SPID", "SupplyPointId");
sbc.ColumnMappings.Add("D2002_ServiceCategory", "D2002_ServiceCategory");
sbc.ColumnMappings.Add("D2025_NotifyDisconnection/Reconnection", "D2025_NotifyDisconnectionReconnection");
sbc.ColumnMappings.Add("WaterBatchId", "WaterBatchId");
sbc.ColumnMappings.Add("D2003_Schedule3", "D2003_Schedule3");
sbc.ColumnMappings.Add("D2004_ExemptCustomerFlag", "D2004_ExemptCustomerFlag");
sbc.ColumnMappings.Add("D2005_CustomerClassification", "D2005_CustomerClassification");
sbc.ColumnMappings.Add("D2006_29e", "D2006_29e");
sbc.ColumnMappings.Add("D2007_LargeVolAgreement", "D2007_LargeVolAgreement");
sbc.ColumnMappings.Add("D2008_SICCode", "D2008_SICCode");
sbc.ColumnMappings.Add("D2011_RateableValue", "D2011_RateableValue");
sbc.ColumnMappings.Add("D2015_SPIDVacant", "D2015_SPIDVacant");
sbc.ColumnMappings.Add("D2018_TroughsDrinkingBowls", "D2018_TroughsDrinkingBowls");
sbc.ColumnMappings.Add("D2019_WaterServicesToCaravans", "D2019_WaterServicesToCaravans");
sbc.ColumnMappings.Add("D2020_OutsideTaps", "D2020_OutsideTaps");
sbc.ColumnMappings.Add("D2022_TransitionalArrangements", "D2022_TransitionalArrangements");
sbc.ColumnMappings.Add("D2024_Unmeasurable", "D2024_Unmeasurable");
sbc.ColumnMappings.Add("D2014_FarmCroft", "D2014_FarmCroft");
// Finally write to server
System.Diagnostics.Debug.WriteLine("Write Bulk Copy to Server " + DateTime.Now.ToString());
sbc.WriteToServer(dataTable); // Fails here when I upload a 20MB CSV with 190,000 rows
sbc.Close();
}
// Ignore this I don't get to this code unless loading a file thats only got a few records
WaterBatch obj = GetWaterBatch(identity); // Now we can get the WaterBatch
obj.StopDateTime = DateTime.Now;
Edit(obj);
Save();
System.Diagnostics.Debug.WriteLine("Finished " + DateTime.Now.ToString());
}
catch (Exception ex)
{
Exception ex2 = ex;
while (ex2.InnerException != null)
{
ex2 = ex2.InnerException;
}
Console.WriteLine(ex.InnerException);
throw;
}
}
My $Exception says :
$exception {"A transport-level error has occurred when receiving results from the server. (provider: TCP Provider, error: 0 - An existing connection was forcibly closed by the remote host.)"} System.Exception {System.Data.SqlClient.SqlException}
My InnerException is, if I go into Inner then Inner exception etc its the same message with Hresult of -2146232060 then -2147467259:
InnerException {"An existing connection was forcibly closed by the remote host"} System.Exception {System.ComponentModel.Win32Exception}
UPDATED INFO :
Explanation of Error from Microsoft is (below). I am getting an Error number 40197. Then Microsoft say to look for the %d code - which I get to be 4815. Question is what now, where can I go from here to get into on a 40197 with a %d of 4815:
I got the following info regarding my error from this link: http://msdn.microsoft.com/en-us/library/windowsazure/ff394106.aspx
40197
17
The service has encountered an error processing your request. Please try again. Error code %d.
You will receive this error, when the service is down due to software or hardware upgrades, hardware failures, or any other failover problems. The error code (%d) embedded within the message of error 40197 provides additional information about the kind of failure or failover that occurred. Some examples of the error codes embedded within the message of error 40197 are 40020, 40143, 40166, and 40540.
Reconnecting to your SQL Database server will automatically connect you to a healthy copy of your database. Your application must catch error 40197, log the embedded error code (%d) within the message for troubleshooting, and try reconnecting to SQL Database until the resources are available, and your connection is established again.
I was getting the exact same error during a Bulk Insert. In my case, it was a varchar column that was overflowing. I just needed to increase the character limit and the problem was solved.
Just increase the Length of variable even if the value being stored is much lesser than than the size of the variable, worked for me.
We have a couple of mirrored SQL Server databases.
My first problem - the key problem - is to get a notification when the db fails over. I don't need to know because, erm, its mirrored and so it (almost) all carries on working automagically but it would useful to be advised and I'm currently getting failovers when I don't think I should be so it want to know when they occur (without too much digging) to see if I can determine why.
I have services running that I could fairly easily use to monitor this - so the alternative question would be "How do I programmatically determine which is the principal and which is the mirror" - preferably in a more intelligent fashion than just attempting to connect each in turn (which would mostly work but...).
Thanks, Murph
Addendum:
One of the answers queries why I don't need to know when it fails over - the answer is that we're developing using ADO.NET and that has automatic failover support, all you have to do is add Failover Partner=MIRRORSERVER (where MIRRORSERVER is the name of your mirror server instance) to your connection string and your code will fail over transparently - you may get some errors depending on what connections are active but in our case very few.
Right,
The two answers and a little thought got me to something approaching an answer.
First a little more clarification:
The app is written in C# (2.0+) and uses ADO.NET to talk to SQL Server 2005.
The mirror setup is two W2k3 servers hosting the Principal and the Mirror plus a third server hosting an express instance as a monitor. The nice thing about this is a failover is all but transparent to the app using the database, it will throw an error for some connections but fundamentally everything will carry on nicely. Yes we're getting the odd false positive but the whole point is to have the system carry on working with the least amount of fuss and mirror does deliver this very nicely.
Further, the issue is not with serious server failure - that's usually a bit more obvious but with a failover for other reasons (c.f. the false positives above) as we do have a couple of things that can't, for various reasons, fail over and in any case so we can see if we can identify the circumstance where we get false positives.
So, given the above, simply checking the status of the boxes is not quite enough and chasing through the event log is probably overly complex - the answer is, as it turns out, fairly simple: sp_helpserver
The first column returned by sp_helpserver is the server name. If you run the request at regular intervals saving the previous server name and doing a comparison each time you'll be able to identify when a change has taken place and then take the appropriate action.
The following is a console app that demonstrates the principal - although it needs some work (e.g. the connection ought to be non-pooled and new each time) but its enough for now (so I'd then accept this as "the" answer"). Parameters are Principal, Mirror, Database
using System;
using System.Data.SqlClient;
namespace FailoverMonitorConcept
{
class Program
{
static void Main(string[] args)
{
string server = args[0];
string failover = args[1];
string database = args[2];
string connStr = string.Format("Integrated Security=SSPI;Persist Security Info=True;Data Source={0};Failover Partner={1};Packet Size=4096;Initial Catalog={2}", server, failover, database);
string sql = "EXEC sp_helpserver";
SqlConnection dc = new SqlConnection(connStr);
SqlCommand cmd = new SqlCommand(sql, dc);
Console.WriteLine("Connection string: " + connStr);
Console.WriteLine("Press any key to test, press q to quit");
string priorServerName = "";
char key = ' ';
while(key.ToString().ToLower() != "q")
{
dc.Open();
try
{
string serverName = cmd.ExecuteScalar() as string;
Console.WriteLine(DateTime.Now.ToLongTimeString() + " - Server name: " + serverName);
if (priorServerName == "")
{
priorServerName = serverName;
}
else if (priorServerName != serverName)
{
Console.WriteLine("***** SERVER CHANGED *****");
Console.WriteLine("New server: " + serverName);
priorServerName = serverName;
}
}
catch (System.Data.SqlClient.SqlException ex)
{
Console.WriteLine("Error: " + ex.ToString());
}
finally
{
dc.Close();
}
key = Console.ReadKey(true).KeyChar;
}
Console.WriteLine("Finis!");
}
}
}
I wouldn't have arrived here without a) asking the question and then b) getting the responses which made me actually think
Murph
If the failover logic is in your application you could write a status screen that shows which box you're connected by writing to a var when the first connection attempt fails.
I think your best bet would be a ping daemon/cron job that checks the status of each box periodically and sends an email if one doesn't respond.
Use something like Host Monitor http://www.ks-soft.net/hostmon.eng/ to monitor the Event Log for messages related to the failover event, which can send you an alert via email/SMS.
I'm curious though how you wouldn't need to know that the failover happened, because don't you have to then update the datasources in your applications to point to the new server that you failed over to? Mirroring takes place on different hosts (the primary and the mirror), unlike clustering which has multiple nodes that appear to be a single device from the outside.
Also, are you using a witness server in order to automatically fail over from the primary to the mirror? This is the only way I know of to make it happen automatically, and in my experience, you get a lot of false-positives where network hiccups can fool the mirror and witness into thinking the primary is down when in fact it is not.