I am trying to use Redis with lambda functions. How can I use the connection pooling in the lamdba if it does not maintain state. Is it ok to always have the lambda make new connections to redis? Or can I use the connection pool example I pasted below? Where in my Handle func I will do conn := pool.Get(). Not sure what approach I should take. Any help.
func Handle(ctx context.context, req events.APIGatewayWebsocketProxyRequest)(interface{},error){
//make new redis connection
}
or
func newPool(addr string) *redis.Pool {
return &redis.Pool{
MaxIdle: 3,
IdleTimeout: 240 * time.Second,
// Dial or DialContext must be set. When both are set, DialContext takes precedence over Dial.
Dial: func () (redis.Conn, error) { return redis.Dial("tcp", addr) },
}
}
var (
pool *redis.Pool
redisServer = flag.String("redisServer", ":6379", "")
)
func main() {
flag.Parse()
pool = newPool(*redisServer)
...
}
Although not the perfect solution (like e.g. AWS RDS Proxy), creating the connection pool outside the lambda's handler seems to be your best bet. The actual technique depends on your language/environment as you can also lazily initialize your connection pool and save it to a class variable, singleton, etc. In your example, use a global pool variable from anywhere in the lambda handler function.
var (
pool *redis.Pool
redisServer = flag.String("redisServer", ":6379", "")
)
func newPool(addr string) *redis.Pool {
...
}
func HandleRequest(ctx context.Context, name MyEvent) (string, error) {
conn := pool.Get()
defer conn.Close()
err = conn.Send("ZINCRBY", ...)
if err != nil {
return err
}
...
}
func main() {
flag.Parse()
pool = newPool(*redisServer)
lambda.Start(HandleRequest)
}
Check this Redislabs article for some more info.
AWS lambda's processing model is a rather simple one. As long as the requests are coming (lambda is being called from API gateway, Kinesis stream, etc.), lambda's lifecycle management code will keep the lambda alive as it synchronously loops over the batches of requests. When this single-threaded solution can't keep up with the incoming processing requests (by constantly checking on some latency metrics), lambda's lifecycle manager will create more lambdas. So, in a sense, when there's a lot to process, you will have your lambda(s) up for some time, and that's when you want to be conservative with your server's free connections anyhow.
Related
I'm using https://github.com/DATA-DOG/go-sqlmock
and trying to mock a connection to a db.
Now, I need to mock a ping command (for load balancing purposes). However, I can't find any information about how to do that.
For example, I would like to write a test like this
db, mock, _ := sqlmock.New()
// ExpectPing does not exist, but it is there anything similar?
mock.ExpectPing().WillReturnError("mock error")
err := db.Ping()
if err==nil{
t.Fatal("there should have been an error")
}
in addition, I need to inject this mocked object into a function. let's say New(*sql.DB) that outputs a new structure. so it must be compatible with *sql.DB struct. for this reason, sqlmock seems a good choice. However, I've not found the way to mock the ping command.
it is there any way to do this using this library?
if not, is there any other database/sql mock library that could do the trick?
Update: As of Jan 6, 2020, this functionality has been added to go-sqlmock, and is included in the v1.4.0 release.
Original answer:
No, there is "nothing similar." The Ping and PingContext methods depend on the driver implementing the Pinger interface, so you can't mimic it by, for example, expecting a 'SELECT 1' or something.
There is already an issue requesting to add this. It seems to not have gotten much attention. I suspect creating a PR (which is probably about 3 lines of code) would greatly increase the chance of having that feature added.
If your goal is to make a Ping fail, you should be able to mimic that by connecting to an invalid DB endpoint (either with or without sqlmock).
You certainly can mock db itself:
type mockedDB struct {
*sql.DB
}
func (db *mockedDB) Ping() error {
return errors.New("not implemented")
}
func Example_mockedDB_Ping() {
db, _, _ := sqlmock.New()
defer db.Close()
mdb := mockedDB{db}
fmt.Println("mdb.Ping(): ", mdb.Ping())
// Output: mdb.Ping(): not implemented
}
but I don't understand what is the purpose of such experiment.
In the same way you can put mock into new type and define ExpectPing on it.
I needed to mock the Ping() functionality as well. This is how I solved it:
type mockDB struct {
ReturnError error
}
func (db *mockDB) Ping() error {
return db.ReturnError
}
func (db *mockDB) PingContext(ctx context.Context) error {
return db.ReturnError
}
// Pinger to be able to mock & ask just for a ping
type Pinger interface {
PingContext(ctx context.Context) error
Ping() error
}
// DatabasePingCheck returns a Check that validates connectivity to a database/sql.DB using Ping().
func DatabasePingCheck(pinger Pinger, timeout time.Duration) Check {
return func() error {
ctx, cancel := context.WithTimeout(context.Background(), timeout)
defer cancel()
if pinger == nil {
return fmt.Errorf("pinger is nil")
}
return pinger.PingContext(ctx)
}
}
In my Rust application I am using Diesel to interact with an SQLite database. I have multiple threads that may query at the same time the database, and I am using the crate r2d2 to create a pool of connections.
The issue that I am seeing is that I am not able to concurrently query the database. If I try to do that, I always get the error database is locked, which is unrecoverable (any following request will fail from the same error even if only a single thread is querying).
The following code reproduces the issue.
# Cargo.toml
[dependencies]
crossbeam = { version = "0.7.1" }
diesel = { version = "1.4.2", features = ["sqlite", "r2d2"] }
-- The database table
CREATE TABLE users (
name TEXT PRIMARY KEY NOT NULL
);
#[macro_use]
extern crate diesel;
mod schema;
use crate::schema::*;
use crossbeam;
use diesel::r2d2::{ConnectionManager, Pool};
use diesel::RunQueryDsl;
use diesel::{ExpressionMethods, SqliteConnection};
#[derive(Insertable, Queryable, Debug, Clone)]
#[table_name = "users"]
struct User {
name: String,
}
fn main() {
let db_url = "test.sqlite3";
let pool = Pool::builder()
.build(ConnectionManager::<SqliteConnection>::new(db_url))
.unwrap();
crossbeam::scope(|scope| {
let pool2 = pool.clone();
scope.spawn(move |_| {
let conn = pool2.get().unwrap();
for i in 0..100 {
let name = format!("John{}", i);
diesel::delete(users::table)
.filter(users::name.eq(&name))
.execute(&conn)
.unwrap();
}
});
let conn = pool.get().unwrap();
for i in 0..100 {
let name = format!("John{}", i);
diesel::insert_into(users::table)
.values(User { name })
.execute(&conn)
.unwrap();
}
})
.unwrap();
}
This is the error as shown when the application panics:
thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: DatabaseError(__Unknown, "database is locked")'
AFAIK, I should be able to use the connection pool with multiple threads (that is, multiple connections for multiple threads), as shown in the r2d2_sqlite crate example.
Moreover, the sqlite3 library I have installed in my system supports the Serialized threading model, which from here:
In serialized mode, SQLite can be safely used by multiple threads with
no restriction.
How can I avoid the database is locked errors? Also, if these errors are not avoidable for any reason, how can I unlock the database?
Recently I also stumbled onto this problem. Here's what I found.
SQLite does not support multiple writers.
From documentation:
When SQLite tries to access a file that is locked by another process, the default behavior is to return SQLITE_BUSY.
So how to get around this limitation ? There are two solutions I see.
Busy timeout
You can retry the query multiple times until lock has been acquired.
In fact SQLite provides built-in mechanism.
You can instruct the SQLite to try lock the database multiple times.
Now the only thing you need is to somehow pass this pragma to SQLite.
Fortunately diesel::r2d2 gives an easy way to pass initial setup for a newly established connection:
#[derive(Debug)]
pub struct ConnectionOptions {
pub enable_wal: bool,
pub enable_foreign_keys: bool,
pub busy_timeout: Option<Duration>,
}
impl diesel::r2d2::CustomizeConnection<SqliteConnection, diesel::r2d2::Error>
for ConnectionOptions
{
fn on_acquire(&self, conn: &mut SqliteConnection) -> Result<(), diesel::r2d2::Error> {
(|| {
if self.enable_wal {
conn.batch_execute("PRAGMA journal_mode = WAL; PRAGMA synchronous = NORMAL;")?;
}
if self.enable_foreign_keys {
conn.batch_execute("PRAGMA foreign_keys = ON;")?;
}
if let Some(d) = self.busy_timeout {
conn.batch_execute(&format!("PRAGMA busy_timeout = {};", d.as_millis()))?;
}
Ok(())
})()
.map_err(diesel::r2d2::Error::QueryError)
}
}
// ------------- Example -----------------
let pool = Pool::builder()
.max_size(16)
.connection_customizer(Box::new(ConnectionOptions {
enable_wal: true,
enable_foreign_keys: true,
busy_timeout: Some(Duration::from_secs(30)),
}))
.build(ConnectionManager::<SqliteConnection>::new(db_url))
.unwrap();
WAL mode
The second variant you might want to use is WAL mode. It improves concurrency by letting readers and writer to work at the same time (WAL mode is waaay faster than default journal mode).
Note however that busy timeout is still required for all of this to work.
(Please, read also about consequences of "synchronous" mode set to "NORMAL".)
SQLITE_BUSY_SNAPSHOT is the next thing that may occur with WAL mode. But there is easy remedy to that - use BEGIN IMMEDIATE to start transaction in write mode.
This way you can have multiple readers/writers which makes life easier. Multiple writers use locking mechanism (through busy_timeout), so there is one active writer at the time. You certainly don't want to qualify connections as read and write and do locking manually in your application, e.g. with Mutex.
I find setting r2d2::Pool::builder().max_size(1) solves the issue, but you need to be careful with your connection management and don't ask if you already got one, for example:
fn create(pool: &DbPool, data: User) {
let conn = pool.get().unwrap(); // One connection
if !exist(pool, data) { // Two connection
diesel::insert_into(users::table)
.values(User { name: data.name })
.execute(&conn)
.unwrap();
}
}
fn exist(pool: &DbPool, data: User) -> bool {
let conn = pool.get().unwrap();
Ok(
select(exists(users::table.filter(col_user_name.eq(data.name))))
.get_result(&conn)
.unwrap(),
)
}
I removed all let conn = pool.get().unwrap(); and only try to get it inside the request.
diesel::insert_into(users::table)
.values(User { name })
.execute(&pool.get().unwrap())
.unwrap();
If I set the namespace of my context.Context and call a delay function:
ctx := appengine.NewContext(r)
ctx, err := appengine.Namespace(ctx, "mynamespace")
delayFunc.Call(ctx)
How can I find its name:
var delayFunc = delay.Func("my-func", func(ctx context.Context) {
// How do I extract "mynamespace" from ctx?
})
Is the following acceptable practice?
var delayFunc = delay.Func("my-func", func(ctx context.Context) {
n := datastore.NewKey(ctx, "E", "e", 0, nil).Namespace()
// n == "mynamespace"
})
It works but feels like a hack.
Unfortunately you're out of luck. Appengine does not provide an (exported) API call to access the namespace associated with the context.
The namespace association with the context is handled by the appengine/internal package, but "programs should not use this package directly". The context with namespace is obtained by the internal.NamespacedContext() call, and the namespace "extraction" from context is implemented in internal.NamespaceFromContext(). These are not part of the public API, so you can't (shouldn't) use them.
You basically have 2 options. One is the "hacky" way you presented, which works and you may continue to use it.
Another one is to manually handle it, e.g. by manually putting the namespace into the context with your own key, e.g.:
var namespaceKey = "myNsKey"
ctx = context.WithValue(ctx, namespaceKey, "mynamespace")
And when you need it, you can get it like:
ns := ctx.Value(namespaceKey)
Yes, this has the burden of having to manually update it, and if you'd forgot, you would get an "invalid" or empty namespace. So personally I would go with your "hacky" way for now (until this functionality is added to the public API, if ever).
If you go with the manual way, to get rid of the "risk" factor, you could create a helper function which could take care of this along with the appengine.Namespace() call, so you would not forget about it and it would be safe. It could look like this:
func SetNS(ctx context.Context, ns string) context.Context {
ctx = ctx, err := appengine.Namespace(ctx, ns)
if err != nil {
// handle error
}
ctx = context.WithValue(ctx, namespaceKey, ns)
return ctx
}
And using it:
ctx = SetNS(ctx, "mynamespace")
But it may be a rare case when you need to access namespace from the context, as when you would need it, it might just be enough to pass the context to the proper (Appengine) API call which can extract it from the context.
I'm using this driver: https://github.com/denisenkom/go-mssqldb and on production with an Azure SQL Database Standard S3 level we are getting way too much ErrBadconn - driver: Bad connection returned.
How can I prevent or at least gracefully handle that. Here's some code to show how things are setup.
A typical database function call
package dal
var db *sql.DB
type Database struct{}
func (d Database) Open() {
newDB, err := sql.Open("mssql", os.Getenv("dbconnestion"))
if err != nil {
panic(err)
}
err = newDB.Ping()
if err != nil {
panic(err)
}
db = newDB
}
func (d Database) Close() {
db.Close()
}
// ... in another file
func (e *Entities) Add(entity Entity) (int64, error) {
stmt, err := db.Prepare("INSERT INTO Entities VALUES(?, ?)")
if err != nil {
return -1, err
}
defer stmt.Close()
result, err := stmt.Exec(entity.Field1, entity.Field2)
if err != nil {
return -1, err
}
return result.LastInsertId()
}
On a web api
func main() {
db := dal.Database{}
db.Open()
defer db.Close()
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
entities := &dal.Entites{}
id, err := entities.Add(dal.Entity{Field1: "a", Field2: "b"})
if err != nil {
// here all across my web api and other web package or cli cmd that uses the dal I'm getting random ErrBadConn
}
})
}
So in short, the dal package is shared across multiple Azure web apps and command line Go apps.
I cannot see a pattern, those errors, which are frequent and randomly occurring. We are using Bugsnag to log the errors from all our apps.
For completion, sometimes our Standard S3 limit of 200 concurrent connections is reached.
I've triple checked everywhere on the package that access the database, making sure that all sql.Rows were closed, all db.Prepare statement are closed. As and example here's how a typical query function looks like:
func (e *Entities) GetByID(id int64) ([]Entity, error) {
rows, err := db.Query("SELECT * FROM Entities WHERE ID = ?", id)
if err != nil {
return nil, err
}
defer rows.Close()
var results []Entity
for rows.Next() {
var r Entity
err := readEntity(rows, &r)
if err != nil {
return nil, err
}
results = append(results, r)
}
if err = rows.Err(); err != nil {
return nil, err
}
return results, nil
}
The readEntity is basically only doing Scan on the fields.
I don't think it's code related, unit tests run well locally. It's just once deployed to Azure after running for sometimes, the driver: Bad connection start to show up very frequently.
I've ran this query to try and see as suggested in this question: Azure SQL server max pool size was reached error
select * from sys.dm_exeC_requests
But I'm not exactly sure what should I be paying attention here.
Things I've did / made sure of.
As it's suggested, the database/sql should handle the connection pool, so having a global variable for the database connection should be fine.
Making sure sql.Rows and db.Prepare statement are closed everywhere.
Increased the Azure SQL level to S3.
There's an issue for the sql driver I'm using talking about Azure SQL making database connection is a bad state if they are idling for more thant 2 minutes.
https://github.com/denisenkom/go-mssqldb/issues/81
Does the way database/sql handle the connection pooling is in any way not working with the way Azure SQL Database are manage.
Is there a way to gracefully handle this? I know that C# / Entity Framework have a connection resiliency / retry logic for Azure SQL, is it for the similar reasons? How could I implement this without having to pass everywhere on my error handling? I mean I don't want to do something like this clearly:
if err == sql.ErrBadConn {
// close and re-open the global db object
// retry
}
This is certainly not my only option here?
Any help would be extremely welcome.
Thank you
I'm not seeing anywhere that you close your database. Best practice (in other languages - not positive about Go) is to close / deallocate / dereference the database object after use, to release the connection back into the pool. If you're running out of connection resources, you're being told that you need to release things. Holding the reference open means nobody else can use that connection, so it'll stay around until it gets recycled because it's timed out. This is why you're getting it intermittently, rather than consistently - it depends on having a certain number of connections taking place w/in a certain period of time.
Pierre,
How often do you run into these connection issues? I recommend you build retry logic to graciously fail from bad connections. Here is how you would do it with C#: https://azure.microsoft.com/en-us/documentation/articles/sql-database-develop-csharp-retry-windows/
If you still feel you need assistance, feel free to shoot me an email with your server name and database name at meetb#microsoft.com and we will get our team to look into this issue.
Cheers,
Meet
The following example seems to work, but is it safe to use? My goal is to do some very light background processing (whereas an actual task queue job feels too heavy).
func MyHandler(w http.ResponseWriter, r *http.Request) {
go func() {
// do something ...
}()
return // 200
}
Goroutines that outlive the request are not supported, but you can use runtime.RunInBackground to execute code in a background goroutine:
func MyHandler(w http.ResponseWriter, r *http.Request) {
err := runtime.RunInBackground(c, func(c appengine.Context) {
// do something...
})
return // 200
}
The provided function will be invoked with a background context that is distinct from (and may outlast) the provided context. Note that there is a limit of 10 simultaneous background requests per instance. Here is another example.
Please note that Goroutines that live within the context of a request, are supported though:
The Go runtime environment for App Engine provides full support for
goroutines, but not for parallel execution: goroutines are scheduled
onto a single operating system thread. This single-thread restriction
may be lifted in future versions. Multiple requests may be handled
concurrently by a given instance; that means that if one request is,
say, waiting for a datastore API call, another request may be
processed by the same instance. (Source)