The database I am using is a global variable, initialized reading the file sql-repo.db:
const dbFile = "sql-repo.db"
var globalDB *LocalDB
type LocalDB struct {
Path string
handle *sql.DB
}
func InitSqlDB(dbDir string) error {
if globalDB != nil {
return nil
}
db := LocalDB{Path: filepath.Join(dbDir, dbFile)}
var err error
db.handle, err = sql.Open("sqlite3", db.Path)
if err != nil {
return err
}
globalDB = &db
return nil
}
From time to time, I will have an updated version of this database that I can download and store in dbDir.
Ideas that I have:
Use ATTACH DATABASE sql-repo.db AS dbMain to attach a copy of the first database and use it by default.
When I have my new .db file, I attach it as well ATTACH DATABASE sql-repo-new.db AS dbNew
Then I detach dbMain and rename dbNew to dbMain
Simply change the address my globalDB is pointing to:
const newDBFile = "sql-repo-new.db"
func PullNewDB(dbDir string) error {
db := LocalDB{Path: filepath.Join(dbDir, newDBFile)}
var err error
db.handle, err = sql.Open("sqlite3", db.Path)
if err != nil {
return err
}
globalDB = &db
return nil
}
How can I or how should I update my globalDB with the new version/file as I want to avoid any interference if the clients that I have in my code are connected to the DB and are querying it ?
Should I attach a sync.RWMutex to my LocalDB struct and then lock/unlock it when I do the update ?
Or should I use a channel to ask every client to stop querying the DB ?
Thank you for any help / advice / suggestion !
Or you can start a separate go routine or process that syncs your old database with the new file. Do an insert or update on all rows from new to old, then delete on missing rows. If it's all done in a single transaction, all queries will either read all old or new data without ever blocking.
An additional benefit is the separation of concerns, your application code doesn't get clustered with update logic, and in the case, the new file is corrupted, the update transaction errors out and no harm is done.
Related
As we know service (controller/usecase) layer is to handle business logic,
repo is to handle db queries
Right now I have:
func (s *OrderService) Create(order models.Order) (models.Order, error) {
...
user := models.User{
Contact: order.Address.Contact,
}
createdUser, err := s.UserRepo.Save(user)
// err handling...
order.User = user
createdOrder, err := s.OrderRepo.save(order)
// err handling...
return order, nil
}
// user_repo.go
func (repo *UserRepo) Save(user models.User) (models.User, error) {
err := repo.DB.Debug().Save(&user).Error
// err handing...
return user, nil
}
// order_repo.go
func (repo *OrderRepo) Save(order models.Order) (models.Order, error) {
err := repo.DB.Debug().Save(&order).Error
// err handing...
return order, nil
}
I want apply gorm db.Begin() transaction to become more flexible instead of my current code is too static.
So should I remove the gorm.DB repo but
i. passing in the gorm.DB through params??
tx := s.DB.Begin()
createdUser, err := s.UserRepo.Save(user, tx)
ii. or straight away running query in service layer?? (but it broke ddd design concept)
tx := s.DB.Begin()
createdUser, err := tx.Create(&user)
if err != nil {
tx.Rollback()
}
createdOrder, err := tx.Create(&order)
if err != nil {
tx.Rollback()
}
tx.Commit()
As per DDD, Transactions should not cross aggregate boundaries.
References:
https://martinfowler.com/bliki/DDD_Aggregate.html
Defining Aggregate Boundaries
If we have a need to update them in a transaction for some reason, you might wanna relook if they should be part of some Aggregate
While writing the repository for aggregate, you can neatly hide the transactions in the repository layer
I usually follow the following interface
// holds the business logic to modify the aggregate, provided by business layer
type AggregateUpdateFunction func (a *Aggregate) error
type Repository interface {
Create(ctx context.Context, aggregate *Aggregate)
Read(ctx context.Context, id string) *Aggregate
// starts a read-modify-write cycle internally in a transaction
Update(ctx context.Context, id string, updateFunc AggregateUpdateFunction) error
}
GORM calls should definitely stay abstracted away in storage layer. If you leak implementation details like transaction handle to business logic, storage layer will become tightly coupled with that particular implementation of storage.
In domain-driven world one should probably model interface of storage layer in such way that it has all operations business logic needs to operate with domain objects rather than basic operations underlying database offers (point is if you later switch from SQL database to S3 REST API the interface towards business logic will stay same). So instead (or on top) of OrderRepo.Save() I would also create OrderRepo.SaveAsNewUser() (Order, User, err) which will internally leverage database transaction.
I'm using this driver: https://github.com/denisenkom/go-mssqldb and on production with an Azure SQL Database Standard S3 level we are getting way too much ErrBadconn - driver: Bad connection returned.
How can I prevent or at least gracefully handle that. Here's some code to show how things are setup.
A typical database function call
package dal
var db *sql.DB
type Database struct{}
func (d Database) Open() {
newDB, err := sql.Open("mssql", os.Getenv("dbconnestion"))
if err != nil {
panic(err)
}
err = newDB.Ping()
if err != nil {
panic(err)
}
db = newDB
}
func (d Database) Close() {
db.Close()
}
// ... in another file
func (e *Entities) Add(entity Entity) (int64, error) {
stmt, err := db.Prepare("INSERT INTO Entities VALUES(?, ?)")
if err != nil {
return -1, err
}
defer stmt.Close()
result, err := stmt.Exec(entity.Field1, entity.Field2)
if err != nil {
return -1, err
}
return result.LastInsertId()
}
On a web api
func main() {
db := dal.Database{}
db.Open()
defer db.Close()
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
entities := &dal.Entites{}
id, err := entities.Add(dal.Entity{Field1: "a", Field2: "b"})
if err != nil {
// here all across my web api and other web package or cli cmd that uses the dal I'm getting random ErrBadConn
}
})
}
So in short, the dal package is shared across multiple Azure web apps and command line Go apps.
I cannot see a pattern, those errors, which are frequent and randomly occurring. We are using Bugsnag to log the errors from all our apps.
For completion, sometimes our Standard S3 limit of 200 concurrent connections is reached.
I've triple checked everywhere on the package that access the database, making sure that all sql.Rows were closed, all db.Prepare statement are closed. As and example here's how a typical query function looks like:
func (e *Entities) GetByID(id int64) ([]Entity, error) {
rows, err := db.Query("SELECT * FROM Entities WHERE ID = ?", id)
if err != nil {
return nil, err
}
defer rows.Close()
var results []Entity
for rows.Next() {
var r Entity
err := readEntity(rows, &r)
if err != nil {
return nil, err
}
results = append(results, r)
}
if err = rows.Err(); err != nil {
return nil, err
}
return results, nil
}
The readEntity is basically only doing Scan on the fields.
I don't think it's code related, unit tests run well locally. It's just once deployed to Azure after running for sometimes, the driver: Bad connection start to show up very frequently.
I've ran this query to try and see as suggested in this question: Azure SQL server max pool size was reached error
select * from sys.dm_exeC_requests
But I'm not exactly sure what should I be paying attention here.
Things I've did / made sure of.
As it's suggested, the database/sql should handle the connection pool, so having a global variable for the database connection should be fine.
Making sure sql.Rows and db.Prepare statement are closed everywhere.
Increased the Azure SQL level to S3.
There's an issue for the sql driver I'm using talking about Azure SQL making database connection is a bad state if they are idling for more thant 2 minutes.
https://github.com/denisenkom/go-mssqldb/issues/81
Does the way database/sql handle the connection pooling is in any way not working with the way Azure SQL Database are manage.
Is there a way to gracefully handle this? I know that C# / Entity Framework have a connection resiliency / retry logic for Azure SQL, is it for the similar reasons? How could I implement this without having to pass everywhere on my error handling? I mean I don't want to do something like this clearly:
if err == sql.ErrBadConn {
// close and re-open the global db object
// retry
}
This is certainly not my only option here?
Any help would be extremely welcome.
Thank you
I'm not seeing anywhere that you close your database. Best practice (in other languages - not positive about Go) is to close / deallocate / dereference the database object after use, to release the connection back into the pool. If you're running out of connection resources, you're being told that you need to release things. Holding the reference open means nobody else can use that connection, so it'll stay around until it gets recycled because it's timed out. This is why you're getting it intermittently, rather than consistently - it depends on having a certain number of connections taking place w/in a certain period of time.
Pierre,
How often do you run into these connection issues? I recommend you build retry logic to graciously fail from bad connections. Here is how you would do it with C#: https://azure.microsoft.com/en-us/documentation/articles/sql-database-develop-csharp-retry-windows/
If you still feel you need assistance, feel free to shoot me an email with your server name and database name at meetb#microsoft.com and we will get our team to look into this issue.
Cheers,
Meet
I am currently trying to test a piece of my code that runs a query on the datastore before putting in a new entity to ensure that duplicates are not created. The code I wrote works fine in the context of the app, but the tests I wrote for that methods are failing. It seems that I cannot access data put into the datastore through queries in the context of the testing package.
One possibility might lie in the output from goapp test which reads: Applying all pending transactions and saving the datastore. This line prints out after both the get and put methods are called (I verified this with log statements).
I tried closing the context and creating a new one for the different operations, but unfortunately that didn't help either. Below is a simple test case that Puts in an object and then runs a query on it. Any help would be appreciated.
type Entity struct {
Value string
}
func TestEntityQuery(t *testing.T) {
c, err := aetest.NewContext(nil)
if err != nil {
t.Fatal(err)
}
defer c.Close()
key := datastore.NewIncompleteKey(c, "Entity", nil)
key, err = datastore.Put(c, key, &Entity{Value: "test"})
if err != nil {
t.Fatal(err)
}
q := datastore.NewQuery("Entity").Filter("Value =", "test")
var entities []Entity
keys, err := q.GetAll(c, &entities)
if err != nil {
t.Fatal(err)
}
if len(keys) == 0 {
t.Error("No keys found in query")
}
if len(entities) == 0 {
t.Error("No entities found in query")
}
}
There is nothing wrong with your test code. The issue lies in the Datastore itself. Most queries in the HR Datastore are not "immediately consistent" but eventually consistent. You can read more about this in the Datastore documentation.
So basically what happens is that you put an entity into the Datastore, and the SDK's Datastore "simulates" the latency that you can observe in production, so if you run a query right after that (which is not an ancestor query), the query result will not include the new entity you just saved.
If you put a few seconds sleep between the datastore.Put() and q.GetAll(), you will see the test passes. Try it. In my test it was enough to sleep just 100ms, and the test always passed. But when writing tests for such cases, use the StronglyConsistentDatastore: true option as can be seen in JonhGB's answer.
You would also see the test pass without sleep if you'd use Ancestor queries because they are strongly consistent.
The way to do this is to force the datastore to be strongly consistent by setting up the context like this:
c, err := aetest.NewContext(&aetest.Options{StronglyConsistentDatastore: true})
if err != nil {
t.Fatal(err)
}
Now the datastore won't need any sleep to work, which is faster, and better practice in general.
Update: This only works with the old aetest package which was imported via appengine/aetest. It does not work with the newer aetest package which is imported with google.golang.org/appengine/aetest. App Engine has changed from using an appengine.Context to using a context.Context, and consequently the way that the test package now works is quite different.
To compliment #JohnGB's answer in the latest version of aetest, there are more steps to get a context with strong consistency. First create an instance, then create a request from that instance, which you can use to produce a context.
inst, err := aetest.NewInstance(
&aetest.Options{StronglyConsistentDatastore: true})
if err != nil {
t.Fatal(err)
}
defer inst.Close()
req, err := inst.NewRequest("GET", "/", nil)
if err != nil {
t.Fatal(err)
}
ctx := appengine.NewContext(req)
Im using Google Cloud Storage Go Runtime client API to write a string from GO server into a file in the default bucket . Here is the code
wc := storage.NewWriter(d.ctx, bucket, fileName)
wc.ContentType = "text/plain"
if _, err := wc.Write([]byte(myData)); err != nil {
d.errorf("createFile: unable to write data to bucket %q, file %q: %v", bucket, fileName, err)
return
}
But im Not able set right ACL permissions on the wc object to make the file public ?? How to do achieve it so that the created file becomes public ?
Try setting:
wc.ACL = []storage.ACLRule{{storage.AllUsers, storage.RoleReader}}
Alternately, you can also change the default object ACL on the bucket so that newly created objects default to being publicly readable.
You can do the same by this also.
acl := client.Bucket(bucket).Object(object).ACL()
if err := acl.Set(ctx, storage.AllUsers, storage.RoleReader); err != nil {
return err
}
Here is the link to the docs.
I'm getting started with Google App Engine, and I'm using Objectify. How do I create a root entity in the data store, but err if it already exists? I didn't find anything built in for this (e.g. DatastoreService.put() and therefore ofy().save() will overwrite an existing entity instead of err). The simple technique I am used to is to do this in a transaction:
Err if already exists
Save
However, that is not idempotent; it would err in step 1 if the transaction executes twice. Here is the best I've come up with so far, not in a transaction:
Err if already exists
Save
Fetch
Err if it's not the data we just created
Or, if I don't mind two requests to save the same data both succeeding, I can skip the initial lookup:
Fetch
Report success if it's the same data we are about to create
Err if already exists, but is not the same data we are about to create
Save
That is doable, but it gets a little bulky to accomplish what I thought would be a very simple operation. Is there a better way?
This should guarantee consistent behavior:
final String id = // pick the unique id
final long txnId = // pick a uuid, timestamp, or even just a random number
ofy().transact(new VoidWork() {
public void vrun() {
Thing th = ofy().load().type(thing.class).id(id).now();
if (th != null) {
if (th.getTxnId() == txnId)
return;
else
throw ThingAlreadyExistsException();
}
th = createThing(id, txnId);
ofy().save().entity(th);
}
});