I've got a test that follows the following structure:
var testInstance aetest.Instance // initialized by a TestMain
TestThing(t *testing.T) {
defer cleanupGoogleDatastore(t, testInstance)
// insert basic test fixtures
// insert some new records here
// test assertions, etc.
}
The cleanupGoogleDatastore method just runs a datastore Query for all entities of a specific entity kind, and then deletes them one by one. Source here:
func cleanupGoogleDatastore(t *testing.T, testInstance aetest.Instance) {
q := datastore.NewQuery("Order")
ctx := GetContext(t, testInstance)
scanner := q.Run(ctx)
for {
var o model.Order
key, err := scanner.Next(&o)
if err == datastore.Done {
return
}
if err != nil {
t.Fatal(err.Error())
}
err = datastore.Delete(ctx, key)
if err != nil {
t.Fatal(err.Error())
}
}
}
The problem I'm having is that records inserted after the "basic test fixtures" aren't deleted by this deferred cleanup statement.
If I change this test function to instead look like the following:
var testInstance aetest.Instance // initialized by a TestMain
TestThing(t *testing.T) {
defer cleanupGoogleDatastore(t, testInstance)
// insert basic test fixtures
// insert some new records here
defer cleanupGoogleDatastore(t, testInstance)
// note that I have to call it TWICE - just moving it here is not enough.
// test assertions, etc.
}
then the newly created records are also deleted at the end of the test. My understanding was that deferred functions are just called at the end of the original function scope, which would imply that the query wouldn't be constructed and run until the end of the test, but that doesn't seem to be the case here. It seems like the query is being constructed when the defer statement is called and then executed at the end of the test.
I've tried moving cleanupGoogleDatastore into a closure (e.g. defer func() { cleanupGoogleDatastore(t, testInstance }() and that didn't change anything.
I suspect that somehow this is a case of the function arguments being evaluated at the point where I call defer instead of when the function is invoked, but since both t and testInstance are pointers rather than direct values I'm not sure how that could be happening. There's no other evidence that the two values have changed. Printing out the fields of both structs at the different logical evaluation points doesn't reveal anything new.
What's going on here?
This turned out to be a deliberate property of the local App Engine Datastore stub, which returns weakly consistent queries by default. I was able to resolve this by changing the code for cleanupGoogleDatastore to rely on an ancestor query, which successfully found and then deleted all of the entities. An alternative approach would have been to set aetest.Instance to be strongly consistent, but I didn't want to force strongly consistent behavior on all of my tests - just this one.
Related
As we know service (controller/usecase) layer is to handle business logic,
repo is to handle db queries
Right now I have:
func (s *OrderService) Create(order models.Order) (models.Order, error) {
...
user := models.User{
Contact: order.Address.Contact,
}
createdUser, err := s.UserRepo.Save(user)
// err handling...
order.User = user
createdOrder, err := s.OrderRepo.save(order)
// err handling...
return order, nil
}
// user_repo.go
func (repo *UserRepo) Save(user models.User) (models.User, error) {
err := repo.DB.Debug().Save(&user).Error
// err handing...
return user, nil
}
// order_repo.go
func (repo *OrderRepo) Save(order models.Order) (models.Order, error) {
err := repo.DB.Debug().Save(&order).Error
// err handing...
return order, nil
}
I want apply gorm db.Begin() transaction to become more flexible instead of my current code is too static.
So should I remove the gorm.DB repo but
i. passing in the gorm.DB through params??
tx := s.DB.Begin()
createdUser, err := s.UserRepo.Save(user, tx)
ii. or straight away running query in service layer?? (but it broke ddd design concept)
tx := s.DB.Begin()
createdUser, err := tx.Create(&user)
if err != nil {
tx.Rollback()
}
createdOrder, err := tx.Create(&order)
if err != nil {
tx.Rollback()
}
tx.Commit()
As per DDD, Transactions should not cross aggregate boundaries.
References:
https://martinfowler.com/bliki/DDD_Aggregate.html
Defining Aggregate Boundaries
If we have a need to update them in a transaction for some reason, you might wanna relook if they should be part of some Aggregate
While writing the repository for aggregate, you can neatly hide the transactions in the repository layer
I usually follow the following interface
// holds the business logic to modify the aggregate, provided by business layer
type AggregateUpdateFunction func (a *Aggregate) error
type Repository interface {
Create(ctx context.Context, aggregate *Aggregate)
Read(ctx context.Context, id string) *Aggregate
// starts a read-modify-write cycle internally in a transaction
Update(ctx context.Context, id string, updateFunc AggregateUpdateFunction) error
}
GORM calls should definitely stay abstracted away in storage layer. If you leak implementation details like transaction handle to business logic, storage layer will become tightly coupled with that particular implementation of storage.
In domain-driven world one should probably model interface of storage layer in such way that it has all operations business logic needs to operate with domain objects rather than basic operations underlying database offers (point is if you later switch from SQL database to S3 REST API the interface towards business logic will stay same). So instead (or on top) of OrderRepo.Save() I would also create OrderRepo.SaveAsNewUser() (Order, User, err) which will internally leverage database transaction.
I'm using https://github.com/DATA-DOG/go-sqlmock
and trying to mock a connection to a db.
Now, I need to mock a ping command (for load balancing purposes). However, I can't find any information about how to do that.
For example, I would like to write a test like this
db, mock, _ := sqlmock.New()
// ExpectPing does not exist, but it is there anything similar?
mock.ExpectPing().WillReturnError("mock error")
err := db.Ping()
if err==nil{
t.Fatal("there should have been an error")
}
in addition, I need to inject this mocked object into a function. let's say New(*sql.DB) that outputs a new structure. so it must be compatible with *sql.DB struct. for this reason, sqlmock seems a good choice. However, I've not found the way to mock the ping command.
it is there any way to do this using this library?
if not, is there any other database/sql mock library that could do the trick?
Update: As of Jan 6, 2020, this functionality has been added to go-sqlmock, and is included in the v1.4.0 release.
Original answer:
No, there is "nothing similar." The Ping and PingContext methods depend on the driver implementing the Pinger interface, so you can't mimic it by, for example, expecting a 'SELECT 1' or something.
There is already an issue requesting to add this. It seems to not have gotten much attention. I suspect creating a PR (which is probably about 3 lines of code) would greatly increase the chance of having that feature added.
If your goal is to make a Ping fail, you should be able to mimic that by connecting to an invalid DB endpoint (either with or without sqlmock).
You certainly can mock db itself:
type mockedDB struct {
*sql.DB
}
func (db *mockedDB) Ping() error {
return errors.New("not implemented")
}
func Example_mockedDB_Ping() {
db, _, _ := sqlmock.New()
defer db.Close()
mdb := mockedDB{db}
fmt.Println("mdb.Ping(): ", mdb.Ping())
// Output: mdb.Ping(): not implemented
}
but I don't understand what is the purpose of such experiment.
In the same way you can put mock into new type and define ExpectPing on it.
I needed to mock the Ping() functionality as well. This is how I solved it:
type mockDB struct {
ReturnError error
}
func (db *mockDB) Ping() error {
return db.ReturnError
}
func (db *mockDB) PingContext(ctx context.Context) error {
return db.ReturnError
}
// Pinger to be able to mock & ask just for a ping
type Pinger interface {
PingContext(ctx context.Context) error
Ping() error
}
// DatabasePingCheck returns a Check that validates connectivity to a database/sql.DB using Ping().
func DatabasePingCheck(pinger Pinger, timeout time.Duration) Check {
return func() error {
ctx, cancel := context.WithTimeout(context.Background(), timeout)
defer cancel()
if pinger == nil {
return fmt.Errorf("pinger is nil")
}
return pinger.PingContext(ctx)
}
}
If I set the namespace of my context.Context and call a delay function:
ctx := appengine.NewContext(r)
ctx, err := appengine.Namespace(ctx, "mynamespace")
delayFunc.Call(ctx)
How can I find its name:
var delayFunc = delay.Func("my-func", func(ctx context.Context) {
// How do I extract "mynamespace" from ctx?
})
Is the following acceptable practice?
var delayFunc = delay.Func("my-func", func(ctx context.Context) {
n := datastore.NewKey(ctx, "E", "e", 0, nil).Namespace()
// n == "mynamespace"
})
It works but feels like a hack.
Unfortunately you're out of luck. Appengine does not provide an (exported) API call to access the namespace associated with the context.
The namespace association with the context is handled by the appengine/internal package, but "programs should not use this package directly". The context with namespace is obtained by the internal.NamespacedContext() call, and the namespace "extraction" from context is implemented in internal.NamespaceFromContext(). These are not part of the public API, so you can't (shouldn't) use them.
You basically have 2 options. One is the "hacky" way you presented, which works and you may continue to use it.
Another one is to manually handle it, e.g. by manually putting the namespace into the context with your own key, e.g.:
var namespaceKey = "myNsKey"
ctx = context.WithValue(ctx, namespaceKey, "mynamespace")
And when you need it, you can get it like:
ns := ctx.Value(namespaceKey)
Yes, this has the burden of having to manually update it, and if you'd forgot, you would get an "invalid" or empty namespace. So personally I would go with your "hacky" way for now (until this functionality is added to the public API, if ever).
If you go with the manual way, to get rid of the "risk" factor, you could create a helper function which could take care of this along with the appengine.Namespace() call, so you would not forget about it and it would be safe. It could look like this:
func SetNS(ctx context.Context, ns string) context.Context {
ctx = ctx, err := appengine.Namespace(ctx, ns)
if err != nil {
// handle error
}
ctx = context.WithValue(ctx, namespaceKey, ns)
return ctx
}
And using it:
ctx = SetNS(ctx, "mynamespace")
But it may be a rare case when you need to access namespace from the context, as when you would need it, it might just be enough to pass the context to the proper (Appengine) API call which can extract it from the context.
I am currently trying to test a piece of my code that runs a query on the datastore before putting in a new entity to ensure that duplicates are not created. The code I wrote works fine in the context of the app, but the tests I wrote for that methods are failing. It seems that I cannot access data put into the datastore through queries in the context of the testing package.
One possibility might lie in the output from goapp test which reads: Applying all pending transactions and saving the datastore. This line prints out after both the get and put methods are called (I verified this with log statements).
I tried closing the context and creating a new one for the different operations, but unfortunately that didn't help either. Below is a simple test case that Puts in an object and then runs a query on it. Any help would be appreciated.
type Entity struct {
Value string
}
func TestEntityQuery(t *testing.T) {
c, err := aetest.NewContext(nil)
if err != nil {
t.Fatal(err)
}
defer c.Close()
key := datastore.NewIncompleteKey(c, "Entity", nil)
key, err = datastore.Put(c, key, &Entity{Value: "test"})
if err != nil {
t.Fatal(err)
}
q := datastore.NewQuery("Entity").Filter("Value =", "test")
var entities []Entity
keys, err := q.GetAll(c, &entities)
if err != nil {
t.Fatal(err)
}
if len(keys) == 0 {
t.Error("No keys found in query")
}
if len(entities) == 0 {
t.Error("No entities found in query")
}
}
There is nothing wrong with your test code. The issue lies in the Datastore itself. Most queries in the HR Datastore are not "immediately consistent" but eventually consistent. You can read more about this in the Datastore documentation.
So basically what happens is that you put an entity into the Datastore, and the SDK's Datastore "simulates" the latency that you can observe in production, so if you run a query right after that (which is not an ancestor query), the query result will not include the new entity you just saved.
If you put a few seconds sleep between the datastore.Put() and q.GetAll(), you will see the test passes. Try it. In my test it was enough to sleep just 100ms, and the test always passed. But when writing tests for such cases, use the StronglyConsistentDatastore: true option as can be seen in JonhGB's answer.
You would also see the test pass without sleep if you'd use Ancestor queries because they are strongly consistent.
The way to do this is to force the datastore to be strongly consistent by setting up the context like this:
c, err := aetest.NewContext(&aetest.Options{StronglyConsistentDatastore: true})
if err != nil {
t.Fatal(err)
}
Now the datastore won't need any sleep to work, which is faster, and better practice in general.
Update: This only works with the old aetest package which was imported via appengine/aetest. It does not work with the newer aetest package which is imported with google.golang.org/appengine/aetest. App Engine has changed from using an appengine.Context to using a context.Context, and consequently the way that the test package now works is quite different.
To compliment #JohnGB's answer in the latest version of aetest, there are more steps to get a context with strong consistency. First create an instance, then create a request from that instance, which you can use to produce a context.
inst, err := aetest.NewInstance(
&aetest.Options{StronglyConsistentDatastore: true})
if err != nil {
t.Fatal(err)
}
defer inst.Close()
req, err := inst.NewRequest("GET", "/", nil)
if err != nil {
t.Fatal(err)
}
ctx := appengine.NewContext(req)
I'm getting started with Google App Engine, and I'm using Objectify. How do I create a root entity in the data store, but err if it already exists? I didn't find anything built in for this (e.g. DatastoreService.put() and therefore ofy().save() will overwrite an existing entity instead of err). The simple technique I am used to is to do this in a transaction:
Err if already exists
Save
However, that is not idempotent; it would err in step 1 if the transaction executes twice. Here is the best I've come up with so far, not in a transaction:
Err if already exists
Save
Fetch
Err if it's not the data we just created
Or, if I don't mind two requests to save the same data both succeeding, I can skip the initial lookup:
Fetch
Report success if it's the same data we are about to create
Err if already exists, but is not the same data we are about to create
Save
That is doable, but it gets a little bulky to accomplish what I thought would be a very simple operation. Is there a better way?
This should guarantee consistent behavior:
final String id = // pick the unique id
final long txnId = // pick a uuid, timestamp, or even just a random number
ofy().transact(new VoidWork() {
public void vrun() {
Thing th = ofy().load().type(thing.class).id(id).now();
if (th != null) {
if (th.getTxnId() == txnId)
return;
else
throw ThingAlreadyExistsException();
}
th = createThing(id, txnId);
ofy().save().entity(th);
}
});