Is there a Go analogue to Python/Java's async datastore APIs? Or can one just use the normal API with the go keyword?
There is no Go equivalent to the Python or Java asynchronous APIs for any AppEngine service. In fact, the Go standard library has nothing in the standard asynchronous style either. The reason is that in Go, you write functions using a blocking style and compose them using some basic concurrency primitives based on need. While you cannot just tack go at the beginning of a dastore.Get call, it is still relatively straightforward. Consider the following, contrived example:
func loadUser(ctx appengine.Context, name strings) (*User, err) {
var u User
var entries []*Entry
done := make(chan error)
go func() {
// Load the main features of the User
key := datastore.NewKey(ctx, "user", name, 0, nil)
done <- datastore.Get(ctx, key)
}()
go func() {
// Load the entries associated with the user
q := datastore.NewQuery("entries").Filter("user", name)
keys, err := q.GetAll(ctx, &entries)
for i, k := range keys {
entries[i].key = k
}
done <- err
}()
success := true
// Wait for the queries to finish in parallel
for i := 0; i < 2 /* count the funcs above */; i++ {
if err := <-done; err != nil {
ctx.Errorf("loaduser: %s", err)
success = false
}
}
if !success {
return
}
// maybe more stuff here
}
This same approach can be used in pretty much any context in which you need to run more than one thing that might take awhile at the same time, whether it's a datastore call, urlfetch, file load, etc.
There is no explicit API for async in Go. You should use go routines instead. I haven't seen any source on this, but I suspect the async API isn't there because of how easy it is to use go routines.
Related
I want to create a database driven application using Golang. I am trying to do it TDD way.
When I try to test methods that make Sql queries, What all are the packages available ?
I don't want to connect to the default database that I use for development. I can write code to take up another test database while running a test, but is there any go library that already does it.
Is there any library that does db tests without connecting to database at all ?
What is the standard way to do database test with golang ?
I had a similar question not long ago when refactoring some of my own tests, and there's a couple of ways you can do it:
a) Provide an exported type and an Open or Connect function that returns it - e.g.
type DB struct {
db *sql.DB
}
// Using http://jmoiron.github.io/sqlx/ for this example, but
// it has the same interface as database/sql
func Open(opts *Options) (*DB, error) {
db, err := sqlx.Connect(opts.Driver, fmt.Sprintf("host=%s user=%s dbname=%s sslmode=%s", opts.Host, opts.User, opts.Name, opts.SSL))
if err != nil {
return nil, err
}
return &DB{db}, nil
}
... and then each of your tests, write setup & teardown functions that return an instance of *DB that you define your database functions on (as methods - i.e. func (db *DB) GetUser(user *User) (bool, error)):
// Setup the test environment.
func setup() (*DB, error) {
err := withTestDB()
if err != nil {
return nil, err
}
// testOptions is a global in this case, but you could easily
// create one per-test
db, err := Open(testOptions)
if err != nil {
return nil, err
}
// Loads our test schema
db.MustLoad()
return db, nil
}
// Create our test database.
func withTestDB() error {
db, err := open()
if err != nil {
return err
}
defer db.Close()
_, err = db.Exec(fmt.Sprintf("CREATE DATABASE %s;", testOptions.Name))
if err != nil {
return err
}
return nil
}
Note that this is somewhat "integration" testing, but I strongly prefer to test against a "real" database since mocking the interface won't help you catch issues with your queries/query syntax.
b) The alternative, although less extensible on the application side, is to have a global db *sql.DB variable that you initialise in init() within your tests—since tests have no guaranteed order you'll need to use init()—and then run your tests from there. i.e.
var db *sql.DB
func init() {
var err error
// Note the = and *not* the assignment - we don't want to shadow our global
db, err = sqlx.Connect(...)
if err != nil {
...
}
err := db.loadTestSchema
// etc.
}
func TestGetUser(t *testing.T) {
user := User{}
exists, err := db.GetUser(user)
...
}
You can find some practical examples in drone.io's GitHub repo, and I'd also recommend this article on structuring Go applications (especially the DB stuff).
I use a global variable to store the data source (or connection string) of current database and set to different value in test function. Since there is only one database I need to operate so I choose the easiest way.
I'm trying to push the message to google pub-sub asynchronously through goroutine but I'm facing below error
panic: not an App Engine context
I'm using mux and have an api handler
n = 1 million
func apihandler(w http.ResponseWriter, r *http.Request) {
go createuniquecodes(n)
return "request running in background"
}
func createuniquecodes(n) {
c := make(chan string)
go createuniquecodes(c, n)
for val := range c {
publishtopubsub(val)
}
}
func createuniquecodes(n) {
for i := 0; i < n; i++ {
uniquecode := some random string
// publish to channel and pubsub
c <- uniquecode
}
close(c)
}
func publishuq(msg string) error {
ctx := context.Background()
client, err := pubsub.NewClient(ctx, projectId)
if err != nil {
log.Fatalf("Could not create pubsub Client: %v", err)
}
t := client.Topic(topicName)
result := t.Publish(ctx, &pubsub.Message{
Data: []byte(msg),
})
id, err := result.Get(ctx)
if err != nil {
return err
}
fmt.Printf("Published a message; msg ID: %v\n", id)
return nil
}
Please note that I need to generate 5 million unique codes,
How will I define a context in go routine since I'm doing everything asynchronously
I assume you're using the App Engine standard (not flexible) environment. Please note that a "request handler (apihandler in your case) has a limited amount of time to generate and return a response to a request, typically around 60 seconds. Once the deadline has been reached, the request handler is interrupted".
You're trying to "break out" of the request when calling go createuniquecodes(n) and then ctx := context.Background() down the line is what panics with not an App Engine context. You could technically use NewContext(req *http.Request) to derive a valid context from the original context, but again, you'd only have 60s before your request times out.
Please have a look at TaskQueues, as they " let applications perform work, called tasks, asynchronously outside of a user request."
I am learning how to create Golang tests for an Appengine app.
The documentation examples don't make sense to me.
https://cloud.google.com/appengine/docs/standard/go/tools/localunittesting/reference
Documentation seems to say you can create a context := aetest.NewContext()
When I attempt to do so, I'm getting an error that aetest.NewContext requires arguments.
$ go test -v
./skincare_test.go:12: not enough arguments in call to aetest.NewContext
have ()
want (*aetest.Options)
./skincare_test.go:12: assignment count mismatch: 3 = 2
FAIL _/Users/Bryan/work/gocode/skincarereview [build failed]
content of skincare_test.go:
package skincare
import (
"net/http"
"net/http/httptest"
"testing"
"appengine/aetest"
)
func TestIndexHandler(t *testing.T) {
ctx, done, err := aetest.NewContext()
if err != nil {
t.Fatal(err)
}
defer done()
req, err := http.NewRequest("GET", "/", nil)
if err != nil {
t.Fatal(err)
}
rr := httptest.NewRecorder()
handler := http.HandlerFunc(root)
handler.ServeHTTP(rr, req)
if status := rr.Code; status != http.StatusOK {
t.Errorf("handler returned wrong status code: got %v want %v",
status, http.StatusOK)
}
expected := "<div>Name"
if rr.Body.String() != expected {
t.Errorf("handler returned expected body: got %v want %v",
rr.Body.String(), expected)
}
}
I learn best by looking at example code, where can I find examples of Tests for Go web applications that use Appengine datastore?
The examples in the documentation are so simple that I don't see how I'm supposed to do more complicated testing.
It says 2 things:
1) You are missing a required parameter *aetest.Options
2) that you can NOT assign result aetest.NewContext() that consist of 2 variable to a set of 3 variables.
Check what is the output of the function. I guess it is just (context.Context, error) - I suspect the done is moved to the *aetest.Options somehow.
Unfortunately my access to docs is blocked right now.
You are using the old version of the app engine package (appengine/aetest instead of google.golang.org/appengine/aetest). The newer version does not require arguments.
I have recently started programming with Go on Google App Engine and I have run into a road block. I come from Java land so it's been a slight struggle to adapt to Go.
I want to have a method that allows me to pass in a pointer to a slice that I can then pass into the datastore.GetAll call to retrieve the results. I then want to iterate through the results and use an assertion to cast as a specific interface (Queryable) in order to call a method Map().
Initially, I had this functioning properly:
func (s ProjectService) RunQuery(context context.Context, q *datastore.Query, projects *[]Project) error {
keys, err := q.GetAll(context, projects)
if err != nil {
return err
}
for i, key := range keys {
(*projects)[i].Id = key.Encode()
(*projects)[i].CompanyId = (*projects)[i].Company.Encode()
}
return nil
}
I want to have a more generic method that can be applied to any entity that implements a Queryable interface. The idea is to have a hook that allows me to perform some post processing after retrieving the results. I've looked into the ProperyLoadSaver interface however I have no access to the actual key that is associated to the entity. I would like to store the string representation of the datastore.Key in the entity.
This is the Queryable interface:
type Queryable interface {
Map(*datastore.Key) error
}
Here's an example entity that I am persisting to the GAE store:
type Camera struct {
Id string `datastore:"-"`
ProjectId string `datastore:"-"`
Name string
Project *datastore.Key `json:"-"`
Active bool
Timestamp Timestamp
}
// Implement Queryable interface. Let me perform any additional mapping
func (c *Camera) Map(key *datastore.Key) error {
c.Name = "Maybe do other things here"
c.Id = key.Encode()
return nil
}
The idea is to have something like the snippet below.
func (c Crud) RunQuery(context context.Context, q *datastore.Query, entities interface{}) error {
keys, err := q.GetAll(context, entities)
v := reflect.ValueOf(entities)
dv := v.Elem()
for i, key := range keys {
// I left this in to show that this worked however this won't let me enforce the interface contract
//dv.Index(i).FieldByName("Id").Set(reflect.ValueOf(key.Encode()))
entity := dv.Index(i).Interface().(Queryable)
entity.Map(key)
}
return err
}
However, when this executes, it panics with the following:
PANIC: interface conversion: entity.Camera is not entity.Queryable: missing method Map goroutine 9 [running]:
Just as a note, I realize the appropriate way to perform an assertion is to do if as, ok := elem.(Type); ok {} but I just wanted to see what the error was
I am guessing I am getting this error because I have defined my parameter with a pointer receiver func (c *Camera) Map(key *datastore.Key) error and not func (c Camera) Map(key *datastore.Key) error However, I want to modify the actual value.
Where am I going wrong with this? Is my Java-ness showing?
Being that I am very new to Go, I may be approaching this completely wrong.
Because the method is on a pointer receiver (as it should be), use the address of the slice element:
entity := dv.Index(i).Addr().Interface().(Queryable)
An alternative approach is to use a slice of pointers for the result:
var result []*Camera
err := c.RunQuery(ctx, q, &result)
The code can be written to work with both []Camera or []*Camera as follows:
var queryableType = reflect.TypeOf((*Queryable)(nil)).Elem()
needAddr := !dv.Type().Implements(queryableType)
...
var entity Queryable
if needAddr {
entity = dv.Index(i).Addr().Interface().(Queryable)
} else {
entity = dv.Index(i).Interface().(Queryable)
}
I have implemented the levigo wrapper in my project so I can use LevelDB. The declaration is fairly boilerplate, like so:
func NewLeveldbStorage(dbPath string) *leveldbStorage {
opts := levigo.NewOptions()
opts.SetCache(levigo.NewLRUCache(3<<30))
opts.SetCreateIfMissing(true)
log.Debugf("Entering Open")
db, err := levigo.Open(dbPath, opts); if err != nil {
log.Fatal("BOOM %v", err)
}
log.Debugf("Finished calling open")
opts.Close()
return &leveldbStorage{db:db}
}
Here is the struct returned:
type leveldbStorage struct {
db *levigo.DB
}
I then made a few simple GET and STORE commands on the struct that essentially just use s.db.Get and s.db.Put. This works fine in my tests, but when I run the following benchmark:
func BenchmarkLeviDbGet(b *testing.B) {
s := storage.NewLeveldbStorage("/path/to/db")
value := "value"
uid,_ := s.Store(value)
b.ResetTimer()
for i := 0; i < b.N; i++ {
s.Get(uid)
}
This benchmark, when run, returns:
2014/10/12 21:17:09 BOOM %vIO error: lock /path/to/db/LOCK: already held by process
Is there an appropriate way to use levigo/leveldb to enable multithreaded reading? What about writing? I would not be surprised if multithreaded writing is not possible, but multithreaded reading seems like it should be. What am I doing wrong here?
You either need to close the database file or use a global instance to it, you can't open the file multiple times, you can however access the same instance from multiple goroutines.