Slice and interface manipulation - google-app-engine

I have recently started programming with Go on Google App Engine and I have run into a road block. I come from Java land so it's been a slight struggle to adapt to Go.
I want to have a method that allows me to pass in a pointer to a slice that I can then pass into the datastore.GetAll call to retrieve the results. I then want to iterate through the results and use an assertion to cast as a specific interface (Queryable) in order to call a method Map().
Initially, I had this functioning properly:
func (s ProjectService) RunQuery(context context.Context, q *datastore.Query, projects *[]Project) error {
keys, err := q.GetAll(context, projects)
if err != nil {
return err
}
for i, key := range keys {
(*projects)[i].Id = key.Encode()
(*projects)[i].CompanyId = (*projects)[i].Company.Encode()
}
return nil
}
I want to have a more generic method that can be applied to any entity that implements a Queryable interface. The idea is to have a hook that allows me to perform some post processing after retrieving the results. I've looked into the ProperyLoadSaver interface however I have no access to the actual key that is associated to the entity. I would like to store the string representation of the datastore.Key in the entity.
This is the Queryable interface:
type Queryable interface {
Map(*datastore.Key) error
}
Here's an example entity that I am persisting to the GAE store:
type Camera struct {
Id string `datastore:"-"`
ProjectId string `datastore:"-"`
Name string
Project *datastore.Key `json:"-"`
Active bool
Timestamp Timestamp
}
// Implement Queryable interface. Let me perform any additional mapping
func (c *Camera) Map(key *datastore.Key) error {
c.Name = "Maybe do other things here"
c.Id = key.Encode()
return nil
}
The idea is to have something like the snippet below.
func (c Crud) RunQuery(context context.Context, q *datastore.Query, entities interface{}) error {
keys, err := q.GetAll(context, entities)
v := reflect.ValueOf(entities)
dv := v.Elem()
for i, key := range keys {
// I left this in to show that this worked however this won't let me enforce the interface contract
//dv.Index(i).FieldByName("Id").Set(reflect.ValueOf(key.Encode()))
entity := dv.Index(i).Interface().(Queryable)
entity.Map(key)
}
return err
}
However, when this executes, it panics with the following:
PANIC: interface conversion: entity.Camera is not entity.Queryable: missing method Map goroutine 9 [running]:
Just as a note, I realize the appropriate way to perform an assertion is to do if as, ok := elem.(Type); ok {} but I just wanted to see what the error was
I am guessing I am getting this error because I have defined my parameter with a pointer receiver func (c *Camera) Map(key *datastore.Key) error and not func (c Camera) Map(key *datastore.Key) error However, I want to modify the actual value.
Where am I going wrong with this? Is my Java-ness showing?
Being that I am very new to Go, I may be approaching this completely wrong.

Because the method is on a pointer receiver (as it should be), use the address of the slice element:
entity := dv.Index(i).Addr().Interface().(Queryable)
An alternative approach is to use a slice of pointers for the result:
var result []*Camera
err := c.RunQuery(ctx, q, &result)
The code can be written to work with both []Camera or []*Camera as follows:
var queryableType = reflect.TypeOf((*Queryable)(nil)).Elem()
needAddr := !dv.Type().Implements(queryableType)
...
var entity Queryable
if needAddr {
entity = dv.Index(i).Addr().Interface().(Queryable)
} else {
entity = dv.Index(i).Interface().(Queryable)
}

Related

How to make scanning DB rows in Go DRY?

I have a table in the database containing user account information. I have a struct called User defined.
type User struct {
Id uint
Username string
Password string
FirstName string
LastName string
Address1 string
Address2 string
.... a bunch more fields ...
}
For fetching individual user accounts, I have a method defined
func (user *User) GetById(db *sql.DB, id uint) error {
query := `SELECT
...a whole bunch of SQL ...
WHERE id = $1
... more SQL ...
LIMIT 1`
row := db.QueryRow(query, id)
err := row.Scan(
&user.Id,
&user.UserName,
&user.Password,
&user.FirstName,
&user.LastName,
... some 20 more lines of fields read into the struct ...
)
if err != nil {
return err
}
return nil
}
And there are several places in the system where I need to fetch user information as part of a larger query. That is, I am fetching some other type of object, but also a user account related to it.
That means, I have to repeat the whole rows.Scan(&user.Username, &user...) thing over and over again and it takes a whole page and it is error prone and if I ever change the user table structure I would have to change the code in a whole bunch of places. How can I make this more DRY?
Edit: I am not sure why this was marked as a duplicate, but since this edit is required, I will try to explain one more time. I am not asking how to scan a row into a struct. I already know how to do that, as the code above clearly shows. I am asking how to structure the struct scanning code in such a way that I do not have to repeat the same page of scanning code every time I am scanning the same type of struct.
Edit: also, yes, I am aware of sqlstruct and sqlx and similar libraries. I am deliberately avoiding these, because they depend on reflect package with well documented performance issues. And I intend to potentially scan millions of rows using these techniques (not millions of users, but this question extends to other record types).
Edit: so, yes, I know I should write a function. I am not sure what this function should take as arguments and what results it should return. Lets say that the other query I want to accommodate looks like this
SELECT
s.id,
s.name,
... more site fields ...
u.id,
u.username,
... more user fields ...
FROM site AS s
JOIN user AS u ON (u.id = s.user_id)
JOIN some_other_table AS st1 ON (site.id = st1.site_id)
... more SQL ...
And I have a site struct method that embeds a user struct. I don't want to repeat the user scanning code here. I want to call a function that will scan the user portion of the raw into a user struct the same way it does in the user method above.
To eliminate the repetition of the required steps to scan the *sql.Rows structure you could introduce two interfaces. One that describes the already implemented behaviour of *sql.Rows and *sql.Row.
// This interface is already implemented by *sql.Rows and *sql.Row.
type Row interface {
Scan(...interface{}) error
}
And another one that abstracts away the actual scanning step of the row(s).
// have your entity types implement this one
type RowScanner interface {
ScanRow(Row) error
}
An example implementation of the RowScanner interface could look like this:
type User struct {
Id uint
Username string
// ...
}
// Implements RowScanner
func (u *User) ScanRow(r Row) error {
return r.Scan(
&u.Id,
&u.Username,
// ...
)
}
type UserList struct {
Items []*User
}
// Implements RowScanner
func (list *UserList) ScanRow(r Row) error {
u := new(User)
if err := u.ScanRow(r); err != nil {
return err
}
list.Items = append(list.Items, u)
return nil
}
With these interfaces you can now dry your rows-scanning code for all of your types that implement the RowScanner interface by using these two functions.
func queryRows(query string, rs RowScanner, params ...interface{}) error {
rows, err := db.Query(query, params...)
if err != nil {
return err
}
defer rows.Close()
for rows.Next() {
if err := rs.ScanRow(rows); err != nil {
return err
}
}
return rows.Err()
}
func queryRow(query string, rs RowScanner, params ...interface{}) error {
return rs.ScanRow(db.QueryRow(query, params...))
}
// example
ulist := new(UserList)
if err := queryRows(queryString, ulist, arg1, arg2); err != nil {
panic(err)
}
// or
u := new(User)
if err := queryRow(queryString, u, arg1, arg2); err != nil {
panic(err)
}
If you have composite types that you want to scan but you want to avoid having to repeat the enumeration of its elements' fields, then you could introduce a method that returns a type's fields and reuse that method where you need it. For example:
func (u *User) ScannableFields() []interface{} {
return []interface{}{
&u.Id,
&u.Username,
// ...
}
}
func (u *User) ScanRow(r Row) error {
return r.Scan(u.ScannableFields()...)
}
// your other entity type
type Site struct {
Id uint
Name string
// ...
}
func (s *Site) ScannableFields() []interface{} {
return []interface{}{
&p.Id,
&p.Name,
// ...
}
}
// Implements RowScanner
func (s *Site) ScanRow(r Row) error {
return r.Scan(s.ScannableFields()...)
}
// your composite
type UserWithSite struct {
User *User
Site *Site
}
// Implements RowScanner
func (u *UserWithSite) ScanRow(r Row) error {
u.User = new(User)
u.Site = new(Site)
fields := append(u.User.ScannableFields(), u.Site.ScannableFields()...)
return r.Scan(fields...)
}
// retrieve from db
u := new(UserWithSite)
if err := queryRow(queryString, u, arg1, arg2); err != nil {
panic(err)
}

type interface {} does not support indexing in golang

I have such map:
Map := make(map[string]interface{})
This map is supposed to contain mapping from string to array of objects. Arrays can be of different types, like []Users or []Hosts. I populated this array:
TopologyMap["Users"] = Users_Array
TopologyMap["Hosts"] = Hosts_Array
but when I try to get an elements from it:
Map["Users"][0]
it gives an error:
(type interface {} does not support indexing)
How can I overcome it?
You have to explicitly convert your interface{} to a slice of the expected type to achieve it. Something like this:
package main
import "fmt"
type Host struct {
Name string
}
func main() {
Map := make(map[string]interface{})
Map["hosts"] = []Host{Host{"test.com"}, Host{"test2.com"}}
hm := Map["hosts"].([]Host)
fmt.Println(hm[0])
}
Playground link
First thing to be noted is the interface{} can hold any data type including function and struct or []struct. Since the error gives you :
(type interface {} does not support indexing)
It means that it holds no slice or no array values. Because you directly call the index in this case is 0 to an interface{} and you assume that the Map["Users"] is an array. But it is not. This is one of very good thing about Go it is statically type which mean all the data type is check at compiled time.
if you want to be avoid the parsing error like this:
panic: interface conversion: interface {} is []main.User, not
[]main.Host
To avoid that error while your parsing it to another type like Map["user"].([]User) just in case that another data type pass to the interface{} consider the code snippet below :
u, ok := myMap["user"].([]User)
if ok {
log.Printf("value = %+v\n", u)
}
Above code is simple and you can use it to check if the interface match to the type you are parsing.
And if you want to be more general passing the value to your interface{} at runtime you can check it first using reflect.TypeOf() please consider this code :
switch reflect.TypeOf(myMap["user"]).String() {
case "[]main.User":
log.Println("map = ", "slice of user")
logger.Debug("map = ", myMap["user"].([]User)[0])
case "[]main.Host":
log.Println("map = ", "slice of host")
logger.Debug("map = ", myMap["user"].([]Host)[0])
}
after you know what's the value of the interface{} you can confidently parse it the your specific data type in this case slice of user []User. Not that the main there is a package name you can change it to yours.
This is how I solved it for unstructured data. You have to parse to index string until you reach the end. Then you can print key value pairs.
yamlStr := `
input:
bind: 0.0.0.0:12002
interface: eth0
reaggregate: {}
versions: {}
output:
destination: owl-opsw-sw-dev-4.opsw:32001
interface: eth0
`
var obj map[string]interface{}
if err := yaml.Unmarshal([]byte(yamlStr), &obj); err != nil {
// Act on error
}
// Set nested object to map[string]
inputkv := streamObj["input"].(map[string]interface{})
for key, value := range inputkv {
// Each value is an interface{} type, that is type asserted as a string
fmt.Println(key, value.(string))
}
Result
bind 0.0.0.0:12002
interface eth0

Golang implementing database funcs using interfaces

Sorry for a bit dummy questions, but I'm kinda stuck.
So, I'm implementing wrapper above database driver for my application, and I need to keep it as portable as possible.
I came to decision that interfaces are perfect match for this task. So, I have my database struct with some variables and app-specific methods, and two interface functions:
query(request string) error
flush() int? string?? struct?? slice????, error
Now you probably got the main question. How do I do return data that type "flush()" doesn't know? Can I return it by interface, and if I can, how to work with that?
Second question is pretty basic, but still it isn't clear to me.
So, I have this database struct with two methods designed to be implemented by package user to use db driver he want.
Ho do I write it and how future implementation will look (there is an example on tour of go, but it's about interface for different structs with similar methods)
Hope you'll help me find an understanding :)
Yes, flush can just have the signiture;
flush() interface{}, error
How do you implement? Something like this with reasonable method bodies should do it for you;
type MyDbDriver struct {
//fields
}
func (d *MyDbDriver) query(request string) error {
return nil
}
func (d *MyDbDriver) flush() interface{}, error {
return nil, nil
}
In Go all interface implementations are implicit, meaning, if your type has methods that match the interfaces signature, then you've implemented it. No need for something like public class MyType: IMyInterface, IThisIsntCSharp. Note that in the example above *MyDbDriver has implemented your interface but MyDbDriver has not.
Edit: below a some pseudo calling code;
e := driver.query("select * from thatTable where ThatProperty = 'ThatValue'")
if e != nil {
return nil, e
}
i, err := driver.flush()
if err != nil {
return nil, err
}
MyConcreteInstance := i.(MyConcreteType)
// note that this will panic if the type of i is not MyConcreteType
// that can be avoided with the familiar object, err calling syntax
MyConcreteIntance, ok := i.(MyConcreteType)
if !ok {
// the type of i was not MyConcreteType :\
}

Weird datastore error in Go, "The kind is the empty string"

I am recently getting an error that I have never seen before when making a simple datastore.GetAll() request. I can't figure out what it means and I can't find any documentation with the error message or any help from Googleing the error message.
Here's my code:
type MyUnderlyingStruct struct {
ApplyTo *datastore.Key
ApplyFrom *datastore.Key
Amount float64
LocationKey *datastore.Key
DepartmentKey *datastore.Key
SubjectAreaKey *datastore.Key
}
type MyStruct []MyUnderlyingStruct
//In the case where I get the error someKey is a valid, complete Key value
// of a different kind that what we are querying for and there is actually
// an entity in my datastore that matches this query
func (x *MyStruct) Load(w http.ResponseWriter, r *http.Request, someKey *datastore.Key) (error) {
c := appengine.NewContext(r)
q := datastore.NewQuery("MyUnderlyingStruct_KindName").Order("-Amount")
if someKey != nil { q = q.Filter("ApplyTo=", someKey) }
keys, err := q.GetAll(c,x)
if _, ok := err.(*datastore.ErrFieldMismatch); ok { err = nil }
if err != nil && err != datastore.Done {return err}
return nil
}
Which returns this error:
API error 1 (datastore_v3: BAD_REQUEST): The kind is the empty string.
Can anyone tell me why I am getting this error, or what it is trying to tell me?
Looking at your issue on the first glance (because I am not familiar with Google's datastore API), it seems to me the problem is a result of zeroed-memory initialization using new keyword.
When a struct is created with the keyword without assigning starting values for the fields, 0's are given as default. When mapped to string, it's "" (empty). Go actually threw a very helpful error for you.
As you have pointed out, you have used Mykey := new(datastore.Key). Thanks for your generosity and this can serve as answer for future users.

Passing reflect.Value to datastore.GetMulti in Google App Engine

I have a wrapper function mypkg.GetStart around datastore.GetMulti. The arguments of the wrapper function must be identical to appengine.GetMulti. I would like to get the first two entities of dst, for the sake of this example. My code currently looks like below but does not work. datastore.GetMulti produces the error datastore: dst has invalid type.
type myEntity struct {
Val Int
}
keys := []*datastore.Key{keyOne, keyTwo, keyThree}
entities := make([]myEntity, 3)
mypkg.GetStart(c, keys, enities)
My mypkg.GetStart code is as follows:
func GetStart(c appengine.Context, keys []*datastore.Key, dst interface{}) error{
v := reflect.ValueOf(dst)
dstSlice := v.Slice(0, 2)
return datastore.GetMulti(c, keys, dstSlice)
}
How can I make this work? Note this is a follow up question to How to sub slice an interface{} that is a slice?
I got this to work by adding Interface() to dstSlice:
func GetStart(c appengine.Context, keys []*datastore.Key, dst interface{}) error{
v := reflect.ValueOf(dst)
dstSlice := v.Slice(0, 2)
return datastore.GetMulti(c, keys, dstSlice.Interface())
}

Resources