In order to reuse CGO pointers (type C.uintptr_t) between multiple applications, I tried to use go rpc to pass the initialized pointer, but the program reported an error: rpc: gob error encoding body: gob: type not registered for interface: main._Ctype_ulong. I think there might be some issues with pointer types.
1. init func
func initApp(configPath *C.char) C.uintptr_t
2. App1, daemon process, call the init func, and pass the pointer to another by go rpc
var globalSDKPtr C.ulong
type HelloService struct{}
func (p *HelloService) Hello(request string, reply *C.ulong) error {
*reply = globalSDKPtr
return nil
}
func startRPS() {
rpc.RegisterName("HelloService", new(HelloService))
listener, err := net.Listen("tcp", ":1234")
if err != nil {
log.Fatal("ListenTCP error:", err)
}
conn, err := listener.Accept()
if err != nil {
log.Fatal("Accept error:", err)
}
rpc.ServeConn(conn)
}
3. App2, recevie the pointer reuse it.
client, err := rpc.Dial("tcp", "localhost:1234")
if err != nil {
log.Fatal("dialing:", err)
}
var reply C.ulong
err = client.Call("HelloService.Hello", "hello", &reply)
if err != nil {
log.Fatal(err)
}
res := C.query(reply)
I guess the reason for the problem is that my thinking is wrong. The way to reuse cgo pointers may not be the way of go rpc, but shared memory, but in any case, passing cgo-related things is always confusing. . Can anyone help me out.
I've been dealing with an issue where I have to put an io.Reader instance as a parameter to a function provided as an end-point by an api. The task I need to do is to upload local folder to company's cloud storage.
func (s *server) uploadFileToPutIo(w http.ResponseWriter, r *http.Request) {
tokenSource := oauth2.StaticTokenSource(&oauth2.Token{AccessToken: token})
oauthClient := oauth2.NewClient(context.TODO(), tokenSource)
client := putio.NewClient(oauthClient)
var testIO io.Reader // ?
upload, err := client.Files.Upload(context.TODO(), testIO, "test", 0)
if err != nil {
log.Fatal(err)
}
fmt.Println(upload.File)
sendResponse(w, []byte("successful"), http.StatusOK)
}
When I make a request to this end-point /upload under POST method. I get the following error.
2021/12/01 18:28:47 http: panic serving 127.0.0.1:61057: runtime error: invalid memory address or nil pointer dereference
goroutine 8 [running]:
net/http.(*conn).serve.func1(0xc000108d20)
/usr/local/Cellar/go/1.16.6/libexec/src/net/http/server.go:1804 +0x153
panic(0x1390ae0, 0x164fdd0)
/usr/local/Cellar/go/1.16.6/libexec/src/runtime/panic.go:971 +0x499
io.copyBuffer(0x1462700, 0xc000026360, 0x0, 0x0, 0xc000170000, 0x8000, 0x8000, 0x0, 0x0, 0x13d5e01)
/usr/local/Cellar/go/1.16.6/libexec/src/io/io.go:423 +0x10b
io.Copy(...)
/usr/local/Cellar/go/1.16.6/libexec/src/io/io.go:382
github.com/putdotio/go-putio/putio.(*FilesService).Upload(0xc000010088, 0x1468390, 0xc00001c088, 0x0, 0x0, 0x13ef46f, 0x6, 0x0, 0x170d108, 0x90, ...)
/Users/barisertas/go/pkg/mod/github.com/putdotio/go-putio/putio#v0.0.0-20200123120452-16d982cac2b8/files.go:235 +0x187
main.(*server).uploadFileToPutIo(0xc000010028, 0x1467d60, 0xc00014a2a0, 0xc000154500)
/Users/barisertas/workspace/mini-project/api/handler.go:79 +0xe5
net/http.HandlerFunc.ServeHTTP(0xc000012db0, 0x1467d60, 0xc00014a2a0, 0xc000154500)
/usr/local/Cellar/go/1.16.6/libexec/src/net/http/server.go:2049 +0x44
github.com/gorilla/mux.(*Router).ServeHTTP(0xc000144000, 0x1467d60, 0xc00014a2a0, 0xc000154300)
/Users/barisertas/go/pkg/mod/github.com/gorilla/mux#v1.8.0/mux.go:210 +0xd3
net/http.serverHandler.ServeHTTP(0xc00014a000, 0x1467d60, 0xc00014a2a0, 0xc000154300)
/usr/local/Cellar/go/1.16.6/libexec/src/net/http/server.go:2867 +0xa3
net/http.(*conn).serve(0xc000108d20, 0x1468400, 0xc00005e300)
/usr/local/Cellar/go/1.16.6/libexec/src/net/http/server.go:1932 +0x8cd
created by net/http.(*Server).Serve
/usr/local/Cellar/go/1.16.6/libexec/src/net/http/server.go:2993 +0x39b
2021/12/01 18:28:47 http: panic serving 127.0.0.1:61059: runtime error: invalid memory address or nil pointer dereference
goroutine 11 [running]:
net/http.(*conn).serve.func1(0xc000108dc0)
/usr/local/Cellar/go/1.16.6/libexec/src/net/http/server.go:1804 +0x153
panic(0x1390ae0, 0x164fdd0)
/usr/local/Cellar/go/1.16.6/libexec/src/runtime/panic.go:971 +0x499
io.copyBuffer(0x1462700, 0xc000026400, 0x0, 0x0, 0xc000198000, 0x8000, 0x8000, 0x0, 0x0, 0x13d5e01)
/usr/local/Cellar/go/1.16.6/libexec/src/io/io.go:423 +0x10b
io.Copy(...)
/usr/local/Cellar/go/1.16.6/libexec/src/io/io.go:382
github.com/putdotio/go-putio/putio.(*FilesService).Upload(0xc0000100c8, 0x1468390, 0xc00001c088, 0x0, 0x0, 0x13ef46f, 0x6, 0x0, 0x170d108, 0x90, ...)
/Users/barisertas/go/pkg/mod/github.com/putdotio/go-putio/putio#v0.0.0-20200123120452-16d982cac2b8/files.go:235 +0x187
main.(*server).uploadFileToPutIo(0xc000010028, 0x1467d60, 0xc00014a380, 0xc000154800)
/Users/barisertas/workspace/mini-project/api/handler.go:79 +0xe5
net/http.HandlerFunc.ServeHTTP(0xc000012db0, 0x1467d60, 0xc00014a380, 0xc000154800)
/usr/local/Cellar/go/1.16.6/libexec/src/net/http/server.go:2049 +0x44
github.com/gorilla/mux.(*Router).ServeHTTP(0xc000144000, 0x1467d60, 0xc00014a380, 0xc000154600)
/Users/barisertas/go/pkg/mod/github.com/gorilla/mux#v1.8.0/mux.go:210 +0xd3
net/http.serverHandler.ServeHTTP(0xc00014a000, 0x1467d60, 0xc00014a380, 0xc000154600)
/usr/local/Cellar/go/1.16.6/libexec/src/net/http/server.go:2867 +0xa3
net/http.(*conn).serve(0xc000108dc0, 0x1468400, 0xc00005e580)
/usr/local/Cellar/go/1.16.6/libexec/src/net/http/server.go:1932 +0x8cd
created by net/http.(*Server).Serve
/usr/local/Cellar/go/1.16.6/libexec/src/net/http/server.go:2993 +0x39b
And this is the documentation of the function I am trying to use:
// Upload reads from given io.Reader and uploads the file contents to Put.io
// servers under directory given by parent. If parent is negative, user's
// preferred folder is used.
//
// If the uploaded file is a torrent file, Put.io will interpret it as a
// transfer and Transfer field will be present to represent the status of the
// tranfer. Likewise, if the uploaded file is a regular file, Transfer field
// would be nil and the uploaded file will be represented by the File field.
//
// This method reads the file contents into the memory, so it should be used for
// <150MB files.
func (f *FilesService) Upload(ctx context.Context, r io.Reader, filename string, parent int64) (Upload, error) {
if filename == "" {
return Upload{}, fmt.Errorf("filename cannot be empty")
}
var buf bytes.Buffer
mw := multipart.NewWriter(&buf)
// negative parent means use user's preferred download folder.
if parent >= 0 {
err := mw.WriteField("parent_id", itoa(parent))
if err != nil {
return Upload{}, err
}
}
formfile, err := mw.CreateFormFile("file", filename)
if err != nil {
return Upload{}, err
}
_, err = io.Copy(formfile, r)
if err != nil {
return Upload{}, err
}
err = mw.Close()
if err != nil {
return Upload{}, err
}
req, err := f.client.NewRequest(ctx, "POST", "/v2/files/upload", &buf)
if err != nil {
return Upload{}, err
}
req.Header.Set("Content-Type", mw.FormDataContentType())
var response struct {
Upload
}
_, err = f.client.Do(req, &response)
if err != nil {
return Upload{}, err
}
return response.Upload, nil
}
I am confused about putting the io.Reader instance to a function. How can I put the io.Reader function properly under this case. Just an instance after creating as follows:
var testIO io.Reader or should I do some extra operations?
runtime error: invalid memory address or nil pointer dereference
As you surely know, this is because you've declared an io.Reader but you haven't set its value, so it is still equal to the default value of an interface, which is nil.
var testIO io.Reader // ?
The point of passing the io.Reader to Upload is to provide the data to be uploaded. By passing an io.Reader, an arbitrary source of data can provide an abitrary quantity of bytes, unrestricted by memory availability (unlike []byte, which would require holding all data in memory before uploading). io.Reader is commonly used for providing data for this kind of "streaming" operation.
Upload reads from given io.Reader and uploads the file contents
That io.Reader should be the source of the data to upload.
io.Reader could be a file from os.Open().
But it could be anything that satisfies io.Reader - for example, it could also be a bytes.Buffer.
It could even be something more esoteric, like the result of an GetObject API call against the popular S3 service from AWS, which also returns an io.ReadCloser which satisfies io.Reader.
io.Reader is a great example of how Go interfaces allow independent libraries to be connected to each other. the SDK you're using doesn't care what io.Reader it's passed; it is enough that the value satisfies io.Reader, a requirement enforced at compile time. You can pass it anything that satisfies io.Reader, and the interface type guarantees that Upload() will be able to handle it properly.
Upload takes an io.Reader. If you want to pass it something like an *os.File from os.Open or an io.ReadCloser from say S3 GetObject, that works because *os.File and io.ReadCloser satisfy io.Reader. But since Upload takes io.Reader, you can be confident that it will call nothing but the functions defined in io.Reader. That means you'll have to do any closing yourself, after Upload is called.
Make sure to take the time to understand how io.Reader leaves the input to this function open-ended while also being specific about the interface it expects. This is one of the most important concepts in Go.
This:
var testIO io.Reader
is equivalent to this:
testIO := io.Reader(nil)
so this is why you are getting a panic of a nil-pointer reference:
2021/12/01 18:28:47 http: panic serving 127.0.0.1:61059: runtime error: invalid memory address or nil pointer dereference
goroutine 11 [running]:
An io.Reader is an interface which allows one to pass generic values, provided they implement the interface (i.e. implement the Read method).
Since you are uploading a file, your byte stream should come from an OS file. os.File implements the correct Read method - so is compatible io.Reader.
So try:
f, err := os.Open(uploadFilePath)
if err != nil { /* ... */ }
upload, err := client.Files.Upload(context.TODO(), f, "test", 0)
When you define a variable, it is initial value is the zero value for that type. io.Reader is an interface and the zero value for it is nil. Thus the nil pointer dereference error. Simply initialize the io.Reader before passing it to Upload:
file, err := os.Open("path/to/file")
// if err != nil { ... }
upload, err := client.Files.Upload(context.TODO(), file, "test", 0)
I am learning how to create Golang tests for an Appengine app.
The documentation examples don't make sense to me.
https://cloud.google.com/appengine/docs/standard/go/tools/localunittesting/reference
Documentation seems to say you can create a context := aetest.NewContext()
When I attempt to do so, I'm getting an error that aetest.NewContext requires arguments.
$ go test -v
./skincare_test.go:12: not enough arguments in call to aetest.NewContext
have ()
want (*aetest.Options)
./skincare_test.go:12: assignment count mismatch: 3 = 2
FAIL _/Users/Bryan/work/gocode/skincarereview [build failed]
content of skincare_test.go:
package skincare
import (
"net/http"
"net/http/httptest"
"testing"
"appengine/aetest"
)
func TestIndexHandler(t *testing.T) {
ctx, done, err := aetest.NewContext()
if err != nil {
t.Fatal(err)
}
defer done()
req, err := http.NewRequest("GET", "/", nil)
if err != nil {
t.Fatal(err)
}
rr := httptest.NewRecorder()
handler := http.HandlerFunc(root)
handler.ServeHTTP(rr, req)
if status := rr.Code; status != http.StatusOK {
t.Errorf("handler returned wrong status code: got %v want %v",
status, http.StatusOK)
}
expected := "<div>Name"
if rr.Body.String() != expected {
t.Errorf("handler returned expected body: got %v want %v",
rr.Body.String(), expected)
}
}
I learn best by looking at example code, where can I find examples of Tests for Go web applications that use Appengine datastore?
The examples in the documentation are so simple that I don't see how I'm supposed to do more complicated testing.
It says 2 things:
1) You are missing a required parameter *aetest.Options
2) that you can NOT assign result aetest.NewContext() that consist of 2 variable to a set of 3 variables.
Check what is the output of the function. I guess it is just (context.Context, error) - I suspect the done is moved to the *aetest.Options somehow.
Unfortunately my access to docs is blocked right now.
You are using the old version of the app engine package (appengine/aetest instead of google.golang.org/appengine/aetest). The newer version does not require arguments.
I have recently started programming with Go on Google App Engine and I have run into a road block. I come from Java land so it's been a slight struggle to adapt to Go.
I want to have a method that allows me to pass in a pointer to a slice that I can then pass into the datastore.GetAll call to retrieve the results. I then want to iterate through the results and use an assertion to cast as a specific interface (Queryable) in order to call a method Map().
Initially, I had this functioning properly:
func (s ProjectService) RunQuery(context context.Context, q *datastore.Query, projects *[]Project) error {
keys, err := q.GetAll(context, projects)
if err != nil {
return err
}
for i, key := range keys {
(*projects)[i].Id = key.Encode()
(*projects)[i].CompanyId = (*projects)[i].Company.Encode()
}
return nil
}
I want to have a more generic method that can be applied to any entity that implements a Queryable interface. The idea is to have a hook that allows me to perform some post processing after retrieving the results. I've looked into the ProperyLoadSaver interface however I have no access to the actual key that is associated to the entity. I would like to store the string representation of the datastore.Key in the entity.
This is the Queryable interface:
type Queryable interface {
Map(*datastore.Key) error
}
Here's an example entity that I am persisting to the GAE store:
type Camera struct {
Id string `datastore:"-"`
ProjectId string `datastore:"-"`
Name string
Project *datastore.Key `json:"-"`
Active bool
Timestamp Timestamp
}
// Implement Queryable interface. Let me perform any additional mapping
func (c *Camera) Map(key *datastore.Key) error {
c.Name = "Maybe do other things here"
c.Id = key.Encode()
return nil
}
The idea is to have something like the snippet below.
func (c Crud) RunQuery(context context.Context, q *datastore.Query, entities interface{}) error {
keys, err := q.GetAll(context, entities)
v := reflect.ValueOf(entities)
dv := v.Elem()
for i, key := range keys {
// I left this in to show that this worked however this won't let me enforce the interface contract
//dv.Index(i).FieldByName("Id").Set(reflect.ValueOf(key.Encode()))
entity := dv.Index(i).Interface().(Queryable)
entity.Map(key)
}
return err
}
However, when this executes, it panics with the following:
PANIC: interface conversion: entity.Camera is not entity.Queryable: missing method Map goroutine 9 [running]:
Just as a note, I realize the appropriate way to perform an assertion is to do if as, ok := elem.(Type); ok {} but I just wanted to see what the error was
I am guessing I am getting this error because I have defined my parameter with a pointer receiver func (c *Camera) Map(key *datastore.Key) error and not func (c Camera) Map(key *datastore.Key) error However, I want to modify the actual value.
Where am I going wrong with this? Is my Java-ness showing?
Being that I am very new to Go, I may be approaching this completely wrong.
Because the method is on a pointer receiver (as it should be), use the address of the slice element:
entity := dv.Index(i).Addr().Interface().(Queryable)
An alternative approach is to use a slice of pointers for the result:
var result []*Camera
err := c.RunQuery(ctx, q, &result)
The code can be written to work with both []Camera or []*Camera as follows:
var queryableType = reflect.TypeOf((*Queryable)(nil)).Elem()
needAddr := !dv.Type().Implements(queryableType)
...
var entity Queryable
if needAddr {
entity = dv.Index(i).Addr().Interface().(Queryable)
} else {
entity = dv.Index(i).Interface().(Queryable)
}
I am recently getting an error that I have never seen before when making a simple datastore.GetAll() request. I can't figure out what it means and I can't find any documentation with the error message or any help from Googleing the error message.
Here's my code:
type MyUnderlyingStruct struct {
ApplyTo *datastore.Key
ApplyFrom *datastore.Key
Amount float64
LocationKey *datastore.Key
DepartmentKey *datastore.Key
SubjectAreaKey *datastore.Key
}
type MyStruct []MyUnderlyingStruct
//In the case where I get the error someKey is a valid, complete Key value
// of a different kind that what we are querying for and there is actually
// an entity in my datastore that matches this query
func (x *MyStruct) Load(w http.ResponseWriter, r *http.Request, someKey *datastore.Key) (error) {
c := appengine.NewContext(r)
q := datastore.NewQuery("MyUnderlyingStruct_KindName").Order("-Amount")
if someKey != nil { q = q.Filter("ApplyTo=", someKey) }
keys, err := q.GetAll(c,x)
if _, ok := err.(*datastore.ErrFieldMismatch); ok { err = nil }
if err != nil && err != datastore.Done {return err}
return nil
}
Which returns this error:
API error 1 (datastore_v3: BAD_REQUEST): The kind is the empty string.
Can anyone tell me why I am getting this error, or what it is trying to tell me?
Looking at your issue on the first glance (because I am not familiar with Google's datastore API), it seems to me the problem is a result of zeroed-memory initialization using new keyword.
When a struct is created with the keyword without assigning starting values for the fields, 0's are given as default. When mapped to string, it's "" (empty). Go actually threw a very helpful error for you.
As you have pointed out, you have used Mykey := new(datastore.Key). Thanks for your generosity and this can serve as answer for future users.