In order to reuse CGO pointers (type C.uintptr_t) between multiple applications, I tried to use go rpc to pass the initialized pointer, but the program reported an error: rpc: gob error encoding body: gob: type not registered for interface: main._Ctype_ulong. I think there might be some issues with pointer types.
1. init func
func initApp(configPath *C.char) C.uintptr_t
2. App1, daemon process, call the init func, and pass the pointer to another by go rpc
var globalSDKPtr C.ulong
type HelloService struct{}
func (p *HelloService) Hello(request string, reply *C.ulong) error {
*reply = globalSDKPtr
return nil
}
func startRPS() {
rpc.RegisterName("HelloService", new(HelloService))
listener, err := net.Listen("tcp", ":1234")
if err != nil {
log.Fatal("ListenTCP error:", err)
}
conn, err := listener.Accept()
if err != nil {
log.Fatal("Accept error:", err)
}
rpc.ServeConn(conn)
}
3. App2, recevie the pointer reuse it.
client, err := rpc.Dial("tcp", "localhost:1234")
if err != nil {
log.Fatal("dialing:", err)
}
var reply C.ulong
err = client.Call("HelloService.Hello", "hello", &reply)
if err != nil {
log.Fatal(err)
}
res := C.query(reply)
I guess the reason for the problem is that my thinking is wrong. The way to reuse cgo pointers may not be the way of go rpc, but shared memory, but in any case, passing cgo-related things is always confusing. . Can anyone help me out.
I've been dealing with an issue where I have to put an io.Reader instance as a parameter to a function provided as an end-point by an api. The task I need to do is to upload local folder to company's cloud storage.
func (s *server) uploadFileToPutIo(w http.ResponseWriter, r *http.Request) {
tokenSource := oauth2.StaticTokenSource(&oauth2.Token{AccessToken: token})
oauthClient := oauth2.NewClient(context.TODO(), tokenSource)
client := putio.NewClient(oauthClient)
var testIO io.Reader // ?
upload, err := client.Files.Upload(context.TODO(), testIO, "test", 0)
if err != nil {
log.Fatal(err)
}
fmt.Println(upload.File)
sendResponse(w, []byte("successful"), http.StatusOK)
}
When I make a request to this end-point /upload under POST method. I get the following error.
2021/12/01 18:28:47 http: panic serving 127.0.0.1:61057: runtime error: invalid memory address or nil pointer dereference
goroutine 8 [running]:
net/http.(*conn).serve.func1(0xc000108d20)
/usr/local/Cellar/go/1.16.6/libexec/src/net/http/server.go:1804 +0x153
panic(0x1390ae0, 0x164fdd0)
/usr/local/Cellar/go/1.16.6/libexec/src/runtime/panic.go:971 +0x499
io.copyBuffer(0x1462700, 0xc000026360, 0x0, 0x0, 0xc000170000, 0x8000, 0x8000, 0x0, 0x0, 0x13d5e01)
/usr/local/Cellar/go/1.16.6/libexec/src/io/io.go:423 +0x10b
io.Copy(...)
/usr/local/Cellar/go/1.16.6/libexec/src/io/io.go:382
github.com/putdotio/go-putio/putio.(*FilesService).Upload(0xc000010088, 0x1468390, 0xc00001c088, 0x0, 0x0, 0x13ef46f, 0x6, 0x0, 0x170d108, 0x90, ...)
/Users/barisertas/go/pkg/mod/github.com/putdotio/go-putio/putio#v0.0.0-20200123120452-16d982cac2b8/files.go:235 +0x187
main.(*server).uploadFileToPutIo(0xc000010028, 0x1467d60, 0xc00014a2a0, 0xc000154500)
/Users/barisertas/workspace/mini-project/api/handler.go:79 +0xe5
net/http.HandlerFunc.ServeHTTP(0xc000012db0, 0x1467d60, 0xc00014a2a0, 0xc000154500)
/usr/local/Cellar/go/1.16.6/libexec/src/net/http/server.go:2049 +0x44
github.com/gorilla/mux.(*Router).ServeHTTP(0xc000144000, 0x1467d60, 0xc00014a2a0, 0xc000154300)
/Users/barisertas/go/pkg/mod/github.com/gorilla/mux#v1.8.0/mux.go:210 +0xd3
net/http.serverHandler.ServeHTTP(0xc00014a000, 0x1467d60, 0xc00014a2a0, 0xc000154300)
/usr/local/Cellar/go/1.16.6/libexec/src/net/http/server.go:2867 +0xa3
net/http.(*conn).serve(0xc000108d20, 0x1468400, 0xc00005e300)
/usr/local/Cellar/go/1.16.6/libexec/src/net/http/server.go:1932 +0x8cd
created by net/http.(*Server).Serve
/usr/local/Cellar/go/1.16.6/libexec/src/net/http/server.go:2993 +0x39b
2021/12/01 18:28:47 http: panic serving 127.0.0.1:61059: runtime error: invalid memory address or nil pointer dereference
goroutine 11 [running]:
net/http.(*conn).serve.func1(0xc000108dc0)
/usr/local/Cellar/go/1.16.6/libexec/src/net/http/server.go:1804 +0x153
panic(0x1390ae0, 0x164fdd0)
/usr/local/Cellar/go/1.16.6/libexec/src/runtime/panic.go:971 +0x499
io.copyBuffer(0x1462700, 0xc000026400, 0x0, 0x0, 0xc000198000, 0x8000, 0x8000, 0x0, 0x0, 0x13d5e01)
/usr/local/Cellar/go/1.16.6/libexec/src/io/io.go:423 +0x10b
io.Copy(...)
/usr/local/Cellar/go/1.16.6/libexec/src/io/io.go:382
github.com/putdotio/go-putio/putio.(*FilesService).Upload(0xc0000100c8, 0x1468390, 0xc00001c088, 0x0, 0x0, 0x13ef46f, 0x6, 0x0, 0x170d108, 0x90, ...)
/Users/barisertas/go/pkg/mod/github.com/putdotio/go-putio/putio#v0.0.0-20200123120452-16d982cac2b8/files.go:235 +0x187
main.(*server).uploadFileToPutIo(0xc000010028, 0x1467d60, 0xc00014a380, 0xc000154800)
/Users/barisertas/workspace/mini-project/api/handler.go:79 +0xe5
net/http.HandlerFunc.ServeHTTP(0xc000012db0, 0x1467d60, 0xc00014a380, 0xc000154800)
/usr/local/Cellar/go/1.16.6/libexec/src/net/http/server.go:2049 +0x44
github.com/gorilla/mux.(*Router).ServeHTTP(0xc000144000, 0x1467d60, 0xc00014a380, 0xc000154600)
/Users/barisertas/go/pkg/mod/github.com/gorilla/mux#v1.8.0/mux.go:210 +0xd3
net/http.serverHandler.ServeHTTP(0xc00014a000, 0x1467d60, 0xc00014a380, 0xc000154600)
/usr/local/Cellar/go/1.16.6/libexec/src/net/http/server.go:2867 +0xa3
net/http.(*conn).serve(0xc000108dc0, 0x1468400, 0xc00005e580)
/usr/local/Cellar/go/1.16.6/libexec/src/net/http/server.go:1932 +0x8cd
created by net/http.(*Server).Serve
/usr/local/Cellar/go/1.16.6/libexec/src/net/http/server.go:2993 +0x39b
And this is the documentation of the function I am trying to use:
// Upload reads from given io.Reader and uploads the file contents to Put.io
// servers under directory given by parent. If parent is negative, user's
// preferred folder is used.
//
// If the uploaded file is a torrent file, Put.io will interpret it as a
// transfer and Transfer field will be present to represent the status of the
// tranfer. Likewise, if the uploaded file is a regular file, Transfer field
// would be nil and the uploaded file will be represented by the File field.
//
// This method reads the file contents into the memory, so it should be used for
// <150MB files.
func (f *FilesService) Upload(ctx context.Context, r io.Reader, filename string, parent int64) (Upload, error) {
if filename == "" {
return Upload{}, fmt.Errorf("filename cannot be empty")
}
var buf bytes.Buffer
mw := multipart.NewWriter(&buf)
// negative parent means use user's preferred download folder.
if parent >= 0 {
err := mw.WriteField("parent_id", itoa(parent))
if err != nil {
return Upload{}, err
}
}
formfile, err := mw.CreateFormFile("file", filename)
if err != nil {
return Upload{}, err
}
_, err = io.Copy(formfile, r)
if err != nil {
return Upload{}, err
}
err = mw.Close()
if err != nil {
return Upload{}, err
}
req, err := f.client.NewRequest(ctx, "POST", "/v2/files/upload", &buf)
if err != nil {
return Upload{}, err
}
req.Header.Set("Content-Type", mw.FormDataContentType())
var response struct {
Upload
}
_, err = f.client.Do(req, &response)
if err != nil {
return Upload{}, err
}
return response.Upload, nil
}
I am confused about putting the io.Reader instance to a function. How can I put the io.Reader function properly under this case. Just an instance after creating as follows:
var testIO io.Reader or should I do some extra operations?
runtime error: invalid memory address or nil pointer dereference
As you surely know, this is because you've declared an io.Reader but you haven't set its value, so it is still equal to the default value of an interface, which is nil.
var testIO io.Reader // ?
The point of passing the io.Reader to Upload is to provide the data to be uploaded. By passing an io.Reader, an arbitrary source of data can provide an abitrary quantity of bytes, unrestricted by memory availability (unlike []byte, which would require holding all data in memory before uploading). io.Reader is commonly used for providing data for this kind of "streaming" operation.
Upload reads from given io.Reader and uploads the file contents
That io.Reader should be the source of the data to upload.
io.Reader could be a file from os.Open().
But it could be anything that satisfies io.Reader - for example, it could also be a bytes.Buffer.
It could even be something more esoteric, like the result of an GetObject API call against the popular S3 service from AWS, which also returns an io.ReadCloser which satisfies io.Reader.
io.Reader is a great example of how Go interfaces allow independent libraries to be connected to each other. the SDK you're using doesn't care what io.Reader it's passed; it is enough that the value satisfies io.Reader, a requirement enforced at compile time. You can pass it anything that satisfies io.Reader, and the interface type guarantees that Upload() will be able to handle it properly.
Upload takes an io.Reader. If you want to pass it something like an *os.File from os.Open or an io.ReadCloser from say S3 GetObject, that works because *os.File and io.ReadCloser satisfy io.Reader. But since Upload takes io.Reader, you can be confident that it will call nothing but the functions defined in io.Reader. That means you'll have to do any closing yourself, after Upload is called.
Make sure to take the time to understand how io.Reader leaves the input to this function open-ended while also being specific about the interface it expects. This is one of the most important concepts in Go.
This:
var testIO io.Reader
is equivalent to this:
testIO := io.Reader(nil)
so this is why you are getting a panic of a nil-pointer reference:
2021/12/01 18:28:47 http: panic serving 127.0.0.1:61059: runtime error: invalid memory address or nil pointer dereference
goroutine 11 [running]:
An io.Reader is an interface which allows one to pass generic values, provided they implement the interface (i.e. implement the Read method).
Since you are uploading a file, your byte stream should come from an OS file. os.File implements the correct Read method - so is compatible io.Reader.
So try:
f, err := os.Open(uploadFilePath)
if err != nil { /* ... */ }
upload, err := client.Files.Upload(context.TODO(), f, "test", 0)
When you define a variable, it is initial value is the zero value for that type. io.Reader is an interface and the zero value for it is nil. Thus the nil pointer dereference error. Simply initialize the io.Reader before passing it to Upload:
file, err := os.Open("path/to/file")
// if err != nil { ... }
upload, err := client.Files.Upload(context.TODO(), file, "test", 0)
In querying Gerrit, they intentionally put a )]}' at the beginning of their api response, see: https://gerrit-review.googlesource.com/Documentation/rest-api-changes.html. I am trying to remove it so the JSON is valid, but I'm unsure of the best way to do this in Go
this is my current program to query gerrit and pull out the changeID and the status from its json :
package main
import (
"encoding/json"
"flag"
"fmt"
"io/ioutil"
"log"
"net/http"
)
type gerritData struct {
ChangeID string `json:"change_id"`
Status string `json:"status"`
}
func gerritQuery(gerrit string) (gerritData, error) {
username := "redacted"
password := "redacted"
client := &http.Client{}
req, err := http.NewRequest("GET", "https://gerrit.company.com/a/changes/?q="+gerrit, nil)
req.SetBasicAuth(username, password)
resp, err := client.Do(req)
if err != nil {
log.Fatal(err)
}
respBody, err := ioutil.ReadAll(resp.Body)
// Trying to cut it out manually.
respBody = respBody[:len(respBody)-4]
if err != nil {
log.Fatal(err)
}
defer resp.Body.Close()
var gerritResponse gerritData
if err := json.NewDecoder(resp.Body).Decode(&gerritResponse); err != nil {
panic(err.Error())
}
return gerritResponse, nil
}
func main() {
gerritFlag := flag.String("gerrit", "foo", "The Gerrit you want to query")
flag.Parse()
gerritResponse, _ := gerritQuery(*gerritFlag)
fmt.Println(gerritResponse)
}
Go is still complaining with panic: invalid character ')' looking for beginning of value. I'm still new to the language so any advice would be great.
The code in the question trims four bytes from the end of the response.
Trim the bytes from the beginning of the response:
respBoby = respBody[4:]
I'm working on an appengine app using the datastore. I'm attempting to gob
encode an interface and store it into the datastore. But when I try to load from
the datastore, I get the error:
gob: name not registered for interface: "main27155.strand"
The peculiar thing is that the load() method starts working after having
called the save() method. It no longer returns an error, and everything saved
in the datastore is loaded as expected. But when I restart the intance, the
load() method stops working again.
The load and save methods I mention refer to the methods defined by the
datastore.PropertyLoadSaver interface
From the looks of it, it seems like a problem with registering the
type/interfaces with gob, but I have exactly the same gob.Register() calls in
both the load() and save() methods.
I even tried removing the gob.Register() calls from both load and save methods
and adding it to init(). The exact same behavior was observed.
How can I load my datastore on a cold start?
type bio struct {¬
Id string¬
Hp int¬
godie chan bool //should be buffered¬
dam chan int¬
Genetics dna¬
}¬
type dna interface {
decode() mRNA
Get(int) trait
Set(int, trait)
Duplicate() dna
Len() int
}
type trait interface {
mutate() trait
}
// implements dna{}
type strand []trait
// implements trait{}
type tdecoration string
type color struct {
None bool // If true, colors are not shown in theme
Bg bool // If true, color is a background color
R int // 0-255
G int
B int
}
.
func start(w http.ResponseWriter, r *http.Request) error {
c := appengine.NewContext(r)
var bs []bio
if _, err := datastore.NewQuery("bio").GetAll(c, &bs); err != nil {
log.Println("bs is len: ", len(bs))
return err
}
...
return nil
}
func stop(w http.ResponseWriter, r *http.Request) error {
c := appengine.NewContext(r)
log.Println("Saving top 20 colors")
var k []*datastore.Key
var bs []*bio
stat := getStats()
for i, b := range stat.Leaderboard {
k = append(k, datastore.NewKey(c, "bio", b.Id, 0, nil))
bv := b
bs = append(bs, &bv)
// At most 20 bios survive across reboots
if i > 178 {
break
}
}
// Assemble slice of keys for deletion
dk, err := datastore.NewQuery("bio").KeysOnly().GetAll(c, nil)
if err != nil {
return errors.New(fmt.Sprintf("Query error: %s", err.Error()))
}
fn := func(c appengine.Context) error {
// Delete all old entries
err := datastore.DeleteMulti(c, dk)
if err != nil {
return errors.New(fmt.Sprintf("Delete error: %s", err.Error()))
}
// save the elite in the datastore
_, err = datastore.PutMulti(c, k, bs)
if err != nil {
return err
}
return nil
}
return datastore.RunInTransaction(c, fn, &datastore.TransactionOptions{XG: true})
}
// satisfy datastore PropertyLoadSaver interface ===============================
func (b *bio) Load(c <-chan datastore.Property) error {
gob.Register(&color{})
gob.Register(new(tdecoration))
var str strand
gob.Register(str)
tmp := struct {
Id string
Hp int
Gengob []byte
}{}
if err := datastore.LoadStruct(&tmp, c); err != nil {
return err
}
b.Id = tmp.Id
b.Hp = tmp.Hp
return gob.NewDecoder(strings.NewReader(string(tmp.Gengob))).Decode(&(b.Genetics))
}
func (b *bio) Save(c chan<- datastore.Property) error {
defer close(c)
gob.Register(&color{})
gob.Register(new(tdecoration))
var str strand
gob.Register(str)
var buf bytes.Buffer
gen := b.Genetics
if err := gob.NewEncoder(&buf).Encode(&gen); err != nil {
log.Println(err)
return err
}
dp := []datastore.Property{
{Name: "Id", Value: b.Id},
{Name: "Hp", Value: int64(b.Hp)},
{Name: "Gengob", Value: buf.Bytes(), NoIndex: true},
}
for _, p := range dp {
c <- p
}
return nil
}
Additional info: This behavior was not present before I stuffed the datastore
calls in stop() into datastore.RunInTransaction()
Register all types an in init() functions using RegisterName(). Delete all existing data from the store and you should be good to go.
App Engine generates a mangled name for the main package every time the application is built. The name generated by Register() includes this mangled package name. Any gobs encoded with the mangled name will only be readable using the same build of the app. If you cause the application to be rebuilt by modifying the code, then the app will not be able to decode gobs stored previously.
I need utility for Windows that calculates sha256 file checksum so that when I download
fedora I can verify checksum from here: https://fedoraproject.org/static/checksums/Fedora-18-i386-CHECKSUM
Microsoft utility from http://support.microsoft.com/kb/889768 does only md5 and sha1.
I don't want to use other downloadable tools that are not signed and not available from https or from sources that I don't know about, because it does not make any sense to download unsigned code over unencrypted connection or from untrusted source to verify signature of another code to trust it.
Luckily google provides possibility to use https for all downloads so I can download Go over secure connection and start from there.
Here is simple code that does that for a small file, but it's not very good for big files because it's not streaming.
package main
import (
"io/ioutil"
"crypto/sha256"
"os"
"log"
"encoding/hex"
)
func main() {
hasher := sha256.New()
s, err := ioutil.ReadFile(os.Args[1])
hasher.Write(s)
if err != nil {
log.Fatal(err)
}
os.Stdout.WriteString(hex.EncodeToString(hasher.Sum(nil)))
}
How to make it to use streams so that it works on any file size.
The crypto/sha256 godoc actually has a snippet that shows how to do that (it's basically the same code as James):
package main
import (
"crypto/sha256"
"fmt"
"io"
"log"
"os"
)
func main() {
f, err := os.Open("file.txt")
if err != nil {
log.Fatal(err)
}
defer f.Close()
h := sha256.New()
if _, err := io.Copy(h, f); err != nil {
log.Fatal(err)
}
fmt.Printf("%x", h.Sum(nil))
}
The SHA256 hasher implements the io.Writer interface, so one option would be to use the io.Copy() function to copy the data from an appropriate io.Reader in blocks. Something like this should do:
f, err := os.Open(os.Args[1])
if err != nil {
log.Fatal(err)
}
defer f.Close()
if _, err := io.Copy(hasher, f); err != nil {
log.Fatal(err)
}
Full example of md5sum:
func md5sum(filePath string) (string, error) {
file, err := os.Open(filePath)
if err != nil {
return "", err
}
defer file.Close()
hash := md5.New()
if _, err := io.Copy(hash, file); err != nil {
return "", err
}
return hex.EncodeToString(hash.Sum(nil)), nil
}
EncodeToString does not omits leading 0 bytes, so fmt.Println(hex.EncodeToString([]byte{0x00, 0x00, 0xA, 0xB, 0xC})) gives
00000a0b0c