In order to reuse CGO pointers (type C.uintptr_t) between multiple applications, I tried to use go rpc to pass the initialized pointer, but the program reported an error: rpc: gob error encoding body: gob: type not registered for interface: main._Ctype_ulong. I think there might be some issues with pointer types.
1. init func
func initApp(configPath *C.char) C.uintptr_t
2. App1, daemon process, call the init func, and pass the pointer to another by go rpc
var globalSDKPtr C.ulong
type HelloService struct{}
func (p *HelloService) Hello(request string, reply *C.ulong) error {
*reply = globalSDKPtr
return nil
}
func startRPS() {
rpc.RegisterName("HelloService", new(HelloService))
listener, err := net.Listen("tcp", ":1234")
if err != nil {
log.Fatal("ListenTCP error:", err)
}
conn, err := listener.Accept()
if err != nil {
log.Fatal("Accept error:", err)
}
rpc.ServeConn(conn)
}
3. App2, recevie the pointer reuse it.
client, err := rpc.Dial("tcp", "localhost:1234")
if err != nil {
log.Fatal("dialing:", err)
}
var reply C.ulong
err = client.Call("HelloService.Hello", "hello", &reply)
if err != nil {
log.Fatal(err)
}
res := C.query(reply)
I guess the reason for the problem is that my thinking is wrong. The way to reuse cgo pointers may not be the way of go rpc, but shared memory, but in any case, passing cgo-related things is always confusing. . Can anyone help me out.
Related
i've been trying to wrap my head around unit testing, dependency injection, tdd and all that stuff and i've been stuck on testing functions that make database calls, for example.
Let's say you have a PostgresStore struct that takes in a Database interface, which has a Query() method.
type PostgresStore struct {
db Database
}
type Database interface {
Query(query string, args ...interface{}) (*sql.Rows, error)
}
And your PostgresStore has a GetPatients method, which calls database query.
func (p *PostgresStore) GetPatients() ([]Patient, error) {
rows, err := p.db.Query("SELECT id, name, age, insurance FROM patients")
if err != nil {
return nil, err
}
defer rows.Close()
items := []Patient{}
for rows.Next() {
var i Patient
if err := rows.Scan(
&i.ID,
&i.Name,
&i.Surname,
&i.Age,
&i.InsuranceCompany,
); err != nil {
return nil, err
}
items = append(items, i)
}
if err := rows.Close(); err != nil {
return nil, err
}
if err := rows.Err(); err != nil {
return nil, err
}
return items, nil
}
In the real implementation, you would just pass a *sql.DB as Database argument, but how would you guys write a unit test with a fake database struct?
let me try to clarify some of your doubts. First of all, I'm gonna share a working example to better understand what's going on. Then, I'm gonna mention all of the relevant aspects.
repo/db.go
package repo
import "database/sql"
type Patient struct {
ID int
Name string
Surname string
Age int
InsuranceCompany string
}
type PostgresStore struct {
// rely on the generic DB provided by the "sql" package
db *sql.DB
}
func (p *PostgresStore) GetPatient(id int) ([]Patient, error) {
rows, err := p.db.Query("SELECT id, name, age, insurance FROM patients")
if err != nil {
return nil, err
}
defer rows.Close()
items := []Patient{}
for rows.Next() {
var i Patient
if err := rows.Scan(
&i.ID,
&i.Name,
&i.Surname,
&i.Age,
&i.InsuranceCompany,
); err != nil {
return nil, err
}
items = append(items, i)
}
if err := rows.Close(); err != nil {
return nil, err
}
if err := rows.Err(); err != nil {
return nil, err
}
return items, nil
}
Here, the only relevant change is how you define the PostgresStore struct. As the db field, you should rely on the generic DB provided by the database/sql package of the Go Standard Library. Thanks to this, it's trivial to swap its implementation with a fake one, as we're gonna see later.
Please note that in the GetPatient method you're accepting an id parameter but you're not using it. Your query is more suitable to a method like GetAllPatients or something like that. Be sure to fix it accordingly.
repo/db_test.go
package repo
import (
"testing"
"github.com/DATA-DOG/go-sqlmock"
"github.com/stretchr/testify/assert"
)
func TestGetPatient(t *testing.T) {
// 1. set up fake db and mock
db, mock, err := sqlmock.New()
if err != nil {
t.Fatalf("err not expected: %v", err)
}
// 2. configure the mock. What we expect (query or command)? The outcome (error vs no error).
rows := sqlmock.NewRows([]string{"id", "name", "surname", "age", "insurance"}).AddRow(1, "john", "doe", 23, "insurance-test")
mock.ExpectQuery("SELECT id, name, age, insurance FROM patients").WillReturnRows(rows)
// 3. instantiate the PostgresStore with the fake db
sut := &PostgresStore{
db: db,
}
// 4. invoke the action we've to test
got, err := sut.GetPatient(1)
// 5. assert the result
assert.Nil(t, err)
assert.Contains(t, got, Patient{1, "john", "doe", 23, "insurance-test"})
}
Here, there are a lot to cover. First, you can check the comments within the code that give you a better idea of each step. In the code, we're relying on the package github.com/DATA-DOG/go-sqlmock that allows us to easily mock a database client.
Obviously, the purpose of this code is to give a general idea on how to implement your needs. It can be written in a better way but it can be a good starting point for writing tests in this scenario.
Let me know if this helps, thanks!
I'm trying to issue KVM ioctls from Go code. At the moment I have something as follows:
func (vm *Vm) RegisterIrqFd(efd *EventFd, gsi uint32) error {
irqfd := (*C.struct_kvm_irqfd)(C.calloc(1, C.sizeof_struct_kvm_irqfd))
defer C.free(unsafe.Pointer(irqfd))
irqfd.fd = C.uint(efd.Fd())
irqfd.gsi = C.uint(gsi)
if _, err := sysutils.Ioctl(vm.Fd(), C.KVM_IRQFD, uintptr(unsafe.Pointer(irqfd))); err != nil {
return fmt.Errorf("RegisterIrqFd failed: %v", err)
}
return nil
}
The Ioctl function is implemented as follows :
func Ioctl(fd uintptr, cmd C.uint, arg uintptr) (uintptr, error) {
ret, _, errno := syscall.Syscall(
syscall.SYS_IOCTL,
fd,
uintptr(cmd),
uintptr(arg),
)
if int64(ret) == -1 {
return ret, ErrnoToErr(errno)
}
return ret, nil
}
And the ErrnoToErr function is implemented as follows :
func ErrnoToErr(errno syscall.Errno) error {
return fmt.Errorf("%v", errno.Error())
}
Inside GetSupportedCpuid, the argument for the ioctl, cpuid is allocated using C.calloc. If this were C code cpuid could just be allocated on the stack. Is there any way I can get around allocating using calloc and using free? Any alternatives that would be more idiomatic?
Would there be a way to have ErrnoToErr return a go error type corresponding to the errno? Currently it uses the (e Errno) Error() inside syscall/syscall_unix.go; so it just dereferences a list of strings and returns the string. It would be nice to have a type, so that I can test for specific errors in my unit tests.
Replying just to the updated example, we could do this:
func (vm *Vm) RegisterIrqFd(efd *EventFd, gsi uint32) error {
irqfd := C.struct_kvm_irqfd{}
irqfd.fd = C.uint(efd.Fd())
irqfd.gsi = C.uint(gsi)
if _, err := sysutils.Ioctl(vm.Fd(), C.KVM_IRQFD, uintptr(unsafe.Pointer(&irqfd))); err != nil {
return fmt.Errorf("RegisterIrqFd failed: %v", err)
}
return nil
}
I think this is safe, but we might find that irqfd escapes to heap anyway since we take its address here, in which case there's not much savings. Even so, as long as this doesn't violate Go pointer rules (I don't think it does) it's a lot nicer to read.
Read the documentation syscall.Errno
So convertion may be performed like this
func ErrnoToErr(errno syscall.Errno) error {
if errno != 0 {
return errno
}
return nil
}
For equality it only states
Errno values can be tested against error values from the os package
using errors.Is.
The question of comparing errors was already considered here How to compare Go errors
I'm developing a set of functions (based on KNative Serving + Eventing) and I'm having a lot of problems getting the decoded data on the receiving-end if the stream.
So, I'm implementing a couple of functions that, will use the Google APIs (via google.golang.org/api/slides/v1) to grab a Google Slides Presentation (a struct from the library), encode it as a []byte and send it over the network with protobuf/gRPC.
This appears to be working correctly, however, when I try to decode it back into a Presentation I'm getting an error. Printing out the Decode call returns only EOF.
Here's the proto definition:
syntax = "proto3";
package api;
message ParserRequest {
bytes Presentation = 1;
}
message ParserResponse {
int32 Status = 1;
bytes Document = 2;
}
service ParserService {
// ParsePresentation parse the Google Slides presentation it into the SDR representation
rpc ParsePresentation(ParserRequest) returns (ParserResponse) {}
}
The sending function is similar to:
presentation, err := svc.Presentations.Get(docID).Do()
if err != nil {
log.Fatalf("Unable to retrieve data from document: %+v", err)
}
if presentation != nil {
log.Printf("Calling Parser...")
address := "PORT:IP"
conn, err := grpc.DialContext(ctx, address, grpc.WithInsecure())
if err != nil {
log.Printf("Dial Error! %+v", err)
return nil, fmt.Errorf("could not connect shipping service: %+v", err)
}
defer conn.Close()
log.Printf("Marshalling data...")
// data, err := presentation.MarshalJSON()
// data, err := json.Marshal(presentation)
var buf bytes.Buffer
enc := gob.NewEncoder(&buf)
err = enc.Encode(&presentation)
if err != nil {
log.Println("Encode Error: %+v", err)
log.Fatal("encode error:", err)
}
data := buf.Bytes()
log.Println(data) //[255 211 255 129 3 1 1 12 ...]
cli, err := parser.NewParserServiceClient(conn).ParsePresentation(ctx, &parser.ParserRequest{Presentation: data})
if err != nil {
log.Printf("Cli call Error! %+v", err)
return nil, fmt.Errorf("failed to get parser service: %+v", err)
}
log.Printf("Result: %d", cli.Status)
}
On the other end, I should now decode the data array and "translate" back into a Presentation struct, doing so via:
func (c *parserService) ParsePresentation(ctx context.Context, in *pb.ParserRequest) (*pb.ParserResponse, error) {
log.Printf("ParserService.ParsePresentation was called!")
if in.Presentation == nil {
log.Fatalf("Missing parameter Google Slides document.")
}
sdrDocument, err := gslides_parser.ParsePresentationBytes(in.Presentation)
if err != nil {
log.Fatalf("Unable to parse the Google Slides presentation: %+v", err)
}
presentation, err := json.Marshal(sdrDocument)
return &pb.ParserResponse{Status: 200, Document: presentation}, nil
}
Than, when it gets to the gslides_parser.ParsePresentationBytes(in.Presentation) will then be decoded:
func ParsePresentationBytes(presentationParam []byte) (sdr.Document, error) {
var (
document sdr.Document
presentation slides.Presentation
)
log.Printf("PresentationBytes gob decoder")
log.Println(presentationParam) // Output: [255 211 255 129 3 1 1 12 ...]
// err := json.Unmarshal(presentationParam, &presentation)
buf := bytes.NewBuffer(presentationParam)
log.Printf("New decoder...")
dec := gob.NewDecoder(buf)
log.Printf("Decode...")
log.Println(&presentation) // Output: &{[] [] <nil> <nil> [] {0 map[]} [] []}
err := dec.Decode(&presentation)
log.Printf("Done decoding...")
if err != nil {
// Never gets here
document = sdr.Document{}
(...)
} else {
log.Printf("PresentationBytes Error!")
}
return document, nil
}
So, why can't I decode the information? I don't see anything terribly wrong with this code, but I'm also a golang newb so I may have some error that is eluding me.
Isn't gob the appropriate way of dealing with this? I tried simply marshalling/unmarshalling but that also produces errors.
I try to serialize a structured data to file. I looked through some examples and made such construction:
func (order Order) Serialize(folder string) {
b := bytes.Buffer{}
e := gob.NewEncoder(&b)
err := e.Encode(order)
if err != nil { panic(err) }
os.MkdirAll(folder, 0777)
file, err := os.Create(folder + order.Id)
if err != nil { panic(err) }
defer file.Close()
writer := bufio.NewWriter(file)
n, err := writer.Write(b.Bytes())
fmt.Println(n)
if err != nil {
panic(err)
}
}
Serialize is a method serializing its object to file called by it's id property. I looked through debugger - byte buffer contains data before writing. I mean object is fully initialized. Even n variable representing quantity of written bytes is more than a thousand - the file shouldn't be empty at all. The file is created but it is totally empty. What's wrong?
bufio.Writer (as the package name hints) uses a buffer to cache writes. If you ever use it, you must call Writer.Flush() when you're done writing to it to ensure the buffered data gets written to the underlying io.Writer.
Also note that you can directly write to an os.File, no need to create a buffered writer "around" it. (*os.File implements io.Writer).
Also note that you can create the gob.Encoder directly directed to the os.File, so even the bytes.Buffer is unnecessary.
Also os.MkdirAll() may fail, check its return value.
Also it's better to "concatenate" parts of a file path using filepath.Join() which takes care of extra / missing slashes at the end of folder names.
And last, it would be better to signal the failure of Serialize(), e.g. with an error return value, so the caller party has the chance to examine if the operation succeeded, and act accordingly.
So Order.Serialize() should look like this:
func (order Order) Serialize(folder string) error {
if err := os.MkdirAll(folder, 0777); err != nil {
return err
}
file, err := os.Create(filepath.Join(folder, order.Id))
if err != nil {
return err
}
defer file.Close()
if err := gob.NewEncoder(file).Encode(order); err != nil {
return err
}
return nil
}
I am trying to use a file instead of a DB to get a prototype up and running. I have a program that (1) reads existing content from the file to a map, (2) takes JSON POSTs that add content to the map, (3) on exit, writes to the file.
First, the file is not being created. Then I created an empty file. It is not being written to.
I am trying to read the file, determine if there is existing content. If there is not existing content, create a blank map. If there is existing content, unmarshal it into a new map.
func writeDB() {
eventDBJSON, err := json.Marshal(eventDB)
if err != nil {
panic(err)
}
err2 := ioutil.WriteFile("/Users/sarah/go/dat.txt", eventDBJSON, 0777)
if err2 != nil {
panic(err2)
}
}
func main() {
dat, err := ioutil.ReadFile("/Users/sarah/go/dat.txt")
if err != nil {
panic(err)
}
if dat == nil {
eventDB = DB{
events: map[string]event{},
}
} else {
if err2 := json.Unmarshal(dat, &eventDB); err2 != nil {
panic(err2)
}
}
router := httprouter.New()
router.POST("/join", JoinEvent)
router.POST("/create", CreateEvent)
log.Fatal(http.ListenAndServe(":8080", router))
defer writeDB()
}
There is no way for the server to ever reach defer writeDB().
http.ListenAndServe blocks, and if it did return anything, you log.Fatal that, which exits your app at that point.
You can't intercept all ways an app can exit, getting SIGKILL, machine loss of power, etc.
I'm assuming you really just want to write some code, bounce the server, repeat
If that's the case, then Ctrl-C is good enough.
If you want to write your file on Ctrl-C, look at the signal package.
Also, defer on the last line of a function really has no purpose as defer basically means "do this last".
you can use (*os.File).Stat() to get a file's FileInfo which contain its size
file, err := os.Open( filepath )
if err != nil {
// handle error
}
fi, err := file.Stat()
if err != nil {
// handle error
}
s := fi.Size()