I have an instance of a struct that I defined and I would like to convert it to an array of bytes. I tried []byte(my_struct), but that did not work. Also, I was pointed to the binary package, but I am not sure which function I should use and how I should use it. An example would be greatly appreciated.
One possible solution is the "encoding/gob" standard package. The gob package creates an encoder/decoder that can encode any struct into an array of bytes and then decode that array back into a struct. There's a great post, here.
As others have pointed out, it's necessary to use a package like this because structs, by their nature, have unknown sizes and cannot be converted into arrays of bytes.
I've included some code and a play.
package main
import (
"bytes"
"encoding/gob"
"fmt"
"log"
)
type P struct {
X, Y, Z int
Name string
}
type Q struct {
X, Y *int32
Name string
}
func main() {
// Initialize the encoder and decoder. Normally enc and dec would be
// bound to network connections and the encoder and decoder would
// run in different processes.
var network bytes.Buffer // Stand-in for a network connection
enc := gob.NewEncoder(&network) // Will write to network.
dec := gob.NewDecoder(&network) // Will read from network.
// Encode (send) the value.
err := enc.Encode(P{3, 4, 5, "Pythagoras"})
if err != nil {
log.Fatal("encode error:", err)
}
// HERE ARE YOUR BYTES!!!!
fmt.Println(network.Bytes())
// Decode (receive) the value.
var q Q
err = dec.Decode(&q)
if err != nil {
log.Fatal("decode error:", err)
}
fmt.Printf("%q: {%d,%d}\n", q.Name, *q.X, *q.Y)
}
I assume you want something like the way C handles this. There is no built in way to do that. You will have to define your own serialization and deserialization to and from bytes for your struct. The binary package will help you encode
the fields in your struct to bytes that you can add to the byte array but you will be responsible for specifying the lengths and offsets in the byte array that will hold the fields from your struct.
Your other options are to use one of the encoding packages: http://golang.org/pkg/encoding/ such as gob or json.
EDIT:
Since you want this for making a hash as you say in your comment the easisest thing to do is use []byte(fmt.Sprintf("%v", struct)) like so: http://play.golang.org/p/yY8mSdZ_kf
Just use json marshal, this is a very simple way.
newFsConfig := dao.ConfigEntity{EnterpriseId:"testing"}
newFsConfigBytes, _ := json.Marshal(newFsConfig)
I know this thread is old, but none of the answers were accepted, and there's a pretty simple way to do this.
https://play.golang.org/p/TedsY455EBD
important code from playground
import (
"bytes"
"fmt"
"encoding/json"
)
type MyStruct struct {
Name string `json:"name"`
}
testStruct := MyStruct{"hello world"}
reqBodyBytes := new(bytes.Buffer)
json.NewEncoder(reqBodyBytes).Encode(testStruct)
reqBodyBytes.Bytes() // this is the []byte
Serialization is likely proper answer.
But if you consent to unsafety and actually need to read struct as bytes, then relying on byte array memory representation might be a bit better than relying on byte slice internal structure.
type Struct struct {
Src int32
Dst int32
SrcPort uint16
DstPort uint16
}
const sz = int(unsafe.SizeOf(Struct{}))
var asByteSlice []byte = (*(*[sz]byte)(unsafe.Pointer(&struct_value)))[:]
Works and provides read-write view into struct, zero-copy. Two "unsafe" should hint enough that it may break badly.
You should use a bytes buffer instead of a string, the other suggested methods create a SHA1 of variable length, the SHA1 standard length must be 20 bytes (160 bits)
package main
import (
"crypto/sha1"
"fmt"
"encoding/binary"
"bytes"
)
type myStruct struct {
ID string
Data string
}
func main() {
var bin_buf bytes.Buffer
x := myStruct{"1", "Hello"}
binary.Write(&bin_buf, binary.BigEndian, x)
fmt.Printf("% x", sha1.Sum(bin_buf.Bytes()))
}
Try it yourself: http://play.golang.org/p/8YuM6VIlLV
It's a really easy method and it works great.
package main
import (
"crypto/sha1"
"fmt"
"encoding/binary"
"bytes"
)
type myStruct struct {
ID [10]byte
Data [10]byte
}
func main() {
var bin_buf bytes.Buffer
x := myStruct{"1", "Hello"}
binary.Write(&bin_buf, binary.BigEndian, x)
fmt.Printf("% x", sha1.Sum(bin_buf.Bytes()))
}
binary.Write takes a struct which has fixed length memory allocated datatype.
json.Marshal is the best option to convert a struct to []byte, see example below:
package main
import (
"encoding/json"
"fmt"
)
type ExampleConvertToByteArray struct {
Name string
SurName string
}
func main() {
example := ExampleConvertToByteArray{
Name: "James",
SurName: "Camara",
}
var exampleBytes []byte
var err error
exampleBytes, err := json.Marshal(example)
if err != nil {
print(err)
return
}
fmt.Println(string(exampleBytes))
}
Go playground -> https://play.golang.org/p/mnB9Cxy-2H3
Take a look at https://blog.golang.org/go-slices-usage-and-internals
Specifically slice internals. The idea is to mimic slice's internal structure and point to our struct instead of a byte sequence:
package main
import (
"fmt"
"unsafe"
)
// our structure
type A struct {
Src int32
Dst int32
SrcPort uint16
DstPort uint16
}
// that is how we mimic a slice
type ByteSliceA struct {
Addr *A
Len int
Cap int
}
func main() {
// structure with some data
a := A{0x04030201,0x08070605,0x0A09, 0x0C0B}
// create a slice structure
sb := &ByteSliceA{&a, 12, 12} // struct is 12 bytes long, e.g. unsafe.Sizeof(a) is 12
// take a pointer of our slice mimicking struct and cast *[]byte on it:
var byteSlice []byte = *(*[]byte)(unsafe.Pointer(sb))
fmt.Printf("%v\n", byteSlice)
}
Output:
[1 2 3 4 5 6 7 8 9 10 11 12]
https://play.golang.org/p/Rh_yrscRDV6
Have you considered serializing it to bson? http://labix.org/gobson
var v any
b := (*[unsafe.Sizeof(v)]byte)(unsafe.Pointer(&v))
c := b[:]
Related
What's the best way to Unmarshal bson byte[] data into an array of structs, when the array of structs is passed into an interface{} parameter?
For demonstration purposes, in the following code, I use bson.Marshal() on the inStructArr to get the byte[] type of data. This is so I can use bson.Unmarshal(...) to pipe into the outStructArr.
import "gopkg.in/mgo.v2/bson"
type User struct {
Name string
}
func DecodeArrData(inStructArr, outStructArr interface{}) {
inStructArrData, _ := bson.Marshal(inStructArr)
bson.Unmarshal(inStructArrData, outStructArr) // <-- Error happens here
// What's the right way of accomplishing this?
}
func Main() {
outUsers := &[]User{}
inUsers := []User{User{"A"}, User{"B"}}
DecodeArrData(inUsers, outUsers)
}
When I do this, the error-message I get is: Unsupported document type for unmarshalling: []User. What's the right way of doing this?
Thanks in advance!
The Marshal and Unmarshal functions work with BSON documents, not BSON arrays.
Wrap the slices in a struct to provide the document expected by the package:
func DecodeArrData(inStructArr, outStructArr interface{}) error {
in := struct{ Data interface{} }{Data: inStructArr}
inStructArrData, err := bson.Marshal(in)
if err != nil {
return err
}
var out struct{ Data bson.Raw }
if err := bson.Unmarshal(inStructArrData, &out); err != nil {
return err
}
return out.Data.Unmarshal(outStructArr)
}
If you are willing to take advantage of an undocumented feature of the Marshal function and add some BSON format knowledge to your application, then you can omit the wrapper.
The undocumented feature of Marshal is that it encodes slices as BSON arrays. The BSON array can be decoded using a bson.Raw value with Kind set the the BSON code for arrays (the value 4) and Data set to the array data:
func DecodeArrData(inStructArr, outStructArr interface{}) error {
inStructArrData, err := bson.Marshal(inStructArr)
if err != nil {
return err
}
raw := bson.Raw{Kind: 4, Data: inStructArrData}
return raw.Unmarshal(outStructArr)
}
I want to return a structure that looks like this:
{
results: [
["ooid1", 2.0, "Söme text"],
["ooid2", 1.3, "Åther text"],
]
}
That's an array of arrags that is string, floating point number, unicode character.
If it was Python I'd be able to:
import json
json.dumps({'results': [["ooid1", 2.0, u"Söme text"], ...])
But in Go you can't have an array (or slice) of mixed types.
I thought of using a struct like this:
type Row struct {
Ooid string
Score float64
Text rune
}
But I don't want each to become a dictionary, I want it to become an array of 3 elements each.
We can customize how an object is serialized by implementing the json.Marshaler interface. For our particular case, we seem to have a slice of Row elements that we want to encode as an array of heterogenous values. We can do so by defining a MarshalJSON function on our Row type, using an intermediate slice of interface{} to encode the mixed values.
This example demonstrates:
package main
import (
"encoding/json"
"fmt"
)
type Row struct {
Ooid string
Score float64
Text string
}
func (r *Row) MarshalJSON() ([]byte, error) {
arr := []interface{}{r.Ooid, r.Score, r.Text}
return json.Marshal(arr)
}
func main() {
rows := []Row{
{"ooid1", 2.0, "Söme text"},
{"ooid2", 1.3, "Åther text"},
}
marshalled, _ := json.Marshal(rows)
fmt.Println(string(marshalled))
}
Of course, we also might want to go the other way around, from JSON bytes back to structs. So there's a similar json.Unmarshaler interface that we can use.
func (r *Row) UnmarshalJSON(bs []byte) error {
arr := []interface{}{}
json.Unmarshal(bs, &arr)
// TODO: add error handling here.
r.Ooid = arr[0].(string)
r.Score = arr[1].(float64)
r.Text = arr[2].(string)
return nil
}
This uses a similar trick of first using an intermediate slice of interface{}, using the unmarshaler to place values into this generic container, and then plop the values back into our structure.
package main
import (
"encoding/json"
"fmt"
)
type Row struct {
Ooid string
Score float64
Text string
}
func (r *Row) UnmarshalJSON(bs []byte) error {
arr := []interface{}{}
json.Unmarshal(bs, &arr)
// TODO: add error handling here.
r.Ooid = arr[0].(string)
r.Score = arr[1].(float64)
r.Text = arr[2].(string)
return nil
}
func main() {
rows := []Row{}
text := `
[
["ooid4", 3.1415, "pi"],
["ooid5", 2.7182, "euler"]
]
`
json.Unmarshal([]byte(text), &rows)
fmt.Println(rows)
}
You can read a full example here.
Use []interface{}
type Results struct {
Rows []interface{} `json:"results"`
}
You will then have to use type assertion if you want to access the values stored in []interface{}
for _, row := range results.Rows {
switch r := row.(type) {
case string:
fmt.Println("string", r)
case float64:
fmt.Println("float64", r)
case int64:
fmt.Println("int64", r)
default:
fmt.Println("not found")
}
}
Some clumsy, but you can
type result [][]interface{}
type results struct {
Results result
}
Working example https://play.golang.org/p/IXAzZZ3Dg7
I'm learning Go by writing a simple http server and I need to handle some JSON responses.
With an object response, I can unmarshal it idiomatically with 2 lines of code:
structResult := Foo{}
json.Unmarshal(structBody, &structResult)
I don't know how to do the same for an array response (see the example below). Is there a way to specify (possibly via json tag) that top-level array should go into a given struct field?
package main
import "fmt"
import "encoding/json"
type Foo struct {
Id uint64 `json:"id"`
Name string `json:"name"`
}
type BaseResult struct {
Error string `json:"error"`
}
type FooResult struct {
BaseResult
Foos []Foo
}
func main() {
// Simple and works.
structBody := []byte(`{"id": 1,"name": "foo"}`)
structResult := Foo{}
json.Unmarshal(structBody, &structResult)
fmt.Printf("%#v\n", structResult)
// Doesn't work.
arrayBody := []byte(`[{"id": 1,"name": "foo"},{"id": 2,"name": "bar"},{"id": 3,"name": "foobar"}]`)
arrayResult := FooResult{}
json.Unmarshal(arrayBody, &arrayResult)
fmt.Printf("%#v\n", arrayResult)
}
I know I could make FooResult an array:
type FooResult []Foo
but then I lose the ability to specify base object which I would like to use to store error message and such. I also know that I can unmarshal into &fooResult.Foos directly, but I want the code to work with both objects and arrays.
UPDATE
Implementing UnmarshalJSON as suggested by #dyoo partially solves my problem, but I was hoping that I could use BaseResult to store parse error in case JSON has a different structure:
arrayBody := []byte(`{"error": "foo"}`)
arrayResult := FooResult{}
json.Unmarshal(arrayBody, &arrayResult)
fmt.Printf("%#v\n", arrayResult)
Of course I could implement more complex logic inside UnmarshalJSON - but isn't there a simpler way to do it?
You can implement the json.Unmarshaler interface in your FooResult, to customize exactly how it responds to unmarshaling. (Similarly, there's a json.Marshaler interface.)
Add:
func (f *FooResult) UnmarshalJSON(bs []byte) error {
return json.Unmarshal(bs, &f.Foos)
}
after which your code should otherwise work. http://play.golang.org/p/oMdoB2e-rB
You might try something like:
func (f *FooResult) UnmarshalJSON(bs []byte) error {
err1 := json.Unmarshal(bs, &f.BaseResult)
err2 := json.Unmarshal(bs, &f.Foos)
if err1 != nil && err2 != nil {
// Arbitrarily choose an error.
return err1
}
return nil
}
although even this is beginning to look dubious. Handling union type results is not quite what the json library is designed to handle automatically for you. You'll need to explicitly code the coercion logic if your JSON has dynamic type.
See: How to unmarshall an array of different types correctly? and http://blog.golang.org/json-and-go for related issues.
Just specify Foos when you Unmarshal
package main
import "fmt"
import "encoding/json"
type Foo struct {
Id uint64 `json:"id"`
Name string `json:"name"`
}
type BaseResult struct {
Error string `json:"error"`
}
type FooResult struct {
BaseResult
Foos []Foo
}
func main() {
// Simple and works.
structBody := []byte(`{"id": 1,"name": "foo"}`)
structResult := Foo{}
json.Unmarshal(structBody, &structResult)
fmt.Printf("%#v\n", structResult)
// Doesn't work.
arrayBody := []byte(`[{"id": 1,"name": "foo"},{"id": 2,"name": "bar"},{"id": 3,"name": "foobar"}]`)
arrayResult := FooResult{}
if err := json.Unmarshal(arrayBody, &arrayResult.Foos); err != nil {
arrayResult.BaseResult.Error = string(arrayBody)
}
fmt.Printf("%#v\n", arrayResult)
}
I'm trying to use "appengine/memcache" to store data in the cache,
memcache.Item's Value field is []byte
how do I convert a struct to []byte for storing it ?
for example:
type Link struct {
Files []string
}
See the memcache.Codec type, this can be used to convert memcache items. The appengine/memcache package has two codecs already prepared, memcache.Gob and memcache.JSON. You use these codecs instead of the direct call to store and retrieve items from the cache, for example like this for a gob encoded item:
item := &memcache.Item{
Key: myCacheKey,
Object: &myLinkVar,
}
err := memcache.Gob.Set(context, item)
The encoding/gob package is probably your best option.
You could also use the encoding/json package.
If you use encoding/json you get the benefit of being able to read the values from languages other than Go.
If you use encoding/gob you get more speed.
You can use gob#Encoder.Encode:
package main
import (
"bytes"
"encoding/gob"
"fmt"
)
type link struct {
Files []string
}
func main() {
s := link{
[]string{"south", "north"},
}
b := new(bytes.Buffer)
gob.NewEncoder(b).Encode(s)
// "\x1d\xff\x81\x03\x01\x01\x04link\x01\xff\x82\x00\x01\x01\x01\x05Files\x01\xff\x84\x00\x00\x00\x16\xff\x83\x02\x01\x01\b[]string\x01\xff\x84\x00\x01\f\x00\x00\x11\xff\x82\x01\x02\x05south\x05north\x00"
fmt.Printf("%q\n", b)
}
https://golang.org/pkg/encoding/gob#Encoder.Encode
I'm playing around with wrapping FUSE with Go. However I've come stuck with how to deal with struct fuse_operations. I can't seem to expose the operations struct by declaring type Operations C.struct_fuse_operations as the members are lower case, and my pure-Go sources would have to use C-hackery to set the members anyway. My first error in this case is "can't set getattr" in what looks to be the Go equivalent of a default copy constructor. My next attempt is to expose an interface that expects GetAttr, ReadLink etc, and then generate C.struct_fuse_operations and bind the function pointers to closures that call the given interface.
This is what I've got (explanation continues after code):
package fuse
// #include <fuse.h>
// #include <stdlib.h>
import "C"
import (
//"fmt"
"os"
"unsafe"
)
type Operations interface {
GetAttr(string, *os.FileInfo) int
}
func Main(args []string, ops Operations) int {
argv := make([]*C.char, len(args) + 1)
for i, s := range args {
p := C.CString(s)
defer C.free(unsafe.Pointer(p))
argv[i] = p
}
cop := new(C.struct_fuse_operations)
cop.getattr = func(*C.char, *C.struct_stat) int {}
argc := C.int(len(args))
return int(C.fuse_main_real(argc, &argv[0], cop, C.size_t(unsafe.Sizeof(cop)), nil))
}
package main
import (
"fmt"
"fuse"
"os"
)
type CpfsOps struct {
a int
}
func (me *CpfsOps) GetAttr(string, *os.FileInfo) int {
return -1;
}
func main() {
fmt.Println(os.Args)
ops := &CpfsOps{}
fmt.Println("fuse main returned", fuse.Main(os.Args, ops))
}
This gives the following error:
fuse.go:21[fuse.cgo1.go:23]: cannot use func literal (type func(*_Ctype_char, *_Ctype_struct_stat) int) as type *[0]uint8 in assignment
I'm not sure what to pass to these members of C.struct_fuse_operations, and I've seen mention in a few places it's not possible to call from C back into Go code.
If it is possible, what should I do? How can I provide the "default" values for interface functions that acts as though the corresponding C.struct_fuse_operations member is set to NULL?
http://cheesesun.blogspot.com/2010/04/callbacks-in-cgo.html describes a general - if somewhat convoluted - technique. http://groups.google.com/group/golang-nuts/browse_thread/thread/abd0e30dafdbf297?tvc=2&pli=1 describes the problem.
I suspect that if you want to handle the function pointers as anything other than uintptrs, you'll have to hack on cgo.