Unmarshalling top-level JSON array in Go - arrays

I'm learning Go by writing a simple http server and I need to handle some JSON responses.
With an object response, I can unmarshal it idiomatically with 2 lines of code:
structResult := Foo{}
json.Unmarshal(structBody, &structResult)
I don't know how to do the same for an array response (see the example below). Is there a way to specify (possibly via json tag) that top-level array should go into a given struct field?
package main
import "fmt"
import "encoding/json"
type Foo struct {
Id uint64 `json:"id"`
Name string `json:"name"`
}
type BaseResult struct {
Error string `json:"error"`
}
type FooResult struct {
BaseResult
Foos []Foo
}
func main() {
// Simple and works.
structBody := []byte(`{"id": 1,"name": "foo"}`)
structResult := Foo{}
json.Unmarshal(structBody, &structResult)
fmt.Printf("%#v\n", structResult)
// Doesn't work.
arrayBody := []byte(`[{"id": 1,"name": "foo"},{"id": 2,"name": "bar"},{"id": 3,"name": "foobar"}]`)
arrayResult := FooResult{}
json.Unmarshal(arrayBody, &arrayResult)
fmt.Printf("%#v\n", arrayResult)
}
I know I could make FooResult an array:
type FooResult []Foo
but then I lose the ability to specify base object which I would like to use to store error message and such. I also know that I can unmarshal into &fooResult.Foos directly, but I want the code to work with both objects and arrays.
UPDATE
Implementing UnmarshalJSON as suggested by #dyoo partially solves my problem, but I was hoping that I could use BaseResult to store parse error in case JSON has a different structure:
arrayBody := []byte(`{"error": "foo"}`)
arrayResult := FooResult{}
json.Unmarshal(arrayBody, &arrayResult)
fmt.Printf("%#v\n", arrayResult)
Of course I could implement more complex logic inside UnmarshalJSON - but isn't there a simpler way to do it?

You can implement the json.Unmarshaler interface in your FooResult, to customize exactly how it responds to unmarshaling. (Similarly, there's a json.Marshaler interface.)
Add:
func (f *FooResult) UnmarshalJSON(bs []byte) error {
return json.Unmarshal(bs, &f.Foos)
}
after which your code should otherwise work. http://play.golang.org/p/oMdoB2e-rB
You might try something like:
func (f *FooResult) UnmarshalJSON(bs []byte) error {
err1 := json.Unmarshal(bs, &f.BaseResult)
err2 := json.Unmarshal(bs, &f.Foos)
if err1 != nil && err2 != nil {
// Arbitrarily choose an error.
return err1
}
return nil
}
although even this is beginning to look dubious. Handling union type results is not quite what the json library is designed to handle automatically for you. You'll need to explicitly code the coercion logic if your JSON has dynamic type.
See: How to unmarshall an array of different types correctly? and http://blog.golang.org/json-and-go for related issues.

Just specify Foos when you Unmarshal
package main
import "fmt"
import "encoding/json"
type Foo struct {
Id uint64 `json:"id"`
Name string `json:"name"`
}
type BaseResult struct {
Error string `json:"error"`
}
type FooResult struct {
BaseResult
Foos []Foo
}
func main() {
// Simple and works.
structBody := []byte(`{"id": 1,"name": "foo"}`)
structResult := Foo{}
json.Unmarshal(structBody, &structResult)
fmt.Printf("%#v\n", structResult)
// Doesn't work.
arrayBody := []byte(`[{"id": 1,"name": "foo"},{"id": 2,"name": "bar"},{"id": 3,"name": "foobar"}]`)
arrayResult := FooResult{}
if err := json.Unmarshal(arrayBody, &arrayResult.Foos); err != nil {
arrayResult.BaseResult.Error = string(arrayBody)
}
fmt.Printf("%#v\n", arrayResult)
}

Related

Calling valiable name from another variable

Please help to call the variable from another variable.
I have the script which is taking EC2 instances and return in "*ec2.Instance" variable.
I can print information from static text, for example :
fmt.Println(instance.InstanceType) // t3.small
But I have the list of reqired fields like and don't know how dynamic use name from this list :
fields := []string{"InstanceId", "InstanceType", "PrivateIpAddress"}
for i := range fields {
fmt.Println(fields[i])
fmt.Println(instance.fields[i]) // Not correct ... :(
}
You need to use reflection to do this in go.
The key takeaway is you need to "analyze" the returned value at runtime and access properties by name from the "reflected" structure. Reflection basically means analyzing objects at runtime.
package main
import (
"fmt"
"reflect"
)
type whatever struct {
cat string
dog string
animals int
something string
}
func main() {
wantProps := []string{ "cat", "animals"}
we := whatever{cat: "meow", animals: 22}
r := reflect.ValueOf(we)
for _, propName := range wantProps {
prop := r.FieldByName(propName)
fmt.Println(propName, prop)
}
}
More details:
Golang dynamic access to a struct property

In Go, how to Unmarshal bson byte[] data into an array of structs?

What's the best way to Unmarshal bson byte[] data into an array of structs, when the array of structs is passed into an interface{} parameter?
For demonstration purposes, in the following code, I use bson.Marshal() on the inStructArr to get the byte[] type of data. This is so I can use bson.Unmarshal(...) to pipe into the outStructArr.
import "gopkg.in/mgo.v2/bson"
type User struct {
Name string
}
func DecodeArrData(inStructArr, outStructArr interface{}) {
inStructArrData, _ := bson.Marshal(inStructArr)
bson.Unmarshal(inStructArrData, outStructArr) // <-- Error happens here
// What's the right way of accomplishing this?
}
func Main() {
outUsers := &[]User{}
inUsers := []User{User{"A"}, User{"B"}}
DecodeArrData(inUsers, outUsers)
}
When I do this, the error-message I get is: Unsupported document type for unmarshalling: []User. What's the right way of doing this?
Thanks in advance!
The Marshal and Unmarshal functions work with BSON documents, not BSON arrays.
Wrap the slices in a struct to provide the document expected by the package:
func DecodeArrData(inStructArr, outStructArr interface{}) error {
in := struct{ Data interface{} }{Data: inStructArr}
inStructArrData, err := bson.Marshal(in)
if err != nil {
return err
}
var out struct{ Data bson.Raw }
if err := bson.Unmarshal(inStructArrData, &out); err != nil {
return err
}
return out.Data.Unmarshal(outStructArr)
}
If you are willing to take advantage of an undocumented feature of the Marshal function and add some BSON format knowledge to your application, then you can omit the wrapper.
The undocumented feature of Marshal is that it encodes slices as BSON arrays. The BSON array can be decoded using a bson.Raw value with Kind set the the BSON code for arrays (the value 4) and Data set to the array data:
func DecodeArrData(inStructArr, outStructArr interface{}) error {
inStructArrData, err := bson.Marshal(inStructArr)
if err != nil {
return err
}
raw := bson.Raw{Kind: 4, Data: inStructArrData}
return raw.Unmarshal(outStructArr)
}

How to convert array of errors to JSON in Go

I have an array of errors (type of error), but when I try to return the client in JSON format, it arrives empty.
It was created this way:
var (
ErrEmptyName = errors.New("Nome não pode ser vázio")
ErrInvalidType = errors.New("Tipo de pessoa inválido")
)
func (p Person) Validate() []error {
var errors []error
if govalidator.IsNull(p.Name) {
errors = append(errors, ErrEmptyName)
}
if p.Type != "F" && p.Type != "J" {
errors = append(errors, ErrInvalidType)
}
return errors
)
In my Controller:
err := person.Validate()
c.JSON(422, gin.H{"errors" : err})
My Output:
{"errors":"[{}]"}
The error type is an interface with a single Error() method, but it is not special to the json package (the Error() method is not called on it).
However, error values may hold values of static types which may be nicely marshalable, or they may define their own marshaling logic by implementing json.Marshaler. Simply converting an error to string by calling its Error() method means we're not honoring the custom marshaling logic.
So I would suggest to create our own error slice type, on which we can implement our marshaling logic which should be:
check if the error value implements json.Marshaler, and if so, let it marshal itself
else as a fallback case call error.Error() to "obtain" a string which can easily be marshaled
This is how it could look like:
type JSONErrs []error
func (je JSONErrs) MarshalJSON() ([]byte, error) {
res := make([]interface{}, len(je))
for i, e := range je {
if _, ok := e.(json.Marshaler); ok {
res[i] = e // e knows how to marshal itself
} else {
res[i] = e.Error() // Fallback to the error string
}
}
return json.Marshal(res)
}
And this is how you can use it:
err := person.Validate()
c.JSON(422, gin.H{"errors" : JSONErrs(err)})
Let's test our JSONErrs. We're also use a custom error type which implements custom marshaling logic:
type MyErr struct{ line int }
func (me MyErr) Error() string { return "Invalid input!" }
func (me MyErr) MarshalJSON() ([]byte, error) {
return json.Marshal(
struct {
Type, Error string
AtLine int
}{"MyErr", me.Error(), me.line})
}
And the test code:
errs := []error{
errors.New("first"),
errors.New("second"),
MyErr{16},
}
data, err := json.Marshal(JSONErrs(errs))
fmt.Println(string(data), err)
Output (try it on the Go Playground):
["first","second",{"Type":"MyErr","Error":"Invalid input!","AtLine":16}] <nil>
An error type is an interface which must implement an Error() method that returns an error message as a string. This is defined here: https://golang.org/pkg/builtin/#error. The reason why the error type is an interface, is to allow for error types that can be type casted to retrieve more detailed information.
Functions like fmt.Println and log.Println automatically resolves error types to display the message from Error(), the JSON library however, does not. The simplest way to get around this problem is by converting the error messages in []error to a []string and encoding that.
Here's some example code to do that with a for loop.
errs := person.Validate()
strErrors := make([]string, len(errs))
for i, err := range errs {
strErrors[i] = err.Error()
}
c.JSON(422, gin.H{"errors" : strErrors})

strategy for REST API in go

In my database, each row corresponds to a struct
type datum struct{
Id *string `json:"task_id"`
Status *string `json:"status"`
AccountId *string `json:"account_id"`
.... // many more fields, all of pointer types
}
On the webpage, the user can query on several fields of datum (say account_id and status). The server will return all data that satisfy the query with a projection of the fields (say Id, account_id and status).
Right now, I wrote a HTTP handler to
Extract the query as a datum object from the request:
body, err := ioutil.ReadAll(r.Body)
condition := datum{}
err = json.Unmarshal(body, &condition)
Use the partially filled datum object to query the database, only the non-nil fields translate to SELECT ... WHERE ..=... The query result is saved in query_result []datum
Write the query_result into json object for reply:
reply := map[string]interface{}{
"reply": query_result,
}
data, err := json.Marshal(reply)
The problem is that in the reply many of the fields are nil, but I still send them, which is wasteful. On the other hand, I don't want to change the datum struct to include omitempty tag because in the database a value entry has all fields non-nil.
In this case, shall I define a new struct just for the reply? Is there a way to define this new struct using datum struct, instead of hard code one?
Is there a better design for this query feature?
You have several options, with choice depending what is more wasteful/expensive in your particular case:
Just use pointers+omitempty in the original struct.
Prepare a custom response object. But you'll need to copy/convert the values from the original struct into its export version.
Write a custom marshaller, that will be exploring your struct and creating an export-ready variant, this way being more dynamic/automatic that #1.
While #1 needs no comments, and #2 to some extend covered by Gepser above, here's how you can address this with a custom marshaller (the idea is to re-assemble your output skipping nil fields):
package main
import (
"fmt"
"encoding/json"
"reflect"
)
type datum struct {
Id *string `json:"task_id"`
Status *string `json:"status"`
AccountId *string `json:"account_id"`
}
type Response struct {
Reply []datum `json:"reply"`
}
func main() {
var query_result []datum
// mocking a query result with records with nil fields
val_id_a := "id-a"
val_status := "status-b"
d1 := datum{
Id: &val_id_a,
Status: &val_status,
}
query_result = append(query_result, d1)
val_id_b := "id-b"
val_account_id := "account-id-b"
d2 := datum{
Id: &val_id_b,
AccountId: &val_account_id,
}
query_result = append(query_result, d2)
reply := &Response{
Reply: query_result,
}
data, err := json.Marshal(reply)
if err != nil {
panic(err)
}
fmt.Printf("%+v\n", string(data))
}
// MarshalJSON is a custom JSON marshaller implementation for Response object.
func (r *Response) MarshalJSON() ([]byte, error) {
a := struct {
Reply []map[string]interface{} `json:"reply"`
}{}
for _, v := range r.Reply {
a.Reply = append(a.Reply, converter(v))
}
return json.Marshal(a)
}
// converter converts a struct into a map, skipping fields with nil values.
func converter(in interface{}) map[string]interface{} {
out := make(map[string]interface{})
v := reflect.ValueOf(in)
for i := 0; i < v.NumField(); i++ {
f := v.Type().Field(i)
tag := f.Tag.Get("json")
if tag != "" && !v.Field(i).IsNil() {
out[tag] = v.Field(i).Interface()
}
}
return out
}
The approach I suggest (is the one I use) is the new struct with omitempty tag, for example:
type datumResponse struct{
Id *string `json:"task_id,omitempty"`
Status *string `json:"status,omitempty"`
AccountId *string `json:"account_id,omitempty"`
.... // many more fields
}
and there is no option to write your new struct using the fields of the old one if there is not substructs or you don't write an array of structs.

Serialize a mixed type JSON array in Go

I want to return a structure that looks like this:
{
results: [
["ooid1", 2.0, "Söme text"],
["ooid2", 1.3, "Åther text"],
]
}
That's an array of arrags that is string, floating point number, unicode character.
If it was Python I'd be able to:
import json
json.dumps({'results': [["ooid1", 2.0, u"Söme text"], ...])
But in Go you can't have an array (or slice) of mixed types.
I thought of using a struct like this:
type Row struct {
Ooid string
Score float64
Text rune
}
But I don't want each to become a dictionary, I want it to become an array of 3 elements each.
We can customize how an object is serialized by implementing the json.Marshaler interface. For our particular case, we seem to have a slice of Row elements that we want to encode as an array of heterogenous values. We can do so by defining a MarshalJSON function on our Row type, using an intermediate slice of interface{} to encode the mixed values.
This example demonstrates:
package main
import (
"encoding/json"
"fmt"
)
type Row struct {
Ooid string
Score float64
Text string
}
func (r *Row) MarshalJSON() ([]byte, error) {
arr := []interface{}{r.Ooid, r.Score, r.Text}
return json.Marshal(arr)
}
func main() {
rows := []Row{
{"ooid1", 2.0, "Söme text"},
{"ooid2", 1.3, "Åther text"},
}
marshalled, _ := json.Marshal(rows)
fmt.Println(string(marshalled))
}
Of course, we also might want to go the other way around, from JSON bytes back to structs. So there's a similar json.Unmarshaler interface that we can use.
func (r *Row) UnmarshalJSON(bs []byte) error {
arr := []interface{}{}
json.Unmarshal(bs, &arr)
// TODO: add error handling here.
r.Ooid = arr[0].(string)
r.Score = arr[1].(float64)
r.Text = arr[2].(string)
return nil
}
This uses a similar trick of first using an intermediate slice of interface{}, using the unmarshaler to place values into this generic container, and then plop the values back into our structure.
package main
import (
"encoding/json"
"fmt"
)
type Row struct {
Ooid string
Score float64
Text string
}
func (r *Row) UnmarshalJSON(bs []byte) error {
arr := []interface{}{}
json.Unmarshal(bs, &arr)
// TODO: add error handling here.
r.Ooid = arr[0].(string)
r.Score = arr[1].(float64)
r.Text = arr[2].(string)
return nil
}
func main() {
rows := []Row{}
text := `
[
["ooid4", 3.1415, "pi"],
["ooid5", 2.7182, "euler"]
]
`
json.Unmarshal([]byte(text), &rows)
fmt.Println(rows)
}
You can read a full example here.
Use []interface{}
type Results struct {
Rows []interface{} `json:"results"`
}
You will then have to use type assertion if you want to access the values stored in []interface{}
for _, row := range results.Rows {
switch r := row.(type) {
case string:
fmt.Println("string", r)
case float64:
fmt.Println("float64", r)
case int64:
fmt.Println("int64", r)
default:
fmt.Println("not found")
}
}
Some clumsy, but you can
type result [][]interface{}
type results struct {
Results result
}
Working example https://play.golang.org/p/IXAzZZ3Dg7

Resources