How to convert array of errors to JSON in Go - arrays

I have an array of errors (type of error), but when I try to return the client in JSON format, it arrives empty.
It was created this way:
var (
ErrEmptyName = errors.New("Nome não pode ser vázio")
ErrInvalidType = errors.New("Tipo de pessoa inválido")
)
func (p Person) Validate() []error {
var errors []error
if govalidator.IsNull(p.Name) {
errors = append(errors, ErrEmptyName)
}
if p.Type != "F" && p.Type != "J" {
errors = append(errors, ErrInvalidType)
}
return errors
)
In my Controller:
err := person.Validate()
c.JSON(422, gin.H{"errors" : err})
My Output:
{"errors":"[{}]"}

The error type is an interface with a single Error() method, but it is not special to the json package (the Error() method is not called on it).
However, error values may hold values of static types which may be nicely marshalable, or they may define their own marshaling logic by implementing json.Marshaler. Simply converting an error to string by calling its Error() method means we're not honoring the custom marshaling logic.
So I would suggest to create our own error slice type, on which we can implement our marshaling logic which should be:
check if the error value implements json.Marshaler, and if so, let it marshal itself
else as a fallback case call error.Error() to "obtain" a string which can easily be marshaled
This is how it could look like:
type JSONErrs []error
func (je JSONErrs) MarshalJSON() ([]byte, error) {
res := make([]interface{}, len(je))
for i, e := range je {
if _, ok := e.(json.Marshaler); ok {
res[i] = e // e knows how to marshal itself
} else {
res[i] = e.Error() // Fallback to the error string
}
}
return json.Marshal(res)
}
And this is how you can use it:
err := person.Validate()
c.JSON(422, gin.H{"errors" : JSONErrs(err)})
Let's test our JSONErrs. We're also use a custom error type which implements custom marshaling logic:
type MyErr struct{ line int }
func (me MyErr) Error() string { return "Invalid input!" }
func (me MyErr) MarshalJSON() ([]byte, error) {
return json.Marshal(
struct {
Type, Error string
AtLine int
}{"MyErr", me.Error(), me.line})
}
And the test code:
errs := []error{
errors.New("first"),
errors.New("second"),
MyErr{16},
}
data, err := json.Marshal(JSONErrs(errs))
fmt.Println(string(data), err)
Output (try it on the Go Playground):
["first","second",{"Type":"MyErr","Error":"Invalid input!","AtLine":16}] <nil>

An error type is an interface which must implement an Error() method that returns an error message as a string. This is defined here: https://golang.org/pkg/builtin/#error. The reason why the error type is an interface, is to allow for error types that can be type casted to retrieve more detailed information.
Functions like fmt.Println and log.Println automatically resolves error types to display the message from Error(), the JSON library however, does not. The simplest way to get around this problem is by converting the error messages in []error to a []string and encoding that.
Here's some example code to do that with a for loop.
errs := person.Validate()
strErrors := make([]string, len(errs))
for i, err := range errs {
strErrors[i] = err.Error()
}
c.JSON(422, gin.H{"errors" : strErrors})

Related

Range over elements of a protobuf array in Go

I have a Protobuf structure defined as so in my .proto file:
message Msg{
message SubMsg {
string SubVariable1 = 1;
int32 SubVariable2 = 2;
...
}
string Variable1 = 1;
repeated SubMsg Variable2 = 2;
...
}
I pull data into this structure using the https://godoc.org/google.golang.org/protobuf/encoding/protojson package when consuming data from a JSON API, as so:
Response, err := Client.Do(Request)
if err != nil {
log.Error(err)
}
DataByte, err := ioutil.ReadAll(Response.Body)
if err != nil {
log.Error(err)
}
DataProto := Msg{}
err = protojson.Unmarshal(DataByte, &DataProto)
if err != nil {
log.Error(err)
}
What I want to be able to do is to range over the elements of Variable2 to be able to access the SubVariables using the protoreflect API, for which I have tried both:
Array := DataProto.GetVariable2()
for i := range Array {
Element := Array[i]
}
and also:
DataProto.GetVariable2().ProtoReflect().Range(func(fd protoreflect.FieldDescriptor, v protoreflect.Value) {
…
return true})
The first of which fails with error message:
cannot range over DataProto.GetVariable2() (type *SubMsg)
despite the fact DataProto.GetVariable2() returns a variable of type []*Msg_SubMsg.
The second of which fails with:
DataProto.GetVariable2.ProtoReflect undefined (type []*SubMsg has no field or method ProtoReflect)
which suggests that DataProto.GetVariable2() does indeed return an array unlike what is suggested in the error returned in my first approach. This makes sense to me as the protoreflect API only allows this method to be called on a defined message, not an array of those messages. There therefore must be another way of accessing the elements of these arrays to be able to make use of the protoreflect API (for which I have been unsuccessful in finding and answer to on the web thus far).
Could someone help me make sense of these seemingly conflicting error messages? Has anyone had any success iterating over a Protobuf array themselves?
Thanks in advance.
You'll want to treat your Array variable as a List, which means you can't use Range() as in your second attempt. It's close though. Here is a functional example of iterating through and inspecting nested messages:
import (
"testing"
"google.golang.org/protobuf/reflect/protoreflect"
)
func TestVariable2(t *testing.T) {
pb := &Msg{
Variable2: []*Msg_SubMsg{
{
SubVariable1: "string",
SubVariable2: 1,
},
},
}
pbreflect := pb.ProtoReflect()
fd := pbreflect.Descriptor().Fields().ByJSONName("Variable2")
if !fd.IsList() {
t.Fatal("expected a list")
}
l := pbreflect.Get(fd).List()
for i := 0; i < l.Len(); i++ {
// should test that we are now inspecting a message type
li := l.Get(i).Message()
li.Range(func(lifd protoreflect.FieldDescriptor, liv protoreflect.Value) bool {
t.Logf("%v: %v", lifd.Name(), liv)
return true
})
}
}
Run with go test -v ./... if you want to see output

How to make scanning DB rows in Go DRY?

I have a table in the database containing user account information. I have a struct called User defined.
type User struct {
Id uint
Username string
Password string
FirstName string
LastName string
Address1 string
Address2 string
.... a bunch more fields ...
}
For fetching individual user accounts, I have a method defined
func (user *User) GetById(db *sql.DB, id uint) error {
query := `SELECT
...a whole bunch of SQL ...
WHERE id = $1
... more SQL ...
LIMIT 1`
row := db.QueryRow(query, id)
err := row.Scan(
&user.Id,
&user.UserName,
&user.Password,
&user.FirstName,
&user.LastName,
... some 20 more lines of fields read into the struct ...
)
if err != nil {
return err
}
return nil
}
And there are several places in the system where I need to fetch user information as part of a larger query. That is, I am fetching some other type of object, but also a user account related to it.
That means, I have to repeat the whole rows.Scan(&user.Username, &user...) thing over and over again and it takes a whole page and it is error prone and if I ever change the user table structure I would have to change the code in a whole bunch of places. How can I make this more DRY?
Edit: I am not sure why this was marked as a duplicate, but since this edit is required, I will try to explain one more time. I am not asking how to scan a row into a struct. I already know how to do that, as the code above clearly shows. I am asking how to structure the struct scanning code in such a way that I do not have to repeat the same page of scanning code every time I am scanning the same type of struct.
Edit: also, yes, I am aware of sqlstruct and sqlx and similar libraries. I am deliberately avoiding these, because they depend on reflect package with well documented performance issues. And I intend to potentially scan millions of rows using these techniques (not millions of users, but this question extends to other record types).
Edit: so, yes, I know I should write a function. I am not sure what this function should take as arguments and what results it should return. Lets say that the other query I want to accommodate looks like this
SELECT
s.id,
s.name,
... more site fields ...
u.id,
u.username,
... more user fields ...
FROM site AS s
JOIN user AS u ON (u.id = s.user_id)
JOIN some_other_table AS st1 ON (site.id = st1.site_id)
... more SQL ...
And I have a site struct method that embeds a user struct. I don't want to repeat the user scanning code here. I want to call a function that will scan the user portion of the raw into a user struct the same way it does in the user method above.
To eliminate the repetition of the required steps to scan the *sql.Rows structure you could introduce two interfaces. One that describes the already implemented behaviour of *sql.Rows and *sql.Row.
// This interface is already implemented by *sql.Rows and *sql.Row.
type Row interface {
Scan(...interface{}) error
}
And another one that abstracts away the actual scanning step of the row(s).
// have your entity types implement this one
type RowScanner interface {
ScanRow(Row) error
}
An example implementation of the RowScanner interface could look like this:
type User struct {
Id uint
Username string
// ...
}
// Implements RowScanner
func (u *User) ScanRow(r Row) error {
return r.Scan(
&u.Id,
&u.Username,
// ...
)
}
type UserList struct {
Items []*User
}
// Implements RowScanner
func (list *UserList) ScanRow(r Row) error {
u := new(User)
if err := u.ScanRow(r); err != nil {
return err
}
list.Items = append(list.Items, u)
return nil
}
With these interfaces you can now dry your rows-scanning code for all of your types that implement the RowScanner interface by using these two functions.
func queryRows(query string, rs RowScanner, params ...interface{}) error {
rows, err := db.Query(query, params...)
if err != nil {
return err
}
defer rows.Close()
for rows.Next() {
if err := rs.ScanRow(rows); err != nil {
return err
}
}
return rows.Err()
}
func queryRow(query string, rs RowScanner, params ...interface{}) error {
return rs.ScanRow(db.QueryRow(query, params...))
}
// example
ulist := new(UserList)
if err := queryRows(queryString, ulist, arg1, arg2); err != nil {
panic(err)
}
// or
u := new(User)
if err := queryRow(queryString, u, arg1, arg2); err != nil {
panic(err)
}
If you have composite types that you want to scan but you want to avoid having to repeat the enumeration of its elements' fields, then you could introduce a method that returns a type's fields and reuse that method where you need it. For example:
func (u *User) ScannableFields() []interface{} {
return []interface{}{
&u.Id,
&u.Username,
// ...
}
}
func (u *User) ScanRow(r Row) error {
return r.Scan(u.ScannableFields()...)
}
// your other entity type
type Site struct {
Id uint
Name string
// ...
}
func (s *Site) ScannableFields() []interface{} {
return []interface{}{
&p.Id,
&p.Name,
// ...
}
}
// Implements RowScanner
func (s *Site) ScanRow(r Row) error {
return r.Scan(s.ScannableFields()...)
}
// your composite
type UserWithSite struct {
User *User
Site *Site
}
// Implements RowScanner
func (u *UserWithSite) ScanRow(r Row) error {
u.User = new(User)
u.Site = new(Site)
fields := append(u.User.ScannableFields(), u.Site.ScannableFields()...)
return r.Scan(fields...)
}
// retrieve from db
u := new(UserWithSite)
if err := queryRow(queryString, u, arg1, arg2); err != nil {
panic(err)
}

Problem accumulating/appending values in an array using recursion with Go

First of all, this is my first non-dummy program using Go. Any recommendation will be appreciated.
Code description:
I want to retrieve all the information from an API where the information is being paginated. So I want to iterate through all the pages in order to get all the information.
This is what I did so far:
I have the these two functions:
func request(requestData *RequestData) []*ProjectsResponse {
client := &http.Client{
Timeout: time.Second * 10,
}
projects := []*ProjectsResponse{}
innerRequest(client, requestData.URL, projects)
return projects
}
func innerRequest(client *http.Client, URL string, projects []*ProjectsResponse) {
if URL == "" {
return
}
req, err := http.NewRequest("GET", URL, nil)
if err != nil {
log.Printf("Request creation failed with error %s\n", err)
}
req.Header.Add("Private-Token", os.Getenv("API_KEY"))
res, err := client.Do(req)
log.Printf("Executing request: %s", req.URL)
if err != nil {
log.Printf("The HTTP request failed with error %s\n", err)
}
data, _ := ioutil.ReadAll(res.Body)
var response ProjectsResponse
err = json.Unmarshal(data, &response)
if err != nil {
log.Printf("Unmarshalling failed with error %s\n", err)
}
projects = append(projects, &response)
pagingData := getPageInformation(res)
innerRequest(client, pagingData.NextLink, projects)
}
Undesired behavior:
The values in the projects []*ProjectsResponse array are being appended on each iteration of the recursion, but when the recursion ends I get an empty array list. So, somehow after the innerRequests ends, in the project variable inside the request method I get nothing.
Hope somebody and spot my mistake.
Thanks in advance.
I'm guessing that all of your project objects are scoped to the function so they no longer exist when the function ends. I don't think you need your projects to exist before you call innerRequest, so you should probably just have that method return the projects. I think something like this should work...
func innerRequest(client *http.Client, URL string) []*ProjectsResponse {
if URL == "" {
return nil
}
//More code...
pagingData := getPageInformation(res)
return append([]*ProjectsResponse{&response}, innerRequest(client, pagingData.NextLink)...)
}
The confusion lies in the way a slice is handled in Go. Here is the in-depth explanation, but I will abbreviate.
The common misconception is that the slice you pass around is a reference to the slice, which is false. The actual variable you operate on when you handle a slice is known as a slice header. This is a simple struct with a reference to the underlying array under the covers and follows Go's pass by value rules, i.e. it is copied when passed to a function. Thus, if it is not returned, you won't have the updated header.
Returning data from recursion follows a straightforward pattern. Here is a basic example. I'm also including a version that doesn't require a return, but operates on the slice as a reference, which will modify the original. (Note: Passing around slice pointers is generally not considered idiomatic Go)
Playground link: https://play.golang.org/p/v5XeYpH1VlF
// correct way to return from recursion
func addReturn(s []int) []int {
if finalCondition(s) {
return s
}
s = append(s, 1)
return addReturn(s)
}
// using a reference
func addReference(s *[]int) {
if finalCondition(*s) {
return
}
*s = append(*s, 1)
addReference(s)
}
// whatever terminates the recursion
func finalCondition(s []int) bool {
if len(s) >= 10 {
return true
}
return false
}

In Go, how to Unmarshal bson byte[] data into an array of structs?

What's the best way to Unmarshal bson byte[] data into an array of structs, when the array of structs is passed into an interface{} parameter?
For demonstration purposes, in the following code, I use bson.Marshal() on the inStructArr to get the byte[] type of data. This is so I can use bson.Unmarshal(...) to pipe into the outStructArr.
import "gopkg.in/mgo.v2/bson"
type User struct {
Name string
}
func DecodeArrData(inStructArr, outStructArr interface{}) {
inStructArrData, _ := bson.Marshal(inStructArr)
bson.Unmarshal(inStructArrData, outStructArr) // <-- Error happens here
// What's the right way of accomplishing this?
}
func Main() {
outUsers := &[]User{}
inUsers := []User{User{"A"}, User{"B"}}
DecodeArrData(inUsers, outUsers)
}
When I do this, the error-message I get is: Unsupported document type for unmarshalling: []User. What's the right way of doing this?
Thanks in advance!
The Marshal and Unmarshal functions work with BSON documents, not BSON arrays.
Wrap the slices in a struct to provide the document expected by the package:
func DecodeArrData(inStructArr, outStructArr interface{}) error {
in := struct{ Data interface{} }{Data: inStructArr}
inStructArrData, err := bson.Marshal(in)
if err != nil {
return err
}
var out struct{ Data bson.Raw }
if err := bson.Unmarshal(inStructArrData, &out); err != nil {
return err
}
return out.Data.Unmarshal(outStructArr)
}
If you are willing to take advantage of an undocumented feature of the Marshal function and add some BSON format knowledge to your application, then you can omit the wrapper.
The undocumented feature of Marshal is that it encodes slices as BSON arrays. The BSON array can be decoded using a bson.Raw value with Kind set the the BSON code for arrays (the value 4) and Data set to the array data:
func DecodeArrData(inStructArr, outStructArr interface{}) error {
inStructArrData, err := bson.Marshal(inStructArr)
if err != nil {
return err
}
raw := bson.Raw{Kind: 4, Data: inStructArrData}
return raw.Unmarshal(outStructArr)
}

Unmarshalling top-level JSON array in Go

I'm learning Go by writing a simple http server and I need to handle some JSON responses.
With an object response, I can unmarshal it idiomatically with 2 lines of code:
structResult := Foo{}
json.Unmarshal(structBody, &structResult)
I don't know how to do the same for an array response (see the example below). Is there a way to specify (possibly via json tag) that top-level array should go into a given struct field?
package main
import "fmt"
import "encoding/json"
type Foo struct {
Id uint64 `json:"id"`
Name string `json:"name"`
}
type BaseResult struct {
Error string `json:"error"`
}
type FooResult struct {
BaseResult
Foos []Foo
}
func main() {
// Simple and works.
structBody := []byte(`{"id": 1,"name": "foo"}`)
structResult := Foo{}
json.Unmarshal(structBody, &structResult)
fmt.Printf("%#v\n", structResult)
// Doesn't work.
arrayBody := []byte(`[{"id": 1,"name": "foo"},{"id": 2,"name": "bar"},{"id": 3,"name": "foobar"}]`)
arrayResult := FooResult{}
json.Unmarshal(arrayBody, &arrayResult)
fmt.Printf("%#v\n", arrayResult)
}
I know I could make FooResult an array:
type FooResult []Foo
but then I lose the ability to specify base object which I would like to use to store error message and such. I also know that I can unmarshal into &fooResult.Foos directly, but I want the code to work with both objects and arrays.
UPDATE
Implementing UnmarshalJSON as suggested by #dyoo partially solves my problem, but I was hoping that I could use BaseResult to store parse error in case JSON has a different structure:
arrayBody := []byte(`{"error": "foo"}`)
arrayResult := FooResult{}
json.Unmarshal(arrayBody, &arrayResult)
fmt.Printf("%#v\n", arrayResult)
Of course I could implement more complex logic inside UnmarshalJSON - but isn't there a simpler way to do it?
You can implement the json.Unmarshaler interface in your FooResult, to customize exactly how it responds to unmarshaling. (Similarly, there's a json.Marshaler interface.)
Add:
func (f *FooResult) UnmarshalJSON(bs []byte) error {
return json.Unmarshal(bs, &f.Foos)
}
after which your code should otherwise work. http://play.golang.org/p/oMdoB2e-rB
You might try something like:
func (f *FooResult) UnmarshalJSON(bs []byte) error {
err1 := json.Unmarshal(bs, &f.BaseResult)
err2 := json.Unmarshal(bs, &f.Foos)
if err1 != nil && err2 != nil {
// Arbitrarily choose an error.
return err1
}
return nil
}
although even this is beginning to look dubious. Handling union type results is not quite what the json library is designed to handle automatically for you. You'll need to explicitly code the coercion logic if your JSON has dynamic type.
See: How to unmarshall an array of different types correctly? and http://blog.golang.org/json-and-go for related issues.
Just specify Foos when you Unmarshal
package main
import "fmt"
import "encoding/json"
type Foo struct {
Id uint64 `json:"id"`
Name string `json:"name"`
}
type BaseResult struct {
Error string `json:"error"`
}
type FooResult struct {
BaseResult
Foos []Foo
}
func main() {
// Simple and works.
structBody := []byte(`{"id": 1,"name": "foo"}`)
structResult := Foo{}
json.Unmarshal(structBody, &structResult)
fmt.Printf("%#v\n", structResult)
// Doesn't work.
arrayBody := []byte(`[{"id": 1,"name": "foo"},{"id": 2,"name": "bar"},{"id": 3,"name": "foobar"}]`)
arrayResult := FooResult{}
if err := json.Unmarshal(arrayBody, &arrayResult.Foos); err != nil {
arrayResult.BaseResult.Error = string(arrayBody)
}
fmt.Printf("%#v\n", arrayResult)
}

Resources