Problem accumulating/appending values in an array using recursion with Go - arrays

First of all, this is my first non-dummy program using Go. Any recommendation will be appreciated.
Code description:
I want to retrieve all the information from an API where the information is being paginated. So I want to iterate through all the pages in order to get all the information.
This is what I did so far:
I have the these two functions:
func request(requestData *RequestData) []*ProjectsResponse {
client := &http.Client{
Timeout: time.Second * 10,
}
projects := []*ProjectsResponse{}
innerRequest(client, requestData.URL, projects)
return projects
}
func innerRequest(client *http.Client, URL string, projects []*ProjectsResponse) {
if URL == "" {
return
}
req, err := http.NewRequest("GET", URL, nil)
if err != nil {
log.Printf("Request creation failed with error %s\n", err)
}
req.Header.Add("Private-Token", os.Getenv("API_KEY"))
res, err := client.Do(req)
log.Printf("Executing request: %s", req.URL)
if err != nil {
log.Printf("The HTTP request failed with error %s\n", err)
}
data, _ := ioutil.ReadAll(res.Body)
var response ProjectsResponse
err = json.Unmarshal(data, &response)
if err != nil {
log.Printf("Unmarshalling failed with error %s\n", err)
}
projects = append(projects, &response)
pagingData := getPageInformation(res)
innerRequest(client, pagingData.NextLink, projects)
}
Undesired behavior:
The values in the projects []*ProjectsResponse array are being appended on each iteration of the recursion, but when the recursion ends I get an empty array list. So, somehow after the innerRequests ends, in the project variable inside the request method I get nothing.
Hope somebody and spot my mistake.
Thanks in advance.

I'm guessing that all of your project objects are scoped to the function so they no longer exist when the function ends. I don't think you need your projects to exist before you call innerRequest, so you should probably just have that method return the projects. I think something like this should work...
func innerRequest(client *http.Client, URL string) []*ProjectsResponse {
if URL == "" {
return nil
}
//More code...
pagingData := getPageInformation(res)
return append([]*ProjectsResponse{&response}, innerRequest(client, pagingData.NextLink)...)
}

The confusion lies in the way a slice is handled in Go. Here is the in-depth explanation, but I will abbreviate.
The common misconception is that the slice you pass around is a reference to the slice, which is false. The actual variable you operate on when you handle a slice is known as a slice header. This is a simple struct with a reference to the underlying array under the covers and follows Go's pass by value rules, i.e. it is copied when passed to a function. Thus, if it is not returned, you won't have the updated header.
Returning data from recursion follows a straightforward pattern. Here is a basic example. I'm also including a version that doesn't require a return, but operates on the slice as a reference, which will modify the original. (Note: Passing around slice pointers is generally not considered idiomatic Go)
Playground link: https://play.golang.org/p/v5XeYpH1VlF
// correct way to return from recursion
func addReturn(s []int) []int {
if finalCondition(s) {
return s
}
s = append(s, 1)
return addReturn(s)
}
// using a reference
func addReference(s *[]int) {
if finalCondition(*s) {
return
}
*s = append(*s, 1)
addReference(s)
}
// whatever terminates the recursion
func finalCondition(s []int) bool {
if len(s) >= 10 {
return true
}
return false
}

Related

errno to Go errors

I'm trying to issue KVM ioctls from Go code. At the moment I have something as follows:
func (vm *Vm) RegisterIrqFd(efd *EventFd, gsi uint32) error {
irqfd := (*C.struct_kvm_irqfd)(C.calloc(1, C.sizeof_struct_kvm_irqfd))
defer C.free(unsafe.Pointer(irqfd))
irqfd.fd = C.uint(efd.Fd())
irqfd.gsi = C.uint(gsi)
if _, err := sysutils.Ioctl(vm.Fd(), C.KVM_IRQFD, uintptr(unsafe.Pointer(irqfd))); err != nil {
return fmt.Errorf("RegisterIrqFd failed: %v", err)
}
return nil
}
The Ioctl function is implemented as follows :
func Ioctl(fd uintptr, cmd C.uint, arg uintptr) (uintptr, error) {
ret, _, errno := syscall.Syscall(
syscall.SYS_IOCTL,
fd,
uintptr(cmd),
uintptr(arg),
)
if int64(ret) == -1 {
return ret, ErrnoToErr(errno)
}
return ret, nil
}
And the ErrnoToErr function is implemented as follows :
func ErrnoToErr(errno syscall.Errno) error {
return fmt.Errorf("%v", errno.Error())
}
Inside GetSupportedCpuid, the argument for the ioctl, cpuid is allocated using C.calloc. If this were C code cpuid could just be allocated on the stack. Is there any way I can get around allocating using calloc and using free? Any alternatives that would be more idiomatic?
Would there be a way to have ErrnoToErr return a go error type corresponding to the errno? Currently it uses the (e Errno) Error() inside syscall/syscall_unix.go; so it just dereferences a list of strings and returns the string. It would be nice to have a type, so that I can test for specific errors in my unit tests.
Replying just to the updated example, we could do this:
func (vm *Vm) RegisterIrqFd(efd *EventFd, gsi uint32) error {
irqfd := C.struct_kvm_irqfd{}
irqfd.fd = C.uint(efd.Fd())
irqfd.gsi = C.uint(gsi)
if _, err := sysutils.Ioctl(vm.Fd(), C.KVM_IRQFD, uintptr(unsafe.Pointer(&irqfd))); err != nil {
return fmt.Errorf("RegisterIrqFd failed: %v", err)
}
return nil
}
I think this is safe, but we might find that irqfd escapes to heap anyway since we take its address here, in which case there's not much savings. Even so, as long as this doesn't violate Go pointer rules (I don't think it does) it's a lot nicer to read.
Read the documentation syscall.Errno
So convertion may be performed like this
func ErrnoToErr(errno syscall.Errno) error {
if errno != 0 {
return errno
}
return nil
}
For equality it only states
Errno values can be tested against error values from the os package
using errors.Is.
The question of comparing errors was already considered here How to compare Go errors

Range over elements of a protobuf array in Go

I have a Protobuf structure defined as so in my .proto file:
message Msg{
message SubMsg {
string SubVariable1 = 1;
int32 SubVariable2 = 2;
...
}
string Variable1 = 1;
repeated SubMsg Variable2 = 2;
...
}
I pull data into this structure using the https://godoc.org/google.golang.org/protobuf/encoding/protojson package when consuming data from a JSON API, as so:
Response, err := Client.Do(Request)
if err != nil {
log.Error(err)
}
DataByte, err := ioutil.ReadAll(Response.Body)
if err != nil {
log.Error(err)
}
DataProto := Msg{}
err = protojson.Unmarshal(DataByte, &DataProto)
if err != nil {
log.Error(err)
}
What I want to be able to do is to range over the elements of Variable2 to be able to access the SubVariables using the protoreflect API, for which I have tried both:
Array := DataProto.GetVariable2()
for i := range Array {
Element := Array[i]
}
and also:
DataProto.GetVariable2().ProtoReflect().Range(func(fd protoreflect.FieldDescriptor, v protoreflect.Value) {
…
return true})
The first of which fails with error message:
cannot range over DataProto.GetVariable2() (type *SubMsg)
despite the fact DataProto.GetVariable2() returns a variable of type []*Msg_SubMsg.
The second of which fails with:
DataProto.GetVariable2.ProtoReflect undefined (type []*SubMsg has no field or method ProtoReflect)
which suggests that DataProto.GetVariable2() does indeed return an array unlike what is suggested in the error returned in my first approach. This makes sense to me as the protoreflect API only allows this method to be called on a defined message, not an array of those messages. There therefore must be another way of accessing the elements of these arrays to be able to make use of the protoreflect API (for which I have been unsuccessful in finding and answer to on the web thus far).
Could someone help me make sense of these seemingly conflicting error messages? Has anyone had any success iterating over a Protobuf array themselves?
Thanks in advance.
You'll want to treat your Array variable as a List, which means you can't use Range() as in your second attempt. It's close though. Here is a functional example of iterating through and inspecting nested messages:
import (
"testing"
"google.golang.org/protobuf/reflect/protoreflect"
)
func TestVariable2(t *testing.T) {
pb := &Msg{
Variable2: []*Msg_SubMsg{
{
SubVariable1: "string",
SubVariable2: 1,
},
},
}
pbreflect := pb.ProtoReflect()
fd := pbreflect.Descriptor().Fields().ByJSONName("Variable2")
if !fd.IsList() {
t.Fatal("expected a list")
}
l := pbreflect.Get(fd).List()
for i := 0; i < l.Len(); i++ {
// should test that we are now inspecting a message type
li := l.Get(i).Message()
li.Range(func(lifd protoreflect.FieldDescriptor, liv protoreflect.Value) bool {
t.Logf("%v: %v", lifd.Name(), liv)
return true
})
}
}
Run with go test -v ./... if you want to see output

Golang best practices: empty array response or error?

What are best practices in terms of error handling for a function that accepts slice of objects and returns another slice of objects (ideally of same length as input array) along with error as follows:
func ([]interface{}) ([]interface{}, error)
One way is whenever you get an error for processing one of the objects in a slice, you return an error response, but that way at the receiving function, if you don't discard all slice elements, error response becomes of little use merely telling us that processing of one of the elements or all failed. Another way is you return an error when none of the elements get processed but again this is of little use I feel. One more way is you don't include error as return object and instead with every slice element struct, have it's own error object as a composite so you can send elementwise error as output.
The best way obviously depends on the particular scenario, however, I want to know if there are any best practices people follow or any design patterns around this problem.
PS: This was one of the closest questions, however since its accepting single object as input, not very relevant:
Return empty array or error
... a function that accepts [slice of interface representing an] array of objects and returns another [slice of interface representing an] array of objects along with error ...
You have not told us enough to go on.
Does the returned slice actually have anything to do with the parameter slice?
If so, what relationship do they have? For instance, perhaps the returned slice should be half the size of the input slice, and an error occurs if and only if the number of input objects is odd, in which case the last input object has been ignored.
Must inputs be processed in order, or will they be processed in parallel?
One more way is you don't include error as return object and instead with every array object struct, have it's own error object as a composite so you can send elementwise error as output.
This is probably a wise approach if the outputs are one-to-one with the inputs and you intend to handle them in parallel and/or continue processing the remaining inputs upon reaching one bad one. Equivalently, you can have the output slice include an error.
It's really very problem-dependent.
Edit: consider, e.g., the following (which I don't claim is good, mind you):
const maxWorkers = 10 // tunable
// Process a slice of T's in parallel. The results are either an
// R for each T, or an error. Caller provides the actual function
// f(T), which returns R + error (an empty/zero R for error).
func ProcessInParallel(input []T, f func(T) (R, error)) ([]interface{}, error) {
// Make one channel for sending work to workers,
// and one for receiving results from workers.
type Todo struct {
i int // the index of the item
item T // the item to work on
}
workChan := make(chan Todo)
type Done struct {
i int // the index of the item worked on
r R // result, if we have one
e error // error, if we have one
}
doneChan := make(chan Done)
// Spin off workers: maxWorkers or len(input),
// whichever is smaller.
n := len(input)
if n > maxWorkers {
n = maxWorkers
}
var wg sync.WaitGroup
for i := 0; i < n; i++ {
wg.Add(1)
go func(i int) {
for todo := range workChan {
i := todo.i
r, err := f(input[i])
doneChan <- Done{i, r, err}
}
wg.Done()
}(i)
}
// Close doneChan when all workers finish.
go func() {
wg.Wait()
close(doneChan)
}()
// Hand out work to workers (then close work channel).
go func() {
for i := range input {
workChan <- Todo{i, input[i]}
}
close(workChan)
}()
// Collect results.
var anyErr error
ret := make([]interface{}, len(input))
for done := range doneChan {
i := done.i
r, err := done.r, done.e
if err != nil {
anyErr = err
ret[i] = err
} else {
ret[i] = r
}
}
return ret, anyErr
}
This has an overall error return, and it returns a slice of interface{}. This means you can immediately tell if everything worked. However, it's kind of annoying to use:
ret, err := ProcessInParallel(arg, f)
if err != nil {
fmt.Println("some inputs failed")
for i := range ret {
if e, ok := ret[i].(error); ok {
fmt.Printf("%d: failed: %v\n", i, e)
} else {
fmt.Printf("%d: %s\n", i, ret[i].(R))
}
}
} else {
fmt.Println("all inputs were good")
for i := range ret {
fmt.Printf("%d: %s\n", i, ret[i].(R))
}
}
Why bother with the all-error summary?
Instead, we could have ProcessInParallel return []R, []error, for instance, or—probably better—use a simple error interface return value to store a MultiError as Cerise Limón suggested in a comment:
ret, err := ProcessInParallel(arg, f)
if err != nil {
if merr, ok := err.(datastore.MultiError); ok {
// merr[i] indicates the various failed items
// any ret[i] for which merr[i] is nil is OK
}
} else {
// all ret[i] are ok
}
A working example that doesn't use MultiError is here.
A working example that does use MultiError is here.
While Go supports multiple return values, when one of the return types is an error, it is meant to process either error or the other return values and not both. It means that when error is not nil, the other return values has no specific meaning and should not be processed.
In your case, I'd personally prefer to use an iterator pattern, similar to what is implemented for database/sql.Rows, such that:
func X(values []interface{}) *Result
The Result would hold all processed slice elements associated with their errors. Somewhere in the code I would write something like this:
result := X(values)
for result.Next() {
if err := result.Err(); err != nil {
// Handle the err for this specific element.
// Whether continue or fail the whole process.
}
v := result.Cur()
// Process current element.
}

How to convert array of errors to JSON in Go

I have an array of errors (type of error), but when I try to return the client in JSON format, it arrives empty.
It was created this way:
var (
ErrEmptyName = errors.New("Nome não pode ser vázio")
ErrInvalidType = errors.New("Tipo de pessoa inválido")
)
func (p Person) Validate() []error {
var errors []error
if govalidator.IsNull(p.Name) {
errors = append(errors, ErrEmptyName)
}
if p.Type != "F" && p.Type != "J" {
errors = append(errors, ErrInvalidType)
}
return errors
)
In my Controller:
err := person.Validate()
c.JSON(422, gin.H{"errors" : err})
My Output:
{"errors":"[{}]"}
The error type is an interface with a single Error() method, but it is not special to the json package (the Error() method is not called on it).
However, error values may hold values of static types which may be nicely marshalable, or they may define their own marshaling logic by implementing json.Marshaler. Simply converting an error to string by calling its Error() method means we're not honoring the custom marshaling logic.
So I would suggest to create our own error slice type, on which we can implement our marshaling logic which should be:
check if the error value implements json.Marshaler, and if so, let it marshal itself
else as a fallback case call error.Error() to "obtain" a string which can easily be marshaled
This is how it could look like:
type JSONErrs []error
func (je JSONErrs) MarshalJSON() ([]byte, error) {
res := make([]interface{}, len(je))
for i, e := range je {
if _, ok := e.(json.Marshaler); ok {
res[i] = e // e knows how to marshal itself
} else {
res[i] = e.Error() // Fallback to the error string
}
}
return json.Marshal(res)
}
And this is how you can use it:
err := person.Validate()
c.JSON(422, gin.H{"errors" : JSONErrs(err)})
Let's test our JSONErrs. We're also use a custom error type which implements custom marshaling logic:
type MyErr struct{ line int }
func (me MyErr) Error() string { return "Invalid input!" }
func (me MyErr) MarshalJSON() ([]byte, error) {
return json.Marshal(
struct {
Type, Error string
AtLine int
}{"MyErr", me.Error(), me.line})
}
And the test code:
errs := []error{
errors.New("first"),
errors.New("second"),
MyErr{16},
}
data, err := json.Marshal(JSONErrs(errs))
fmt.Println(string(data), err)
Output (try it on the Go Playground):
["first","second",{"Type":"MyErr","Error":"Invalid input!","AtLine":16}] <nil>
An error type is an interface which must implement an Error() method that returns an error message as a string. This is defined here: https://golang.org/pkg/builtin/#error. The reason why the error type is an interface, is to allow for error types that can be type casted to retrieve more detailed information.
Functions like fmt.Println and log.Println automatically resolves error types to display the message from Error(), the JSON library however, does not. The simplest way to get around this problem is by converting the error messages in []error to a []string and encoding that.
Here's some example code to do that with a for loop.
errs := person.Validate()
strErrors := make([]string, len(errs))
for i, err := range errs {
strErrors[i] = err.Error()
}
c.JSON(422, gin.H{"errors" : strErrors})

Count similar array value

I'm trying to learn Go (or Golang) and can't seem to get it right. I have 2 texts files, each containing a list of words. I'm trying to count the amount of words that are present in both files.
Here is my code so far :
package main
import (
"fmt"
"log"
"net/http"
"bufio"
)
func stringInSlice(str string, list []string) bool {
for _, v := range list {
if v == str {
return true
}
}
return false
}
func main() {
// Texts URL
var list = "https://gist.githubusercontent.com/alexcesaro/c9c47c638252e21bd82c/raw/bd031237a56ae6691145b4df5617c385dffe930d/list.txt"
var url1 = "https://gist.githubusercontent.com/alexcesaro/4ebfa5a9548d053dddb2/raw/abb8525774b63f342e5173d1af89e47a7a39cd2d/file1.txt"
//Create storing arrays
var buffer [2000]string
var bufferUrl1 [40000]string
// Set a sibling counter
var sibling = 0
// Read and store text files
wordList, err := http.Get(list)
if err != nil {
log.Fatalf("Error while getting the url : %v", err)
}
defer wordList.Body.Close()
wordUrl1, err := http.Get(url1)
if err != nil {
log.Fatalf("Error while getting the url : %v", err)
}
defer wordUrl1.Body.Close()
streamList := bufio.NewScanner(wordList.Body)
streamUrl1 := bufio.NewScanner(wordUrl1.Body)
streamList.Split(bufio.ScanLines)
streamUrl1.Split(bufio.ScanLines)
var i = 0;
var j = 0;
//Fill arrays with each lines
for streamList.Scan() {
buffer[i] = streamList.Text()
i++
}
for streamUrl1.Scan() {
bufferUrl1[j] = streamUrl1.Text()
j++
}
//ERROR OCCURRING HERE :
// This code if i'm not wrong is supposed to compare through all the range of bufferUrl1 -> bufferUrl1 values with buffer values, then increment sibling and output FIND
for v := range bufferUrl1{
if stringInSlice(bufferUrl1, buffer) {
sibling++
fmt.Println("FIND")
}
}
// As a testing purpose thoses lines properly paste both array
// fmt.Println(buffer)
// fmt.Println(bufferUrl1)
}
But right now, my build doesn't even succeed. I'm only greeted with this message:
.\hello.go:69: cannot use bufferUrl1 (type [40000]string) as type string in argument to stringInSlice
.\hello.go:69: cannot use buffer (type [2000]string) as type []string in argument to stringInSlice
bufferUrl1 is an array: [4000]string. You meant to use v (each
string in bufferUrl1). But in fact, you meant to use the second
variable—the first variable is the index which is ignored in the code
below using _.
type [2000]string is different from []string. In Go, arrays and slices are not the same. Read Go Slices: usage and internals. I've changed both variable declarations to use slices with the same initial length using make.
These are changes you need to make to compile.
Declarations:
// Create storing slices
buffer := make([]string, 2000)
bufferUrl1 := make([]string, 40000)
and the loop on Line 69:
for _, s := range bufferUrl1 {
if stringInSlice(s, buffer) {
sibling++
fmt.Println("FIND")
}
}
As a side-note, consider using a map instead of a slice for buffer for more efficient lookup instead of looping through the list in stringInSlice.
https://play.golang.org/p/UcaSVwYcIw has the fix for the comments below (you won't be able to make HTTP requests from the Playground).

Resources