I have a Protobuf structure defined as so in my .proto file:
message Msg{
message SubMsg {
string SubVariable1 = 1;
int32 SubVariable2 = 2;
...
}
string Variable1 = 1;
repeated SubMsg Variable2 = 2;
...
}
I pull data into this structure using the https://godoc.org/google.golang.org/protobuf/encoding/protojson package when consuming data from a JSON API, as so:
Response, err := Client.Do(Request)
if err != nil {
log.Error(err)
}
DataByte, err := ioutil.ReadAll(Response.Body)
if err != nil {
log.Error(err)
}
DataProto := Msg{}
err = protojson.Unmarshal(DataByte, &DataProto)
if err != nil {
log.Error(err)
}
What I want to be able to do is to range over the elements of Variable2 to be able to access the SubVariables using the protoreflect API, for which I have tried both:
Array := DataProto.GetVariable2()
for i := range Array {
Element := Array[i]
}
and also:
DataProto.GetVariable2().ProtoReflect().Range(func(fd protoreflect.FieldDescriptor, v protoreflect.Value) {
…
return true})
The first of which fails with error message:
cannot range over DataProto.GetVariable2() (type *SubMsg)
despite the fact DataProto.GetVariable2() returns a variable of type []*Msg_SubMsg.
The second of which fails with:
DataProto.GetVariable2.ProtoReflect undefined (type []*SubMsg has no field or method ProtoReflect)
which suggests that DataProto.GetVariable2() does indeed return an array unlike what is suggested in the error returned in my first approach. This makes sense to me as the protoreflect API only allows this method to be called on a defined message, not an array of those messages. There therefore must be another way of accessing the elements of these arrays to be able to make use of the protoreflect API (for which I have been unsuccessful in finding and answer to on the web thus far).
Could someone help me make sense of these seemingly conflicting error messages? Has anyone had any success iterating over a Protobuf array themselves?
Thanks in advance.
You'll want to treat your Array variable as a List, which means you can't use Range() as in your second attempt. It's close though. Here is a functional example of iterating through and inspecting nested messages:
import (
"testing"
"google.golang.org/protobuf/reflect/protoreflect"
)
func TestVariable2(t *testing.T) {
pb := &Msg{
Variable2: []*Msg_SubMsg{
{
SubVariable1: "string",
SubVariable2: 1,
},
},
}
pbreflect := pb.ProtoReflect()
fd := pbreflect.Descriptor().Fields().ByJSONName("Variable2")
if !fd.IsList() {
t.Fatal("expected a list")
}
l := pbreflect.Get(fd).List()
for i := 0; i < l.Len(); i++ {
// should test that we are now inspecting a message type
li := l.Get(i).Message()
li.Range(func(lifd protoreflect.FieldDescriptor, liv protoreflect.Value) bool {
t.Logf("%v: %v", lifd.Name(), liv)
return true
})
}
}
Run with go test -v ./... if you want to see output
I am new to Go and want to create and initialise an struct array in go. My code is like this
type node struct {
name string
children map[string]int
}
cities:= []node{node{}}
for i := 0; i<47 ;i++ {
cities[i].name=strconv.Itoa(i)
cities[i].children=make(map[string]int)
}
I get the following error:
panic: runtime error: index out of range
goroutine 1 [running]:
panic(0xa6800, 0xc42000a080)
Please help. TIA :)
You are initializing cities as a slice of nodes with one element (an empty node).
You can initialize it to a fixed size with cities := make([]node,47), or you could initialize it to an empty slice, and append to it:
cities := []node{}
for i := 0; i<47 ;i++ {
n := node{name: strconv.Itoa(i), children: map[string]int{}}
cities = append(cities,n)
}
I'd definitely recommend reading this article if you are a bit shaky on how slices work.
This worked for me
type node struct {
name string
children map[string]int
}
cities:=[]*node{}
city:=new(node)
city.name=strconv.Itoa(0)
city.children=make(map[string]int)
cities=append(cities,city)
for i := 1; i<47 ;i++ {
city=new(node)
city.name=strconv.Itoa(i)
city.children=make(map[string]int)
cities=append(cities,city)
}
I have an array of strings, and I'd like to exclude values that start in foo_ OR are longer than 7 characters.
I can loop through each element, run the if statement, and add it to a slice along the way. But I was curious if there was an idiomatic or more golang-like way of accomplishing that.
Just for example, the same thing might be done in Ruby as
my_array.select! { |val| val !~ /^foo_/ && val.length <= 7 }
There is no one-liner as you have it in Ruby, but with a helper function you can make it almost as short.
Here's our helper function that loops over a slice, and selects and returns only the elements that meet a criteria captured by a function value:
func filter(ss []string, test func(string) bool) (ret []string) {
for _, s := range ss {
if test(s) {
ret = append(ret, s)
}
}
return
}
Starting with Go 1.18, we can write it generic so it will work with all types, not just string:
func filter[T any](ss []T, test func(T) bool) (ret []T) {
for _, s := range ss {
if test(s) {
ret = append(ret, s)
}
}
return
}
Using this helper function your task:
ss := []string{"foo_1", "asdf", "loooooooong", "nfoo_1", "foo_2"}
mytest := func(s string) bool { return !strings.HasPrefix(s, "foo_") && len(s) <= 7 }
s2 := filter(ss, mytest)
fmt.Println(s2)
Output (try it on the Go Playground, or the generic version: Go Playground):
[asdf nfoo_1]
Note:
If it is expected that many elements will be selected, it might be profitable to allocate a "big" ret slice beforehand, and use simple assignment instead of the append(). And before returning, slice the ret to have a length equal to the number of selected elements.
Note #2:
In my example I chose a test() function which tells if an element is to be returned. So I had to invert your "exclusion" condition. Obviously you may write the helper function to expect a tester function which tells what to exclude (and not what to include).
Have a look at robpike's filter library. This would allow you to do:
package main
import (
"fmt"
"strings"
"filter"
)
func isNoFoo7(a string) bool {
return ! strings.HasPrefix(a, "foo_") && len(a) <= 7
}
func main() {
a := []string{"test", "some_other_test", "foo_etc"}
result := Choose(a, isNoFoo7)
fmt.Println(result) // [test]
}
Interestingly enough the README.md by Rob:
I wanted to see how hard it was to implement this sort of thing in Go, with as nice an API as I could manage. It wasn't hard.
Having written it a couple of years ago, I haven't had occasion to use it once. Instead, I just use "for" loops.
You shouldn't use it either.
So the most idiomatic way according to Rob would be something like:
func main() {
a := []string{"test", "some_other_test", "foo_etc"}
nofoos := []string{}
for i := range a {
if(!strings.HasPrefix(a[i], "foo_") && len(a[i]) <= 7) {
nofoos = append(nofoos, a[i])
}
}
fmt.Println(nofoos) // [test]
}
This style is very similar, if not identical, to the approach any C-family language takes.
Today, I stumbled on a pretty idiom that surprised me. If you want to filter a slice in place without allocating, use two slices with the same backing array:
s := []T{
// the input
}
s2 := s
s = s[:0]
for _, v := range s2 {
if shouldKeep(v) {
s = append(s, v)
}
}
Here's a specific example of removing duplicate strings:
s := []string{"a", "a", "b", "c", "c"}
s2 := s
s = s[:0]
var last string
for _, v := range s2 {
if len(s) == 0 || v != last {
last = v
s = append(s, v)
}
}
If you need to keep both slices, simply replace s = s[:0] with s = nil or s = make([]T, 0, len(s)), depending on whether you want append() to allocate for you.
There are a couple of nice ways to filter a slice without allocations or new dependencies. Found in the Go wiki on Github:
Filter (in place)
n := 0
for _, x := range a {
if keep(x) {
a[n] = x
n++
}
}
a = a[:n]
And another, more readable, way:
Filtering without allocating
This trick uses the fact that a slice shares the same backing array
and capacity as the original, so the storage is reused for the
filtered slice. Of course, the original contents are modified.
b := a[:0]
for _, x := range a {
if f(x) {
b = append(b, x)
}
}
For elements which must be garbage collected, the following code can
be included afterwards:
for i := len(b); i < len(a); i++ {
a[i] = nil // or the zero value of T
}
One thing I'm not sure about is whether the first method needs clearing (setting to nil) the items in slice a after index n, like they do in the second method.
EDIT: the second way is basically what MicahStetson described in his answer. In my code I use a function similar to the following, which is probably as good as it gets in terms on performance and readability:
func filterSlice(slice []*T, keep func(*T) bool) []*T {
newSlice := slice[:0]
for _, item := range slice {
if keep(item) {
newSlice = append(newSlice, item)
}
}
// make sure discarded items can be garbage collected
for i := len(newSlice); i < len(slice); i++ {
slice[i] = nil
}
return newSlice
}
Note that if items in your slice are not pointers and don't contain pointers you can skip the second for loop.
There isn't an idiomatic way you can achieve the same expected result in Go in one single line as in Ruby, but with a helper function you can obtain the same expressiveness as in Ruby.
You can call this helper function as:
Filter(strs, func(v string) bool {
return strings.HasPrefix(v, "foo_") // return foo_testfor
}))
Here is the whole code:
package main
import "strings"
import "fmt"
// Returns a new slice containing all strings in the
// slice that satisfy the predicate `f`.
func Filter(vs []string, f func(string) bool) []string {
vsf := make([]string, 0)
for _, v := range vs {
if f(v) && len(v) > 7 {
vsf = append(vsf, v)
}
}
return vsf
}
func main() {
var strs = []string{"foo1", "foo2", "foo3", "foo3", "foo_testfor", "_foo"}
fmt.Println(Filter(strs, func(v string) bool {
return strings.HasPrefix(v, "foo_") // return foo_testfor
}))
}
And the running example: Playground
you can use the loop as you did and wrap it to a utils function for reuse.
For multi-datatype support, copy-paste will be a choice. Another choice is writing a generating tool.
And final option if you want to use lib, you can take a look on https://github.com/ledongthuc/goterators#filter that I created to reuse aggregate & transform functions.
It requires the Go 1.18 to use that support generic + dynamic type you want to use with.
filteredItems, err := Filter(list, func(item int) bool {
return item % 2 == 0
})
filteredItems, err := Filter(list, func(item string) bool {
return item.Contains("ValidWord")
})
filteredItems, err := Filter(list, func(item MyStruct) bool {
return item.Valid()
})
It also supports Reduce in case you want to optimize the way you select.
Hope it's useful with you!
"Select Elements from Array" is also commonly called a filter function. There's no such thing in go. There are also no other "Collection Functions" such as map or reduce. For the most idiomatic way to get the desired result, I find https://gobyexample.com/collection-functions a good reference:
[...] in Go it’s common to provide collection functions if and when they are specifically needed for your program and data types.
They provide an implementation example of the filter function for strings:
func Filter(vs []string, f func(string) bool) []string {
vsf := make([]string, 0)
for _, v := range vs {
if f(v) {
vsf = append(vsf, v)
}
}
return vsf
}
However, they also say, that it's often ok to just inline the function:
Note that in some cases it may be clearest to just inline the
collection-manipulating code directly, instead of creating and calling
a helper function.
In general, golang tries to only introduce orthogonal concepts, meaning that when you can solve a problem one way, there shouldn't be too many more ways to solve it. This adds simplicity to the language by only having a few core concepts, such that not every developer uses a different subset of the language.
Take a look at this library: github.com/thoas/go-funk
It provides an implementation of a lot of life-saving idioms in Go (including filtering of elements in array for instance).
r := funk.Filter([]int{1, 2, 3, 4}, func(x int) bool {
return x%2 == 0
}
Here is an elegant example of both Fold and Filter that uses recursion to accomplish filtering. FoldRight is also generally useful. It is not stack safe but could be made so with trampolining. Once Golang has generics it can be entirely generalized for any 2 types:
func FoldRightStrings(as, z []string, f func(string, []string) []string) []string {
if len(as) > 1 {//Slice has a head and a tail.
h, t := as[0], as[1:len(as)]
return f(h, FoldRightStrings(t, z, f))
} else if len(as) == 1 {//Slice has a head and an empty tail.
h := as[0]
return f(h, FoldRightStrings([]string{}, z, f))
}
return z
}
func FilterStrings(as []string, p func(string) bool) []string {
var g = func(h string, accum []string) []string {
if p(h) {
return append(accum, h)
} else {
return accum
}
}
return FoldRightStrings(as, []string{}, g)
}
Here is an example of its usage to filter out all the strings with length < 8
var p = func(s string) bool {
if len(s) < 8 {
return true
} else {
return false
}
}
FilterStrings([]string{"asd","asdfas","asdfasfsa","asdfasdfsadfsadfad"}, p)
I`m developing this library: https://github.com/jose78/go-collection. PLease try this example to filter elements:
package main
import (
"fmt"
col "github.com/jose78/go-collection/collections"
)
type user struct {
name string
age int
id int
}
func main() {
newMap := generateMapTest()
if resultMap, err := newMap.FilterAll(filterEmptyName); err != nil {
fmt.Printf("error")
} else {
fmt.Printf("Result: %v\n", resultMap)
result := resultMap.ListValues()
fmt.Printf("Result: %v\n", result)
fmt.Printf("Result: %v\n", result.Reverse())
fmt.Printf("Result: %v\n", result.JoinAsString(" <---> "))
fmt.Printf("Result: %v\n", result.Reverse().JoinAsString(" <---> "))
result.Foreach(simpleLoop)
err := result.Foreach(simpleLoopWithError)
if err != nil {
fmt.Println(err)
}
}
}
func filterEmptyName(key interface{}, value interface{}) bool {
user := value.(user)
return user.name != "empty"
}
func generateMapTest() (container col.MapType) {
container = col.MapType{}
container[1] = user{"Alvaro", 6, 1}
container[2] = user{"Sofia", 3, 2}
container[3] = user{"empty", 0, -1}
return container
}
var simpleLoop col.FnForeachList = func(mapper interface{}, index int) {
fmt.Printf("%d.- item:%v\n", index, mapper)
}
var simpleLoopWithError col.FnForeachList = func(mapper interface{}, index int) {
if index > 0 {
panic(fmt.Sprintf("Error produced with index == %d\n", index))
}
fmt.Printf("%d.- item:%v\n", index, mapper)
}
Result of execution:
Result: map[1:{Alvaro 6 1} 2:{Sofia 3 2}]
Result: [{Sofia 3 2} {Alvaro 6 1}]
Result: [{Alvaro 6 1} {Sofia 3 2}]
Result: {Sofia 3 2} <---> {Alvaro 6 1}
Result: {Alvaro 6 1} <---> {Sofia 3 2}
0.- item:{Sofia 3 2}
1.- item:{Alvaro 6 1}
0.- item:{Sofia 3 2}
Recovered in f Error produced with index == 1
ERROR: Error produced with index == 1
Error produced with index == 1
The DOC currently are located in wiki section of the project. You can try it in this link. I hope you like it...
REgaRDS...
I want to return a structure that looks like this:
{
results: [
["ooid1", 2.0, "Söme text"],
["ooid2", 1.3, "Åther text"],
]
}
That's an array of arrags that is string, floating point number, unicode character.
If it was Python I'd be able to:
import json
json.dumps({'results': [["ooid1", 2.0, u"Söme text"], ...])
But in Go you can't have an array (or slice) of mixed types.
I thought of using a struct like this:
type Row struct {
Ooid string
Score float64
Text rune
}
But I don't want each to become a dictionary, I want it to become an array of 3 elements each.
We can customize how an object is serialized by implementing the json.Marshaler interface. For our particular case, we seem to have a slice of Row elements that we want to encode as an array of heterogenous values. We can do so by defining a MarshalJSON function on our Row type, using an intermediate slice of interface{} to encode the mixed values.
This example demonstrates:
package main
import (
"encoding/json"
"fmt"
)
type Row struct {
Ooid string
Score float64
Text string
}
func (r *Row) MarshalJSON() ([]byte, error) {
arr := []interface{}{r.Ooid, r.Score, r.Text}
return json.Marshal(arr)
}
func main() {
rows := []Row{
{"ooid1", 2.0, "Söme text"},
{"ooid2", 1.3, "Åther text"},
}
marshalled, _ := json.Marshal(rows)
fmt.Println(string(marshalled))
}
Of course, we also might want to go the other way around, from JSON bytes back to structs. So there's a similar json.Unmarshaler interface that we can use.
func (r *Row) UnmarshalJSON(bs []byte) error {
arr := []interface{}{}
json.Unmarshal(bs, &arr)
// TODO: add error handling here.
r.Ooid = arr[0].(string)
r.Score = arr[1].(float64)
r.Text = arr[2].(string)
return nil
}
This uses a similar trick of first using an intermediate slice of interface{}, using the unmarshaler to place values into this generic container, and then plop the values back into our structure.
package main
import (
"encoding/json"
"fmt"
)
type Row struct {
Ooid string
Score float64
Text string
}
func (r *Row) UnmarshalJSON(bs []byte) error {
arr := []interface{}{}
json.Unmarshal(bs, &arr)
// TODO: add error handling here.
r.Ooid = arr[0].(string)
r.Score = arr[1].(float64)
r.Text = arr[2].(string)
return nil
}
func main() {
rows := []Row{}
text := `
[
["ooid4", 3.1415, "pi"],
["ooid5", 2.7182, "euler"]
]
`
json.Unmarshal([]byte(text), &rows)
fmt.Println(rows)
}
You can read a full example here.
Use []interface{}
type Results struct {
Rows []interface{} `json:"results"`
}
You will then have to use type assertion if you want to access the values stored in []interface{}
for _, row := range results.Rows {
switch r := row.(type) {
case string:
fmt.Println("string", r)
case float64:
fmt.Println("float64", r)
case int64:
fmt.Println("int64", r)
default:
fmt.Println("not found")
}
}
Some clumsy, but you can
type result [][]interface{}
type results struct {
Results result
}
Working example https://play.golang.org/p/IXAzZZ3Dg7
I want to understand some subtle moments of using reflect package. Please, see example below, it describes better what I want to know:
type Robot struct {
id int
model string
}
func change(i interface{}, fields ...string) {
v := reflect.ValueOf(i).Elem()
// here I emulate function by slice that could return any value,
// so here I need to check if I can store incoming values to existing struct
returns := []interface{}{100, "Something"}
for i, name := range fields {
x := reflect.ValueOf(&returns[i]).Elem()
//check if value representing x is the same of struct member
v.FieldByName(name).Set(x)
// ^ here I want to store 100 to Robot.id when i = 0,
// and "Something" to Robot.model when i = 1
}
}
func main() {
robot := &Robot{id: 1, model: "T310"}
change(robot, "model", "id")
// now robot become as follows: &Robot{100, "Something"}
}
Why does it need for?
// It is need for retrieving values from sql DB into struct members
// (only for training purposes :))
// Example:
f := func(q string, structs interface{}, fields ...string) {
rows, _ := db.Query(q)
for i := 0; rows.Next(); i++ {
rows.Scan(&structs[i])
// very dirty here! it's hard to understand how to implement it
}
}
var robots = []*Robot
f("select id, model from robots", robots, "id", "model")
// now each member of robots var should contain values from DB
I tried to be descriptive and explain as short as possible. I hope you understand me..
You can only set exported fields via reflection, so capitalize those first. Otherwise, if you're counting on positional values, make sure they are properly aligned.
Something like this for example: http://play.golang.org/p/ItnjwwJnxe
type Robot struct {
ID int
Model string
}
func change(i interface{}, fields ...string) {
returns := []interface{}{100, "Something"}
v := reflect.ValueOf(i).Elem()
for i, name := range fields {
val := reflect.ValueOf(returns[i])
v.FieldByName(name).Set(val)
}
}
func main() {
robot := &Robot{ID: 1, Model: "T310"}
fmt.Println(robot)
change(robot, "ID", "Model")
fmt.Println(robot)
}