Structs and JSON are not super fun in Go.
I have a simple example of some JSON, and a struct. Everything seems to get parsed Okay, but for some reason the array does not get picked up.
Can anyone tell me what I might be missing.
Code---
package main
import (
"encoding/base64"
"fmt"
"encoding/json"
)
type Oauth struct {
Aud string `json:"aud"`
Cid string `json:"cid"`
Exp int `json:"exp"`
Iat int `json:"iat"`
Iss string `json:"iss"`
Jti string `json:"jti"`
Scp []string `json:"scp"`
Sub string `json:"sub"`
UID string `json:"uid"`
Ver int `json:"ver"`
}
func main () {
// This is the String {"ver":1,"jti":"AT.zgv9oQpw-7l3BCg6Xb5NCG2Pf8zxgiQa1EUBXycmaDk","iss":"https://companyx.okta.com/oauth2/aus1a4ibdat0JYw5s1d8","aud":"http://localhost","iat":1484538606,"exp":1484542206,"cid":"3jmNvVCFZ5F6lWOzIONO","uid":"00uy74c0h7NGTLBSXQOC","scp":["read","remove","reserve"],"sub":"oktaadmin#okta.com"}
encoded := "eyJ2ZXIiOjEsImp0aSI6IkFULnpndjlvUXB3LTdsM0JDZzZYYjVOQ0cyUGY4enhnaVFhMUVVQlh5Y21hRGsiLCJpc3MiOiJodHRwczovL2NvbXBhbnl4Lm9rdGEuY29tL29hdXRoMi9hdXMxYTRpYmRhdDBKWXc1czFkOCIsImF1ZCI6Imh0dHA6Ly9sb2NhbGhvc3QiLCJpYXQiOjE0ODQ1Mzg2MDYsImV4cCI6MTQ4NDU0MjIwNiwiY2lkIjoiM2ptTnZWQ0ZaNUY2bFdPeklPTk8iLCJ1aWQiOiIwMHV5NzRjMGg3TkdUTEJTWFFPQyIsInNjcCI6WyJyZWFkIiwicmVtb3ZlIiwicmVzZXJ2ZSJdLCJzdWIiOiJva3RhYWRtaW5Ab2t0YS5jb20ifQ"
data, _ := base64.StdEncoding.DecodeString(encoded)
fmt.Println(string(data))
fmt.Println ("")
var x Oauth
json.Unmarshal([]byte(data), &x)
fmt.Printf ("%+v",x.Scp);
}
The result is always an Empty Array []
The encoded string isn't a valid JSON, easy to detect because you're ignoring an important error on Unmarshal, try this:
err := json.Unmarshal(data, &x)
fmt.Println(err)
It looks like you missed the last '}' of your JSON.
Your base64-encoded string is not valid:
illegal base64 data at input byte 404
Related
I am trying to parse the response from Wikipedia's API located at https://wikimedia.org/api/rest_v1/metrics/pageviews/per-article/en.wikipedia.org/all-access/all-agents/Smithsonian_Institution/daily/20160101/20170101 into an array of structs of which I will proceed to print out the view count
However, the code that I have tried to implement in order to achieve this returns nothing in the terminal when I build and run it?
The code I am failing to succeed with is as follows.
type Post struct {
Project string `json:"project"`
Article string `json:"article"`
Granularity string `json:"granularity"`
Timestamp string `json:"timestamp"`
Access string `json:"access"`
Agent string `json:"agent"`
Views int `json:"views"`
}
func main(){
//The name of the wikipedia post
postName := "Smithsonian_Institution"
//The frequency of
period := "daily"
//When to start the selection
startDate := "20160101"
//When to end the selection
endDate := "20170101"
url := fmt.Sprintf("https://wikimedia.org/api/rest_v1/metrics/pageviews/per-article/en.wikipedia.org/all-access/all-agents/%s/%s/%s/%s", postName, period, startDate, endDate)
//Get from URL
req, err := http.Get(url)
if err != nil{
return
}
defer req.Body.Close()
var posts []Post
body, err := ioutil.ReadAll(req.Body)
if err != nil {
panic(err.Error())
}
json.Unmarshal(body, &posts)
// Loop over structs and display the respective views.
for p := range posts {
fmt.Printf("Views = %v", posts[p].Views)
fmt.Println()
}
}
What is the optimal method of receiving a json response from a API such as the one mentioned above and thereafter parsing that array into an array of structs, which can then be inserted into a datastore or printed out accordingly.
Thanks
Struct declarations can be nested inside one another.
The following struct should be convertable from that json:
type resp struct {
Items []struct {
Project string `json:"project"`
Article string `json:"article"`
Granularity string `json:"granularity"`
Timestamp string `json:"timestamp"`
Access string `json:"access"`
Agent string `json:"agent"`
Views int `json:"views"`
} `json:"items"`
}
I generated that with json-to-go, which is a great time saver when working with JSON APIs.
Your solution:
data := struct {
Items []struct {
Project string `json:"project"`
Article string `json:"article"`
Granularity string `json:"granularity"`
Timestamp string `json:"timestamp"`
Access string `json:"access"`
Agent string `json:"agent"`
Views int `json:"views"`
} `json:"items"`
}{}
// you don't need to convert body to []byte, ReadAll returns []byte
err := json.Unmarshal(body, &data)
if err != nil { // don't forget handle errors
}
I have a string as: - ["a","b","c"]. How to parse / convert it into a Go array? I can do string parsing but is there any out of the box function in Go for the same.
How about using json.Unmarshal()?
s := `["a","b","c"]`
var arr []string
if err := json.Unmarshal([]byte(s), &arr); err != nil {
fmt.Println("Error:", err)
}
fmt.Printf("%q", arr)
Output (try it on the Go Playground):
["a" "b" "c"]
But know that package json does a lot of reflection kung-fu under the hood, it's faster if you write the parsing yourself. On the other hand, package json will also handle random white-spaces in the input – even newline characters and Unicode sequences, like this one (it's equivalent to ["a","b","c"]):
s := `[ "a" , "b"
,"\u0063" ] `
I'm learning Go by writing a simple http server and I need to handle some JSON responses.
With an object response, I can unmarshal it idiomatically with 2 lines of code:
structResult := Foo{}
json.Unmarshal(structBody, &structResult)
I don't know how to do the same for an array response (see the example below). Is there a way to specify (possibly via json tag) that top-level array should go into a given struct field?
package main
import "fmt"
import "encoding/json"
type Foo struct {
Id uint64 `json:"id"`
Name string `json:"name"`
}
type BaseResult struct {
Error string `json:"error"`
}
type FooResult struct {
BaseResult
Foos []Foo
}
func main() {
// Simple and works.
structBody := []byte(`{"id": 1,"name": "foo"}`)
structResult := Foo{}
json.Unmarshal(structBody, &structResult)
fmt.Printf("%#v\n", structResult)
// Doesn't work.
arrayBody := []byte(`[{"id": 1,"name": "foo"},{"id": 2,"name": "bar"},{"id": 3,"name": "foobar"}]`)
arrayResult := FooResult{}
json.Unmarshal(arrayBody, &arrayResult)
fmt.Printf("%#v\n", arrayResult)
}
I know I could make FooResult an array:
type FooResult []Foo
but then I lose the ability to specify base object which I would like to use to store error message and such. I also know that I can unmarshal into &fooResult.Foos directly, but I want the code to work with both objects and arrays.
UPDATE
Implementing UnmarshalJSON as suggested by #dyoo partially solves my problem, but I was hoping that I could use BaseResult to store parse error in case JSON has a different structure:
arrayBody := []byte(`{"error": "foo"}`)
arrayResult := FooResult{}
json.Unmarshal(arrayBody, &arrayResult)
fmt.Printf("%#v\n", arrayResult)
Of course I could implement more complex logic inside UnmarshalJSON - but isn't there a simpler way to do it?
You can implement the json.Unmarshaler interface in your FooResult, to customize exactly how it responds to unmarshaling. (Similarly, there's a json.Marshaler interface.)
Add:
func (f *FooResult) UnmarshalJSON(bs []byte) error {
return json.Unmarshal(bs, &f.Foos)
}
after which your code should otherwise work. http://play.golang.org/p/oMdoB2e-rB
You might try something like:
func (f *FooResult) UnmarshalJSON(bs []byte) error {
err1 := json.Unmarshal(bs, &f.BaseResult)
err2 := json.Unmarshal(bs, &f.Foos)
if err1 != nil && err2 != nil {
// Arbitrarily choose an error.
return err1
}
return nil
}
although even this is beginning to look dubious. Handling union type results is not quite what the json library is designed to handle automatically for you. You'll need to explicitly code the coercion logic if your JSON has dynamic type.
See: How to unmarshall an array of different types correctly? and http://blog.golang.org/json-and-go for related issues.
Just specify Foos when you Unmarshal
package main
import "fmt"
import "encoding/json"
type Foo struct {
Id uint64 `json:"id"`
Name string `json:"name"`
}
type BaseResult struct {
Error string `json:"error"`
}
type FooResult struct {
BaseResult
Foos []Foo
}
func main() {
// Simple and works.
structBody := []byte(`{"id": 1,"name": "foo"}`)
structResult := Foo{}
json.Unmarshal(structBody, &structResult)
fmt.Printf("%#v\n", structResult)
// Doesn't work.
arrayBody := []byte(`[{"id": 1,"name": "foo"},{"id": 2,"name": "bar"},{"id": 3,"name": "foobar"}]`)
arrayResult := FooResult{}
if err := json.Unmarshal(arrayBody, &arrayResult.Foos); err != nil {
arrayResult.BaseResult.Error = string(arrayBody)
}
fmt.Printf("%#v\n", arrayResult)
}
There is something wrong when I unmarshal the json array.
How do I correct it ? the code is:http://play.golang.org/p/AtU9q8Hlye
package main
import (
"encoding/json"
"fmt"
)
type Server struct {
ServerName string
ServerIP string
}
type Serverslice struct {
Name string
Servers []Server
}
func main() {
var s []Serverslice
str := `{"name":"dxh","servers":[{"serverName":"VPN0","serverIP":"127.0.0.1"},{"serverName":"Beijing_VPN","serverIP":"127.0.0.2"}],
"name":"dxh1,"servers":[{"serverName":"VPN1","serverIP":"127.0.0.1"},{"serverName":"Beijing_VPN","serverIP":"127.0.0.2"}]}`
json.Unmarshal([]byte(str), &s) //the wrong line.....................
fmt.Println(len(s))
}
First of all, you're ignoring the error return value from json.Unmarshal. You probably want something like:
if err := json.Unmarshal([]byte(str), &s); err != nil {
log.Fatalln(err)
}
With that change, we can see that your JSON data isn't valid: invalid character 's' after object key:value pair. There is a missing quote at the end of "dxh1 on the second line.
Fixing that error and rerunning the program you'll get a different error: json: cannot unmarshal object into Go value of type []main.Serverslice. There are two possible problems here:
You meant to decode into an object. In this case, just declare s as a Serverslice. Here is a version of your program that makes that change: http://play.golang.org/p/zgyr_vnn-_
Your JSON is supposed to be an array (possible, since it seems to have duplicate keys). Here's an updated version with the JSON changed to provide an array: http://play.golang.org/p/Wl6kUaivEm
I have an instance of a struct that I defined and I would like to convert it to an array of bytes. I tried []byte(my_struct), but that did not work. Also, I was pointed to the binary package, but I am not sure which function I should use and how I should use it. An example would be greatly appreciated.
One possible solution is the "encoding/gob" standard package. The gob package creates an encoder/decoder that can encode any struct into an array of bytes and then decode that array back into a struct. There's a great post, here.
As others have pointed out, it's necessary to use a package like this because structs, by their nature, have unknown sizes and cannot be converted into arrays of bytes.
I've included some code and a play.
package main
import (
"bytes"
"encoding/gob"
"fmt"
"log"
)
type P struct {
X, Y, Z int
Name string
}
type Q struct {
X, Y *int32
Name string
}
func main() {
// Initialize the encoder and decoder. Normally enc and dec would be
// bound to network connections and the encoder and decoder would
// run in different processes.
var network bytes.Buffer // Stand-in for a network connection
enc := gob.NewEncoder(&network) // Will write to network.
dec := gob.NewDecoder(&network) // Will read from network.
// Encode (send) the value.
err := enc.Encode(P{3, 4, 5, "Pythagoras"})
if err != nil {
log.Fatal("encode error:", err)
}
// HERE ARE YOUR BYTES!!!!
fmt.Println(network.Bytes())
// Decode (receive) the value.
var q Q
err = dec.Decode(&q)
if err != nil {
log.Fatal("decode error:", err)
}
fmt.Printf("%q: {%d,%d}\n", q.Name, *q.X, *q.Y)
}
I assume you want something like the way C handles this. There is no built in way to do that. You will have to define your own serialization and deserialization to and from bytes for your struct. The binary package will help you encode
the fields in your struct to bytes that you can add to the byte array but you will be responsible for specifying the lengths and offsets in the byte array that will hold the fields from your struct.
Your other options are to use one of the encoding packages: http://golang.org/pkg/encoding/ such as gob or json.
EDIT:
Since you want this for making a hash as you say in your comment the easisest thing to do is use []byte(fmt.Sprintf("%v", struct)) like so: http://play.golang.org/p/yY8mSdZ_kf
Just use json marshal， this is a very simple way.
newFsConfig := dao.ConfigEntity{EnterpriseId:"testing"}
newFsConfigBytes, _ := json.Marshal(newFsConfig)
I know this thread is old, but none of the answers were accepted, and there's a pretty simple way to do this.
https://play.golang.org/p/TedsY455EBD
important code from playground
import (
"bytes"
"fmt"
"encoding/json"
)
type MyStruct struct {
Name string `json:"name"`
}
testStruct := MyStruct{"hello world"}
reqBodyBytes := new(bytes.Buffer)
json.NewEncoder(reqBodyBytes).Encode(testStruct)
reqBodyBytes.Bytes() // this is the []byte
Serialization is likely proper answer.
But if you consent to unsafety and actually need to read struct as bytes, then relying on byte array memory representation might be a bit better than relying on byte slice internal structure.
type Struct struct {
Src int32
Dst int32
SrcPort uint16
DstPort uint16
}
const sz = int(unsafe.SizeOf(Struct{}))
var asByteSlice []byte = (*(*[sz]byte)(unsafe.Pointer(&struct_value)))[:]
Works and provides read-write view into struct, zero-copy. Two "unsafe" should hint enough that it may break badly.
You should use a bytes buffer instead of a string, the other suggested methods create a SHA1 of variable length, the SHA1 standard length must be 20 bytes (160 bits)
package main
import (
"crypto/sha1"
"fmt"
"encoding/binary"
"bytes"
)
type myStruct struct {
ID string
Data string
}
func main() {
var bin_buf bytes.Buffer
x := myStruct{"1", "Hello"}
binary.Write(&bin_buf, binary.BigEndian, x)
fmt.Printf("% x", sha1.Sum(bin_buf.Bytes()))
}
Try it yourself: http://play.golang.org/p/8YuM6VIlLV
It's a really easy method and it works great.
package main
import (
"crypto/sha1"
"fmt"
"encoding/binary"
"bytes"
)
type myStruct struct {
ID [10]byte
Data [10]byte
}
func main() {
var bin_buf bytes.Buffer
x := myStruct{"1", "Hello"}
binary.Write(&bin_buf, binary.BigEndian, x)
fmt.Printf("% x", sha1.Sum(bin_buf.Bytes()))
}
binary.Write takes a struct which has fixed length memory allocated datatype.
json.Marshal is the best option to convert a struct to []byte, see example below:
package main
import (
"encoding/json"
"fmt"
)
type ExampleConvertToByteArray struct {
Name string
SurName string
}
func main() {
example := ExampleConvertToByteArray{
Name: "James",
SurName: "Camara",
}
var exampleBytes []byte
var err error
exampleBytes, err := json.Marshal(example)
if err != nil {
print(err)
return
}
fmt.Println(string(exampleBytes))
}
Go playground -> https://play.golang.org/p/mnB9Cxy-2H3
Take a look at https://blog.golang.org/go-slices-usage-and-internals
Specifically slice internals. The idea is to mimic slice's internal structure and point to our struct instead of a byte sequence:
package main
import (
"fmt"
"unsafe"
)
// our structure
type A struct {
Src int32
Dst int32
SrcPort uint16
DstPort uint16
}
// that is how we mimic a slice
type ByteSliceA struct {
Addr *A
Len int
Cap int
}
func main() {
// structure with some data
a := A{0x04030201,0x08070605,0x0A09, 0x0C0B}
// create a slice structure
sb := &ByteSliceA{&a, 12, 12} // struct is 12 bytes long, e.g. unsafe.Sizeof(a) is 12
// take a pointer of our slice mimicking struct and cast *[]byte on it:
var byteSlice []byte = *(*[]byte)(unsafe.Pointer(sb))
fmt.Printf("%v\n", byteSlice)
}
Output:
[1 2 3 4 5 6 7 8 9 10 11 12]
https://play.golang.org/p/Rh_yrscRDV6
Have you considered serializing it to bson? http://labix.org/gobson
var v any
b := (*[unsafe.Sizeof(v)]byte)(unsafe.Pointer(&v))
c := b[:]