AppEngine Datastore Encoding a Slice of Slices with Go - google-app-engine

I have a need to store a slice of slices in that Datastore. Since the Datastore doesn't support two levels of slices, I have encode the child slice as JSON and store it as a []byte. I'm using the PropertyLoadSaver to accomplish this. The saving and loading works, but I've noticed that the saved entity has a empty []byte for every parent element. After some investigation I discovered this is caused by the Variants.OptionsJSON Property being referenced in the Save() method and also an attribute of the struct.
So my question are:
Is this a bug or am I doing something wrong?
Is there a better way to accomplish this?
The one constraint I have is that I need to be able to query the entity by one of the "Variant" attributes, but not the "Variant.Options".
Thanks!
Example:
Given JSON
{
"variants": [{
"options": [
{
"name": "One",
"value": "one"
},{
"name": "Two",
"value": "two"
}]
}, {
"options": [
{
"name": "Three",
"value": "three"
},{
"name": "Four",
"value": "four"
}]
}]
}]
}
How it's stored in the datastore, notice the two empty '', '' attributes at the end:
Entity Kind: Products
Entity Key: ahBkZXZ-c3RvcmVza2ktY29tchULEghQcm9kdWN0cxiAgICAgNDHCww
ID: 6507459568992256
Variants.OptionsJSON (list): `['[{"name":"One","value":"one"},{"name":"Two","value":"two"}]', '[{"name":"Three","value":"three"},{"name":"Four","value":"four"}]', '', '']`
model.go
import (
"appengine/datastore"
"encoding/json"
)
type Option struct {
Name string `json:"name"`
Value string `json:"value"`
}
type Variant struct {
OptionsJSON []byte `json:"-" datastore:"OptionsJSON"`
Options []Option `json:"options" datastore:"-"`
}
type Product struct {
Variants []Variant `json:"variants"`
}
func (x *Product) Load(c <-chan datastore.Property) error {
if err := datastore.LoadStruct(x, c); err != nil {
return err
}
for i, v := range x.Variants {
var opts []Option
json.Unmarshal(v.OptionsJSON, &opts)
x.Variants[i].Options = opts
}
return nil
}
func (x *Product) Save(c chan<- datastore.Property) error {
for _, v := range x.Variants {
b, _ := json.Marshal(v.Options)
c <- datastore.Property{
Name: "Variants.OptionsJSON",
Value: b,
NoIndex: true,
Multiple: true,
}
}
return datastore.SaveStruct(x, c)
}

Related

golang json unstructured data

I have a json (unstructured) and wants to retrieve each and every key from the json data
Then loop through the keys and value. If the value is of type json (nested) or array then keep continuing.
I have found an example of structured json parsing but cannot get this.
Checked this code but could not get the complete one
err := json.Unmarshal([]byte(input), &customers)
Sample json:
{
"components": [
{
"key": "d1",
"components": [
{
"key": "custname",
"value": "Abraham",
"input": true,
"tableView": true
},
{
"key": "type",
"type": "radio",
"label": "Fisrt",
"values": [
{
"label": "Sole",
"value": "sole",
"shortcut": ""
},
{
"label": "Bata",
"value": "Bata",
"shortcut": ""
}
],
"validate": {
"required": true
},
"tableView": false
},
{
"key": "registeredField",
"value": "reg 111",
"input": true,
},
{
"key": "dirc",
"value": "abraham",
},
{
"key": "gst",
"value": "textfield",
"useLocaleSettings": false
},
{
"key": "pan",
"value": "AAAAA0000",
"useLocaleSettings": false
}
],
"collapsible": false
}
]
}
Expected output:
Key: custname Value: Abraham
Key: type Value: {
"label": "Sole",
"value": "sole",
"shortcut": ""
}, {
"label": "Bata",
"value": "Bata",
"shortcut": ""
}
Key: registeredField Value: reg 111
So you have an object with a key components, that is a slice of components. Each of these components can have a number of keys. First thing to do is to take stock of all possible fields a component can have and define a type with said fields:
type Validation struct {
Required bool `json:"required"`
}
type Value struct {
Label string `json:"label"`
Value string `json:"value"`
Shortcut string `json:"shortcut"`
}
type Data struct {
Components []Data `json:"components,omitempty"`
Collapsable bool `json:"collapsable"`
Input bool `json:"input"`
Key string `json:"key"`
TableView bool `json:"tableView"`
Type string `json:"type"`
Value string `json:"value"`
UseLocaleSettings bool `json:"useLocaleSettings"`
Values []Value `json:"values,omitempty"`
Validate *Validation `json:"validate,omitempty"`
}
Now you simply take the input and unmarshal it into the Data type:
data := Data{}
if err := json.Unmarshal([]byte(input), &data); err != nil {
// handle error
fmt.Printf("Oops, something went wrong: %+v", err)
return
}
At this point, we have all the data in a struct, so we can start printing it all out. First thing we notice is how Data basically contains a slice of Data types. A recursive function to print it all out would make sense:
func PrintComponents(data []Data) {
for _, c := range data {
if len(c.Components) > 0 {
PrintComponents(c.Components) // recursive
continue // skip value of this component, remove this line if needed
}
val := c.Value // assign string value
if len(c.Values) > 0 {
// this component has a slice of values, not a single value
vals, err := json.MarshalIndent(c.Values, "", " ") // marshal with indent of 4 spaces, no prefix
if err != nil {
fmt.Printf("Oops, looks like we couldn't format something: %+v\n", err)
return // handle this
}
val = string(vals) // marshalled values as string
}
fmt.Printf("Key: %s Value: %s\n", c.Key, val) // print output
}
}
You could alter this function a bit to pass in an indent parameter for each level of recursion, so you can print out the components in indented blocks:
func PrintComponents(data []Data, indent string) {
for _, c := range data {
if len(c.Components) > 0 {
// print the key for this block of components
fmt.Printf("Component block: %s\n", c.Key)
PrintComponents(data, indent + " ") // current indent + 4 spaces
continue // we're done with this component
}
val := c.Value
if len(c.Values) > 0 {
vals, _ := json.MarshalIndent(c.Values, indent, " ") // pass in indent level here, and DON'T ignore the error, that's just for brevity
val = string(vals)
}
fmt.Printf("%sKey: %s Value: %s\n", indent, c.Key, val) // pass in indent
}
}
Putting it all together, we get this:
func main() {
data := Data{}
if err := json.Unmarshal(input, &data); err != nil {
fmt.Println(err.Error())
return
}
fmt.Println("Printing with simple recursive function")
// print all components, these could be nested, so let's use a recursive function
PrintComponents(data.Components)
fmt.Println("\n\nPrinting with indented recursion:")
PrintComponentsIndent(data.Components, "") // start with indent of 0
}
func PrintComponents(data []Data) {
for _, c := range data {
if len(c.Components) > 0 {
PrintComponents(c.Components) // recursive
continue // skip value of this component, remove this line if needed
}
val := c.Value // assign string value
if len(c.Values) > 0 {
// this component has a slice of values, not a single value
vals, err := json.MarshalIndent(c.Values, "", " ") // marshal with indent of 4 spaces, no prefix
if err != nil {
fmt.Printf("Oops, looks like we couldn't format something: %+v\n", err)
return // handle this
}
val = string(vals) // marshalled values as string
}
fmt.Printf("Key: %s Value: %s\n", c.Key, val) // print output
}
}
func PrintComponentsIndent(data []Data, indent string) {
for _, c := range data {
if len(c.Components) > 0 {
fmt.Printf("%sComponent block: %s\n", indent, c.Key)
PrintComponentsIndent(c.Components, indent + " ")
continue
}
val := c.Value
if len(c.Values) > 0 {
// this component has a slice of values, not a single value
vals, _ := json.MarshalIndent(c.Values, indent, " ")
val = string(vals) // marshalled values as string
}
fmt.Printf("%sKey: %s Value: %s\n", indent, c.Key, val) // print output
}
}
Which outputs:
Printing with simple recursive function
Key: custname Value: Abraham
Key: type Value: [
{
"label": "Sole",
"value": "sole",
"shortcut": ""
},
{
"label": "Bata",
"value": "Bata",
"shortcut": ""
}
]
Key: registeredField Value: reg 111
Key: dirc Value: abraham
Key: gst Value: textfield
Key: pan Value: AAAAA0000
Printing with indented recursion:
Component block: d1
Key: custname Value: Abraham
Key: type Value: [
{
"label": "Sole",
"value": "sole",
"shortcut": ""
},
{
"label": "Bata",
"value": "Bata",
"shortcut": ""
}
]
Key: registeredField Value: reg 111
Key: dirc Value: abraham
Key: gst Value: textfield
Key: pan Value: AAAAA0000
Your desired output doesn't include the square brackets for the values slice. Well, that's a really easy thing to get rid of. The square brackets are always the first and last characters of the string, and json.Marshal returns a byte slice ([]byte). Chopping off the first and last characters is as easy as:
val = string(vals[1:len(vals)-2])
Taking a sub-slice of the byte slice returned by json.Marshal, starting at offset 1 (cutting of offset 0, which is [), and keeping everything until the next to last character (offset len(vals)-2). For the indented example, that will leave you with a blank line, containing an unknown number of spaces (indentation). You can trim the right-hand side of the string using the strings package:
// remove square brackets, trim trailing new-line and spaces
val = strings.TrimRight(string(vals[1:len(vals)-2]), "\n ")

How to format JSON correctly with arrays

I'm trying to send a JSON payload in my POST request but I'm not sure on how to format it correctly to use arrays. This below is what the correct JSON itself looks like:
{
"animal": "dog",
"contents": [{
"name": "daisy",
"VAL": "234.92133",
"age": 3
}]
}
I have this so far:
group := map[string]interface{}{
"animal": "dog",
"contents": map[string]interface{}{
"name": "daisy",
"VAL": "234.92133",
"age": 3,
},
}
But I can't figure out how to do array of contents (the square brackets), only the curly brackets from "contents".
The quick answer:
group := map[string]interface{}{
"animal": "dog",
"contents": []map[string]interface{}{
{
"name": "daisy",
"VAL": "234.92133",
"age": 3,
},
},
}
But as already said in the comments it is better (type safety) to use structs instead:
type Animal struct {
Type string `json:"animal"`
Contents []AnimalContent `json:"contents"`
}
type AnimalContent struct {
Name string `json:"name"`
Value string `json:"VAL"`
Age int `json:"age"`
}
Then create with:
group := Animal{
Type: "dog",
Contents: []AnimalContent{
{
Name: "daisy",
Value: "234.92133",
Age: 3,
},
},
}
// to transform to json format
bts, err := json.Marshal(group)
if err != nil {
log.Fatal(err)
}
fmt.Println(string(bts))

Assigning Alamofire result to array

I am trying to assign the result that is retrieved by Alamofire to an array, and have come across an issue.
I am using the Stripe API for products, it returns back a JSON object called "data:", I am trying to assign that data object only to that products array I have.
Here's my code
var products: [Product] = []
let stripeProducts = "stripe-products-api"
func createArray() {
let stripeAuthHeader: HTTPHeaders = []
AF.request(stripeProducts, headers: stripeAuthHeader).responseJSON {
switch response.result {
case .failure(let error):
print(error)
case .success:
self.products = response.data \\ trying to access the data object from the JSON data
print(self.products)
}
}
}
JSON:
"object": "list",
"data": [
{
"id": "prod_123456",
"object": "product",
"active": true,
"attributes": [
],
"created": 1590423835,
"description": "Test",
"images": [
""
],
"livemode": false,
"metadata": {
"address": "Test"
},
"name": "Blue Jeans",
"statement_descriptor": null,
"type": "service",
"unit_label": null,
"updated": 1590653248
}
]
Thank you for your help.
You need to have Struct of Products
struct Datum: Decodable {
let id, object: String
let active: Bool
let created: Int
let datumDescription: String
let images: [String]
let livemode: Bool
let metadata: Metadata
let name: String
let type: String
let updated: Int
enum CodingKeys: String, CodingKey {
case id, object, active, created
case datumDescription = "description"
case images, livemode, metadata, name
case type
case updated
}
}
// MARK: - Metadata
struct Metadata: Decodable {
let address: String
}
Then parse it like this
let products = try? newJSONDecoder().decode(Products.self, from: response.data)

Elasticsearch must some values and must not all others

so I have an index with the following mapping:
{
"tests": {
"mappings": {
"test": {
"properties": {
"arr": {
"properties": {
"name": {
"type": "long"
}
}
}
}
}
}
}
I have 3 documents with the following values for the field "arr":
arr: [{name: 1}, {name: 2}, {name: 3}]
arr: [{name: 1}, {name: 2}, {name: 4}]
arr: [{name: 1}, {name: 2}]
I would like to search in a way that I could find the documents in which all name values are in an array of name. However, the document doesn't need to have all the values of the array, and if it has one value that is not in the array, then the document is not good.
For example:
If my array of names is [1,2], then I only want document 3, but I have all of them
If my array of names is [1,2,3], then I want document 1 and 3, but I only have document 1 with must and all of them with should
If my array of names is [1,2,4], then I want document 2 and 3, but I only have document 2 with must and all of them with should
If my array of names is [1], then I want no document, but I have all of them
Here, the arrays are small, in my project the arrays in the documents are much more bigger and the array of comparison as well.
So, to be specific, I need to search in a way that:
All the names in the document are in the array of names
The document does not need to have ALL the values in the array of name
Thank you in advance
Using Scripting (No Mapping Change)
No mapping changes are required to what you already have.
I've come up with the below script. I believe the core logic is in the script which I think is self-explanatory.
POST test_index_arr/_search
{
"query":{
"bool": {
"filter": {
"script": {
"script": {
"source" : """
List myList = doc['arr.name'];
int tempVal = myList.size();
for(int i=0; i<params.ids.size(); i++){
long temp = params.ids.get(i);
if(myList.contains(temp))
{
tempVal = tempVal - 1;
}
if(tempVal==0){
break;
}
}
if(tempVal==0)
return true;
else
return false;
""",
"lang" : "painless",
"params": {
"ids": [1,2,4]
}
}
}
}
}
}
}
So what the scripting does is, it checks if a document has all its numbers in its arr present in the input, if so it would return the document.
In the above query, the part "ids": [1,2,4] acts as input. You need to add/remove/update values here this depending on your requirement.
Hope it helps!

mongodb query an array within an array within an array

I have a collection of users. Each of those have an array of bookmarks. Each bookmark has an array of categories it belongs to. Leading to a structure like this:
[
{name: "Bob",
bookmarks: [
{url: "http://duckduckgo.com",
categories: [
"Search",
"Ducks",
],
},
],
},
]
Now given a name and a url and a category name. I want to delete said category of the respective bookmark. But my problem is that all attempts return the whole user or delete the whole bookmark and not just the category.
This is my best attempt using the mgo driver so far:
type arbitraryJson map[string]interface{}
user := "Bob"
bookmarkURL := url.Parse("http://duckduckgo.com")
tagName := "Search"
err = userDB.Update(
arbitraryJson{
"name": user,
"bookmarks.url": bookmarkURL.String(),
},
arbitraryJson{
"$pull": arbitraryJson{
"bookmarks.categories": tagName,
},
},
)
Which translates (I think) to the mongo query:
db.users.updateOne(
{ name: "Bob",
bookmarks.url: "http://duckduckgo.com" }
{
$pull: { bookmarks.categories: "search" }
}
)

Resources