I am introducing namespacing into my application, but I have run into an issue with one of my existing queries that performs the following operation in order to determine whether or not an entity exists for the given key.
// c is of type context.Context
c, _ = appengine.Namespace(c, "name")
k := datastore.NewKey(c, "Kind", "", id, nil)
q := datastore.NewQuery("Kind").Filter("__key__ =", k).KeysOnly()
keys, err := q.GetAll(c, nil)
When this command is executed with a namespace applied to the context, it gives back the following error:
datastore_v3 API error 1: __key__ filter namespace is but query namespace is db
I could just use a Get query instead, but I don't need to actually retrieve the entity at all. Plus, keys-only queries are free!
Update
It seems that all queries are failing after I have introduced namespacing. The documentation doesn't mention any sort of special treatment for the indices:
https://cloud.google.com/appengine/docs/go/multitenancy/multitenancy
"By default, the datastore uses the current namespace for datastore requests. The API applies this current namespace to datastore.Key objects when they are created. Therefore, you need to be careful if an application stores Key objects in serialized forms, since the namespace is preserved in those serializations."
Using namespaces with the Datastore
https://cloud.google.com/appengine/docs/go/multitenancy/multitenancy#Go_Using_namespaces_with_the_Datastore
Related
I have found some similary questions here ,but no solid answer.
How to delete or reset a search index in Appengine
how to delete search index in GAE Search API
How to delete a search index on the App Engine using Go?
How to delete a search Index itself
I see some googler suggest that
You can effectively delete an index by first using index.delete() to remove all of the documents from an index, and then using index.delete_schema() to remove the type mappings from the index 1.
Unfortunately, golang sdk does not have "index.delete_schema()" api.
I can only delete document one by one by getting itemId list from index. And We got a surprisely billing status in dashboard:
Resource Usage Billable Price Cost
Search API Simple Searches 214,748.49 10K Ops 214,748.39 $0.625 / 10K Ops $134,217.74
Can someone tell me how to effectively delete Google App engine Search API index wihout cost so much ?
Unfortunately there is no simple operation that allows you to delete an entire large search index without incurring substantial cost, short of deleting the entire app (which, actually, could be an effective approach in certain circumstances).
The short answer is NO.
There is no pefectly efficient way with GCP to drop full search index in one go.
The only efficient way they themselves suggest in thier "Best Practices" is to delete in bactches of 200 documents per index.delete() method call (in Java and Python app engine sdk).
To add to the disappointment, GO SDK even does not support this too and allows only one doc deletion per call. What a miserable support from GCP!
So if your indexes have grown to some good GBs, you are forced to consume your dollars and days or better say cleanup the mess, left by GCP, at your own cost. Mind it, it costs you a lot with giant index>10GB.
Now, how to do it in GO runtime.
Donot do it with GO runtime
Better, write a micro-service in Java or Python under the same projectId and use these runtimes with thier SDKs/Client Libraries to delete index the only efficient way(200 per call), GCP offers. So very very limited and essentially cost bearing solution with app engine. Have to live with it, dear :)
PS. I have created an bug/issue a year back regarding the same. No actions takent yet :)
As you mentioned, deleting an index is only available for Java 8 at the moment.
Since you're using GO, currently there is no possibility to delete an index, but you can remove documents that are part of it to reduce the cost
To delete documents from an index you can follow this example here
func deleteHandler(w http.ResponseWriter, r *http.Request) {
ctx := appengine.NewContext(r)
index, err := search.Open("users")
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
id := "PA6-5000"
err = index.Delete(ctx, id)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
fmt.Fprint(w, "Deleted document: ", id)
}
When creating an entity using an IncompleteKey so that each record is inherently unique, what is the best way to add the key back into the record so it can be passed around in the structure- at the time of creation?
For example, is something like this (untested code) a good idea, using Transactions?
err = datastore.RunInTransaction(c, func(c appengine.Context) error {
incompleteKey := datastore.NewIncompleteKey(c, ENTITY_TYPE, nil)
entityKey, err := datastore.Put(c, incompleteKey, &MyStruct)
if(err != nil) {
return err
}
MyStruct.SelfID = entityKey.IntID()
_, err = datastore.Put(c, entityKey, &MyStruct)
return err
}, nil)
As a followup- I'm guessing this should almost never fail since it will almost never operate over the same incompleteKey?
You don't need to put the MyStruct into DB twice - it's unnecessary overhead. The key stored as a part of the entity and can be retrieved when needed.
There is a good example in docs on how to store an entity and used it ID as a reference: https://cloud.google.com/appengine/docs/go/datastore/entities#Go_Ancestor_paths
When you want to get keys for entities you can do this using this approach:
https://cloud.google.com/appengine/docs/go/datastore/queries#Go_Retrieving_results - (edited) notice in the example that keys and structs are populated in 1 operation.
If you query the an entity by key you already know it ID.
So there is no need to have an ID as a separate property. If you want to pass it around with the entity for your business logic you can create a wrapper - either generalized using interface() for the entity struct or a strongly typed (1 per each entity struct).
Is there a way for me to use datastore's GetMulti, or another function built into the "appengine/datastore" package, to get all entities of a single kind?
For instance, I have a kind "Queue" with many entities that have two to three properties. Each entity has a unique stringID and what I'm trying to get is a slice or other comparable data type of each unique stringID.
The purpose of Queue is to store some metadata and the unique key names that I'll be looping over and performing a cron task on (e.g. keys "user1", "user2", and "user3" are stored as kind Queue, then - during cron - are looped over and interacted with).
Thanks.
I'm new to Google App Engine and I didn't read the documentation before diving in. Now that I actually read the docs, it looks like I'll be answering my own question. This can be accomplished via a simple query, looping over the Keys, and appending the StringID of each key to a slice of strings:
var queuedUsers []string
q := datastore.NewQuery("Queue").KeysOnly()
keys, _ := q.GetAll(c, nil)
for _, v := range keys {
queuedUsers = append(queuedUsers, v.StringID())
}
I've been developing a couple apps using Google App Engine Go SDK that use Memcache as a buffer for loading data from the Datastore. As the Memcache is only able to store data as []byte, I often find myself creating functions to encode the various structures as strings, and also functions to reverse the process. Needless to say, it is quite tedious when I need to do this sort of thing 5 times over.
Is there an easy way to convert any arbitrary structure that can be stored in Datastore into []byte in order to store it in Memcache and then load it back without having to create custom code for various structures in GAE Golang?
http://golang.org/pkg/encoding/gob or http://golang.org/pkg/encoding/json can turn arbitrary datatypes into []byte slices given certain rules apply to the datastructures being encoded. You probably want one of them gob will encode to smaller sizes but json is more easily shareable with other languages if that is a requirement.
I found myself needing the same thing. So I created a package called:
AEGo/ds
Documentation | Source
go get github.com/scotch/aego/ds
It uses the same API as the "appengine/datastore" so It will work as a drop in replacement.
import "github.com/scotch/aego/v1/ds"
u = &User{Name: "Bob"}
key := datastore.NewKey(c, "User", "bob", 0, nil)
key, err := ds.Put(c, key, u)
u = new(User)
err = ds.Get(c, key, u)
By default it will cache all Puts and Gets to memcache, but you can modify this behavior by calling the ds.Register method:
ds.Register("User", true, false, false)
The Register method takes a string representing the Kind and 3 bool - userDatastore, useMemcache, useMemory. Passing a true value will cause AEgo/ds to persist the record to that store. The Memory store is useful for records that you do not expect to change, but could contain stale data if you have more then one instance running.
Supported methods are:
Put
PutMulti
Get
GetMulti
Delete
DeleteMulti
AllocateIDs
Note: Currently cashing only occures with Get. GetMulti pulls from the datastore.
AEGo/ds is a work in progress, but the code is well tested. Any feedback would be appreciated.
And to answer you question here's how I serialized the entities to gob for the memcache persistence.
I just started developing a GAE app with the Go runtime, so far it's been a pleasure. However, I have encountered the following setback:
I am taking advantage of the flexibility that the datastore provides by having several different structs with different properties being saved with the same entity name ("Item"). The Go language datastore reference states that "the actual types passed do not have to match between Get and Put calls or even across different App Engine requests", since entities are actually just a series of properties, and can therefore be stored in an appropriate container type that can support them.
I need to query all of the entities stored under the entity name "Item" and encode them as JSON all at once. Using that entity property flexibility to my advantage, it is possible to store queried entities into an arbitrary datastore.PropertyList, however, the Get and GetAll functions return ErrFieldMismatch as an error when a property of the queried entities cannot be properly represented (that is to say, incompatible types, or simply a missing value). All of these structs I'm saving are user generated and most values are optional, therefore saving empty values into the datastore. There are no problems while saving these structs with empty values (datastore flexibility again), but there are when retrieving them.
It is also stated in the datastore Go documentation, that it is up to the caller of the Get methods to decide if the errors returned due to empty values are ignorable, recoverable, or fatal. I would like to know how to properly do this, since just ignoring the errors won't suffice, as the destination structs (datastore.PropertyList) of my queries are not filled at all when a query results in this error.
Thank you in advance, and sorry for the lengthy question.
Update: Here is some code
query := datastore.NewQuery("Item") // here I use some Filter calls, as well as a Limit call and an Order call
items := make([]datastore.PropertyList, 0)
_, err := query.GetAll(context, &items) // context has been obviously defined before
if err != nil {
// something to handle the error, which in my case, it's printing it and setting the server status as 500
}
Update 2: Here is some output
If I use make([]datastore.PropertyList, 0), I get this:
datastore: invalid entity type
And if I use make(datastore.PropertyList, 0), I get this:
datastore: cannot load field "Foo" into a "datastore.Property": no such struct field
And in both cases (the first one I assume can be discarded) in items I get this:
[]
According to the following post the go datastore module doesn't support PropertyList yet.
Use a pointer to a slice of datastore.Map instead.