The new bulkloader added into SDK 1.3.4 works great for models that
have a schema. For models inheriting db.Expando (or loosely defined
schemas) i would like to understand what i would have to do to bulk
upload them.
I defined a custom connector, that implemented the ConnectorInterface
and converted my data to the python dict required. How can i use this
dict to define entities that get uploaded to the data store ?
In the documentation there seems to be a post_import_function that can
be used to return the entities that get uploaded. Is there an example
on how this function is used ?
- kind: foo
connector: fooconnector
post_import_function: post_transform.post_import_fn
where,
def post_import_fn(input_dict, entity_instance, bulkload_state):
.....
is the function that is used for post_import. Hope this is useful for someone else.
Related
Although I know there is already a Blob service for Appengine, I want to experiment on storing big blobs within the datastore.
Basically I am trying to persist this object using Objectify:
BigBlob.java
BigBlobFragments.java
However, appengine is complaining that: "BigBlobFragment is not a supported property type"
For the BigBlob type I created a DAO class with CRUD operation and registed the type like this:
static {
ObjectifyService.register(BigBlob.class);
//ObjectifyService.register(BigBlobFragment.class);
}
protected BigblobDaoImpl() {
super(BigBlob.class);
}
I actually also have tried registering BigBlobFragment.
Hopefully someone can share some ideas on how to actually persist Big blobs and fragments using Objectify.
I have not used Blobs myself but I noticed you have #PersistenceCapable above your entities... that should be #Entity.
import com.googlecode.objectify.annotation.Entity;
Then you should be able to register your entity with Objectify.
ObjectifyService.register(BigBlob.class);
You need to register both the BigBlob and the BigBlobFragment and replace all of your JDO annotations with Objectify annotations (assuming you are using ofy4. You might also want to consider embedding the BigBlobFragment object inside of the BigBlob for performance using #Embed.
We're migrating to the HR datastore. We can't find this in the docs and want to make sure it'll work.
We have a few properties defined as:
ListProperty(db.Key)
Will those keys migrate normally? (they're not strings, they're db.Keys)
Same goes for
ListProperty(SomeAwesomeCustomModel)
Will those get migrated normally?
ListProperty(db.Key) gets migrated just like KeyProperty().
I'm not sure how you could have ListProperty(SomeAwesomeCustomModel) -- ListProperty() only supports a limited list of types, not Model subclasses.
I have just finished watching the following videos in an attempt to
understand JDO and Google App Engine datastore:
http://www.youtube.com/watch?v=2jW2iSKDipY
http://www.youtube.com/watch?v=Yl_J-UYE94w
http://www.youtube.com/watch?v=pzctc48c0BM
http://www.youtube.com/watch?v=tx5gdoNpcZM
Now I wonder, take the example where we have an entity of kind Grandparent
having an entity of kind Parent having an entity of kind Child as in one of
the videos. The key for one of the the Child entities could be:
Grandparent:Jane/Parent:Jack/Child:Joe
How do I code the class for this instance in JDO (presumably there will be
three classes) but I would like to see an implementation where we can see
the key values set as part of the key explicitly. Any ideas?
I also wonder, what is the difference between using JDOQL to access my
data and iterating through the various instances using iterators programmatically?
Thanks,
John Goche
There a more concrete example in the App Engine datastore java documentation: Child Objects and Relationships
You can construct Datastore queries using JDOQL string syntax and by calling methods on Query objects, there is no difference when accessible the data both will return a Collection, you can see more concrete examples in Introducing Queries in JDO
I try to get an application running which should interact with a server via RPC (JDO in Google DataStore). So I defined a persistent POJO on the server-side to get it put in the datastore via the PersistenceManager (as shown in the gwt rpc tuts). Everything works fine. But I am not able to receive the callback POJO on the client side because the POJO is only defined on server-side. How can I realize it, that the client knows that kind of object??
(sry for my bad english)
Lars
Put your POJOs in a separate package/directory (e.g. com.example.common) and then add source declaration to your GWT module descriptor (xyz.gwt.xml):
<source path="common"/> //relative to your xyz.gwt.xml location
GWT compiler will then also compile POJOs and they will be seen by your other GWT code.
Edited:
#Lars - now I understand your problem. As I see it you have several options:
If possible use Objectify instead of JDO. Objectify uses pure POJOs and they play nicely with GWT. I use this in my projects. One nice thing that Objectify gives you is #PostLoad & # PrePersist on methods to run some code before/after POJOs are loaded/saved to datastore. I use this to handle serialization of GeoPoint for instance.
Use JDO and make copies of your domain classes. This is a pain but it would work. Use 'transient' java keyword in your server JDO classes to exclude fields you do not want to RPC.
Edit #2: There is a third option that you might prefer:
Create "fake" JDO annotation classes using super-sourcing. This is a common technique to replace classes with a GWT version. Described here: http://fredsa.allen-sauer.com/2009/04/1st-look-at-app-engine-using-jdo.html
You can use DTO(stackoverflow, moar) for transferring data to client.
Basic sample here (method getTenLatestEntries() in your case).
Or you can use some third-party libraries like objectify and stop worry about making DTO`s.
I would like to make a backup of all user data in the datastore. My application is using the new namespace feature to provide multi tenanting on a per user basis (as per the example in the docs).
The bulk loader needs the namespace for each customer to download the data. I don't keep a list of users, so I can't generate the namespaces. Is there a method of detecting all the currently used namespaces?
Since SDK 1.4.0 you can use Metadata Queries:
from google.appengine.ext.db import metadata
for ns in metadata.get_namespaces():
print "namespace: '%s'" % ns.namespace_name
For NDB the import is slightly different:
from google.appengine.ext.ndb import metadata
There is also now a get_namespaces() function:
from google.appengine.ext.db import metadata
namespaces = metadata.get_namespaces()
get_namespaces() returns a list of Namespace objects. The docs also note that "metadata queries that fetch information on namespaces, kinds, and properties are generally slow to execute."
Using ndb
from google.appengine.ext.ndb import metadata
all_namespaces = [ns for ns in metadata.get_namespaces()]
Using datastore
Per Datastore Metadata:
query = client.query(kind='__namespace__')
query.keys_only()
all_namespaces = [entity.key.id_or_name for entity in query.fetch()]
There's no API to get a list of namespaces. You must keep a record of the ones you use. I use a model specifically for this.