I am exploring web development with Google AppEngine (Java). My application has a very basic data storage requirement that is well suited to AppEngine's 'map' like datastore.
The basic unit is one class that will have member variables that will be written or read from the database per transaction (this is because it interacts with an Android app).
I am considering using Objectify for interfacing.
My questions are : What happens if I later change the size (number of variables) in my base class ? I know AppEngine isn't typed but will Objectify cause any problems if some variables are available for some keys and not for others ?
This is addressed extensively in the manual:
http://code.google.com/p/objectify-appengine/wiki/IntroductionToObjectify#Migrating_Schemas
The short answer is that you can add and remove fields at will. In addition, there are facilities for more sophisticated transformations of data.
If you decide to move from the Objectify framework to the Low-level API later, you won't have a problem. App engine's datastore is typed, but not with all the Java types. I don't know whether you'd be able to get JDO or JPA to work without reading and re-writing all your data, but I think you probably would.
Objectify 4's method of storing a Map is pretty nice - storing properties as something like "fieldname-mapkey"
Related
I have an Python appengine app using datastore with ndb API and I want to do background work and store results into the datastore so appengine uses it.
I wanted to use GCE or my computer to do so, but ndb API is not available outside appengine and the alternative seems to be gcloud.datastore API which is very different.
How do you guarantee that what you push (with gcloud API) is consistent with what you get (ie: matches a ndb entity) ?
I can't do unit-tests because the local server is not the same (gcd vs dev_appserver). Here is a workaround (but in Java).
Should I replace ndb code by gcloud.datastore in appengine to ensure consistency (but loosing ndb advantages like auto caching...) ?
Is there an obvious solution I'm missing ? If someone had the same issue, how did you handle it ?
Thanks
If you're really worried about consistency and you're using fancy features of ndb, you should probably look into using the Remote API for App Engine, which effectively let's you run arbitrary code via a remote (HTTP) interface. It might help you get the job done, but remember that the CPU cycles you're using will be in GAE -- not in GCE.
If you're willing to wait a while, we're working on porting the ndb API to run against the Cloud Datastore API which would mean the same code you run in App Engine will work outside of App Engine (on your local machine or inside Google Compute Engine).
The gcloud.datastore (gcloud-python) API is much more low-level, so you should have even more control over the data that ends up in the Datastore. It wasn't built to be identical to ndb (and therefore doesn't have some of the fancy stuff like derived fields or geo-points as first-class citizens), however ndb stores those fields using it's own Python logic so you should be able to write safely if you're comfortable with the lower-level data representations.
I have an application that is using JDBC to manipulate its data.
I'm looking for a way to run the application on Appengine. Unfortunately, it seems like there aren't many options besides datastore (which I just can't get used to).
I've tried to use an embedded JavaDB, but Appengine blocks FileOutputStream (throws an exception "java.io.FileOutputStream is a restricted class" on initial driver loading). Therefore, I haven't tried Sqlite.
I've also tried to use Jiql, a new project that offers a JDBC interface to Datastore. However, I just can't seem to establish a "connection". It is somewhat unclear to me how to use jiql (like: what user/pass do you supply?), despite the few examples on their site.
I've looked at Google Cloud SQL and eventually signed up for it. It seemed to be the perfect solution to our problem, until I realized it's a paid service (it'll be paid soon, once it leaves the beta phase). I prefer not to pay (yet) as I'm still testing and evaluating the potential of GAE.
--
What would you recommend, keeping in mind would like to use JDBC for the persistence layer of the application?
Thanks in advance!
GAE supports two frameworks for persistence, JPA and JDO. There are few frameworks that plays well with GAE rules like Objectify. But to start with datastore you can also take at google tutorial GAE DATASTORE. If you need to learn the relationship with JPA and support for JDBC persistence Check this Link
What is the current status of grails and google app engine deployment. I am new to app engine but wonder worth exploring it. Some specific qns are
the latest plugin, which has high user rating, has any restrictions? or it work seamlessly with all gorm features
is there any issue with high startup time for grails application. How is it in real world scenario? (with a typical small and large scale application)
what about other grails plugins (like, shiro, joda time, nimble etc). I guess they wont play well. So using those libraries directly is the better option
If decided to give up goole-app as a deployment option, how easy to switch to a normal environment. The JPA support ensures the compatibility with other traditional DBs?
Not sure what else are major issues.. probably, this is the foundation for a good discussion.
thanks.
I got few good response from grails mailing list, and the conclusion shares the comment by David. see the thread here
Couple of relevant responses:
From Tomas Lin:
I would suggest looking into Gaelyk if you really want to build a
project on the App Engine. It is built from the ground up with the App
Engine as the target engine, so it can bypass problems like long
loadtimes due to Spring and Hibernate. The newly introduced plugin
mechanism guarantees that your Gaelyk applications can be extended in
a way guaranteed to work on GAE.
Gaelyk has it's own native entity persistence DSL, which is a little
cleaner that the JPA/JDO abstractions on top of the App Engine.
I currently see many HardDeadlineExceeded exceptions with the App
Engine and Grails. It is just not designed to work well with Spring
right now. Hopefully this will improve with the later releases of
Groovy, Grails and the Spring / Google partnership for GAE for
business, but I wouldn't consider Grails on GAE production ready.
Even with Gaelyk, there are reports of slow performance. So imagine
the difficulties that arise with the much bigger Grails stack.
The app-engine comes with it's own implementation of a user / security
management system based on GMail accounts. If you just want to provide
an admin / non-admin implementation, this is supported in the
appengine configuration. Cannot comment on Shiro.
Be aware that one of the major restrictions of the App Engine is the
inability to write a file, so even basic file uploading in Spring
becomes problematic since the default mechanism writes to a temporary
file. I would imagine that most of the plugins would not work out of
the box without digging into their code and changing it.
I think the biggest issue here is lack of support for native JDBC. JPA
is not as well supported as plain JDBC GORM, things like named queries
would probably not work out of the box without retrofitting. If you
want to use the latest and greatest parts of Grails, it might be
worthwhile to consider other hosting solutions.
From Aaron Eischeid
1.The GAE plugin and the JPA-GORM plugins combined do not get you all GORM features seamlessly. Though you should get basics like .save(), .delete(), and maybe .list() the dynamic finders etc. are going to be out (at least for now). I could be way off here, but I think most/all Hibernate dependent features are out or replaced by something else (since it relies on SQL under the hood and GAE doesn't currently have SQL based DB...) so for example any criteria builders are a no go. It is unclear to me how much of the dot drilling you can do on objects. For example, not sure if you could do something like:
def b = new Book()
def stores = b.authors.publishers.bookstores
One place I could use some pointers is how to use JPA in the domain classes. I am sure there is good info out there, but I just haven't found it yet.
unsure
grails plugins that include domain classes or manipulate your current domain classes are bound to have issues since you have to construct your domain classes differently to play nice with JPA which is necessary because Googles Datastore isn't quite like a relational DB. On the flip side. you can use Google's built in security so you shouldn't necessarily need plugins like Acegi or Shiro.
This probably boils down to the different levels of GORM that you can use in controllers and services and the different ways you define domain classes. Some refactoring seems inevitable unless JPA plays just as nice with SQL DB's as it does with Googles Datastore. If JPA can move like that then transferring should be easy, but by using JPA-GORM you give up some stuff you would probably want if you weren't benefiting from due to being on GAE.
Eager to hear what others have to say,
Aaron
In Google App Engine, I can use JDO to persist Java objects into the data store.
Can I also use JDO to turn the object into a byte[], so that I can put it into memcache or send it over HTTP?
Clarification: I want to serialize classes that I have already annotated for JDO persistence. Having to use another serialization mechanism seems to needlessly duplicate the effort, and also potentially tricky since JDO/DataNucleus uses bytecode manipulation on its classes to provide features like lazy-loading.
JDO persists objects to datastores. As part of that it may perform serialisation when a field is marked as "serialized". But that is serialised when stored in the datastore, and deserialised when retrieved from it. If you want to serialise something why not just do it yourself ... why should you need JDO for that?
It appears you can mark JDO objects as detachable, implement serializable and then when you want to cache them just call detach() and then cache as normal. I haven't tried this but from the discussion groups it seems to work.
There's also some more general discussion of enabling the JDO level 2 cache with memcache. Whether that works is sort of inconclusive but would be a nice solution. More about that: http://groups.google.com/group/google-appengine-java/browse_thread/thread/13cb942ceb97dc/3ab7518edf6a8bc6
I would like to develop a web-app requiring data persistence using GWT and GAE. As I understand it, my only (or at least by far the most convenient) option for data persistence is GAE's Datastore, using JDO or JPA annotated objects. I would also like to be able to send my objects back and forth client-server using GWT Remote Procedure Calls (RPC), therefore my objects must be able to "detach". However, GWT RPC serialization cannot handle detached JDO/JPA objects and it doesn't appear as though it will in the near future.
My question: what is the simplest and most direct solution to this? Being able to share the same objects client/server with server-side persistence would be extremely convenient.
EDIT
I should clarify that I still wish to use GWT RPC with GAE's Datastore. I am just looking for the best solution that would allow all these technologies to work together.
Try use http://gilead.sourceforge.net/
I've recently found Objectify, which is designed to be a replacement for JDO. Not much experience with it yet but its simpler to use than JDO, seems more lightweight, and claims to get around the need for DTOs with GWT, though I haven't tried that particular feature yet.
Ray Cromwell has a temporary hack up. I've tried it, and it works.
It forces you to use Transient instead of Detachable entities, because GWT can't serialize a hidden Object[] used by DataNucleus; This means that the objects you send to the client can't be inserted back into the datastore, you must retrieve the actual datastore object, and copy all the persistent fields back into it. Ray's method uses reflection to iterate over the methods, retrieve the getBean() and setBean() methods, and apply the entity setBean() with your transient gwt object's getBean().
You should strive to use JDO, the JPA isn't much more than a wrapper class for now. To use this hack, you must have both getter and setter methods for every persistent field, using PROPER getBean and setBean syntax for every "bean" field. Well, ALMOST PROPER, as it assumes all getters will start with "get", when the default boolean field use is "is".
I've fixed this issue and posted a comment on Ray's blog, but it's awaiting approval and I'm not sure if he'll post it. Basically, I implemented a #GetterPrefix(prefix=MethodPrefix.IS) annotation in the org.datanucleus package to augment his work.
In case it doesn't get posted, and this is an issue, email x_AT_aiyx_DOT_info Re: #GetterPrefix for JDO and I'll send you the fix.
Awhile ago I wrote a post Using an ORM or plain SQL?
This came up last year in a GWT
application I was writing. Lots of
translation from EclipseLink to
presentation objects in the service
implementation. If we were using
ibatis it would've been far simpler to
create the appropriate objects with
ibatis and then pass them all the way
up and down the stack. Some purists
might argue this is Bad™. Maybe so (in
theory) but I tell you what: it
would've led to simpler code, a
simpler stack and more productivity.
which basically matches your observation.
But of course that isn't an option with Google App Engine so you're pretty much stuck having a translation layer between client-side objects and your JPA entities.
JPA entities are quite rigid so they're not really appropriate for sending back and forth between the client anyway. Typically you want little bits from several entities when doing this (thus ending up with some sort of presentation-layer value object). That is your path forward.
Try this. It is a module for serializing GAE core types and send them to the GWT client.
You can consider using JSON. GWT has necessary API to parse & generate JSON string in the client side. You get a lot of JSON API for server side. I tried with google-gson, which is fine. It converts your JSON string to POJO model and viceversa. Hope this helps you providing a decent solution for your requirement
Currently, I use the DTO (DataTransferObject) pattern. Not necessarily as clean and plenty more boilerplate but GAE still requires a fair amount of boilerplate at current. ;)
I have a Domain Object mapped (usually) one-to-one with a DTO. When a client needs Domain info, a DAO(DataAccessObject) coughs up a DTO representation of the Domain object and sends that across the wire. When a DTO comes back, I hand the DAO the DTO which then updates all the appropriate Domain Objects.
Not as clean as being able to pass Domain Objects directly across the wire obviously but the limitations of GAE's JDO implementation and GWT's Serialization process means this is the cleanest way for me to handle this currently.
I believe Google's official answer for this is GWT 2.1 RequestFactory.
Given that you are using GWT and GAE, I'd suggest you stick to the official Google framework... I have a similar GWT / GAE based app and that's what I am doing.
By the way, setting up RequestFactory is a bit of pain in the ass. The current Eclipse plug-in doesn't include all the jars but I was able to find the help I needed, in Stackoverflow
I've been using Objectify as well, and I really like it. You still have to do some dancing around with pre/postLoad methods to translate e.g. Text to String and back.
since GWT ultimately compiles to JavaScript, for detached persistence it would need one of a few services available. the best known are HTML5 stores and Gears (both use SQLite!). of course, neither is widely deployed, so you'd have to convince your users to either use a modern browser or install a little-known plugin. be sure to degrade to a usable subset if the user doesn't comply
What about directly using Datastore API to load/store POJO domain objects?
It should be comparable to DTO approach, meaning e.g. that you have to manually handle all fields (if you don't use tricks like reflection-based automation) while it should give you more flexibility and full access to all Datastore features.