I'm currently migrating project's DAO classes from JDO implementation to Objectify V6.
The requirement I have, is to make sure that in a case of rollback it will be possible to load entities, which were saved by Objectify, with the old version of DAO.
In old code strings are stored as Text. And if I leave Text field in entity definition, Objectify puts it as a String to datastore (because there is no Text type any more).
Currently new DAO implementation is not backward compatible because of a ClassCastException which arise when JDO implementation casts String to Text type.
Is there a way to store Text type to datastore with Objectify V6?
I tried to use String instead of Text in entity definition and create a TranslatorFactory to make the conversion, but I wasn't able to find correct datastore Value implementation type.
public class StringTextTranslatorFactory implements TranslatorFactory<String, Text> {
#Override
public Translator<String, Text> create(TypeKey<String> tk, CreateContext ctx, Path path) {
return new Translator<String, Text>() {
#Override
public String load(Value<Text> node, LoadContext ctx, Path path) throws SkipException {
Text text = node.get();
return text != null ? text.getValue() : "";
}
#Override
public Value<Text> save(String pojo, boolean index, SaveContext ctx, Path path)
throws SkipException {
return ???;
}
};
}
}
Update
The project is using an implementation of JDO 2.3 for the App Engine Datastore. The implementation is based on version 1.0 of the DataNucleus Access Platform.
Data entity defined as the following:
#PersistenceCapable(identityType = IdentityType.APPLICATION)
public class CrmNote {
#PrimaryKey
#Persistent(valueStrategy = IdGeneratorStrategy.IDENTITY)
private Long id;
#Persistent
private Text note;
}
Stacktrace:
java.lang.ClassCastException: java.lang.String cannot be cast to com.google.appengine.api.datastore.Text
at com.timzon.snapabug.server.data.CrmNote.jdoReplaceField(CrmNote.java)
at com.timzon.snapabug.server.data.CrmNote.jdoReplaceFields(CrmNote.java)
at org.datanucleus.state.JDOStateManagerImpl.replaceFields(JDOStateManagerImpl.java:2772)
at org.datanucleus.state.JDOStateManagerImpl.replaceFields(JDOStateManagerImpl.java:2791)
at org.datanucleus.store.appengine.DatastorePersistenceHandler.fetchObject(DatastorePersistenceHandler.java:519)
at org.datanucleus.store.appengine.query.DatastoreQuery.entityToPojo(DatastoreQuery.java:649)
at org.datanucleus.store.appengine.query.DatastoreQuery.entityToPojo(DatastoreQuery.java:603)
at org.datanucleus.store.appengine.query.DatastoreQuery.access$300(DatastoreQuery.java:119)
at org.datanucleus.store.appengine.query.DatastoreQuery$6.apply(DatastoreQuery.java:783)
at org.datanucleus.store.appengine.query.DatastoreQuery$6.apply(DatastoreQuery.java:774)
at org.datanucleus.store.appengine.query.LazyResult.resolveNext(LazyResult.java:94)
at org.datanucleus.store.appengine.query.LazyResult.resolveAll(LazyResult.java:116)
at org.datanucleus.store.appengine.query.LazyResult.size(LazyResult.java:110)
at org.datanucleus.store.appengine.query.StreamingQueryResult.size(StreamingQueryResult.java:130)
at org.datanucleus.store.query.AbstractQueryResult.toArray(AbstractQueryResult.java:399)
at java.util.ArrayList.<init>(ArrayList.java:178)
at com.timzon.snapabug.server.dao.CrmNoteDAO.getOrderedCrmNotes(CrmNoteDAO.java:27)
Exception happens in auto-generated jdoReplaceField method which is added by JDO post-compilation "enhancement". I decompiled enhanced class and I see that datastore object is casted to Text type directly:
public void jdoReplaceField(int index) {
if (this.jdoStateManager == null) {
throw new IllegalStateException("state manager is null");
} else {
switch(index) {
case 0:
this.id = (Long)this.jdoStateManager.replacingObjectField(this, index);
break;
case 1:
this.note = (Text)this.jdoStateManager.replacingObjectField(this, index);
break;
default:
throw new IllegalArgumentException("out of field index :" + index);
}
}
}
So, if note field is saved in data store as a String, then in case of rollback a ClassCastException will be thrown.
There's no way to explicitly store a Text type with the Google-provided SDK that Objectify 6 uses; there is only StringValue. Text is not even in the jar.
However, I don't think this should matter. Ultimately both SDKs (the old appengine one and the new one) are just converting back and forth to protobuf structures. They are supposed to be compatible.
It's especially strange because the old low level API wrote strings into the Entity structure; Text was required only if the strings exceeded a certain length. So JDO should handle String. Do you have some sort of special annotation on your String field to force it to expect Text? What does that stacktrace look like?
Related
Mongodb is a no-schema document database, but in spring data, it's necessary to define entity class and repository class, like following:
Entity class:
#Document(collection = "users")
public class User implements UserDetails {
#Id private String userId;
#NotNull #Indexed(unique = true) private String username;
#NotNull private String password;
#NotNull private String name;
#NotNull private String email;
}
Repository class:
public interface UserRepository extends MongoRepository<User, String> {
User findByUsername(String username);
}
Is there anyway to use map not class in spring data mongodb so that the server can accept any dynamic JSON data then store it in BSON without any pre-class define?
First, a few insightful links about schemaless data:
what does “schemaless” even mean anyway?
“schemaless” doesn't mean “schemafree”
Second... one may wonder if Spring, or Java, is the right solution for your problem - why not a more dynamic tool, such a Ruby, Python or the Mongoshell?
That being said, let's focus on the technical issue.
If your goal is only to store random data, you could basically just define your own controller and use the MongoDB Java Driver directly.
If you really insist on having no predefined schema for your domain object class, use this:
#Document(collection = "users")
public class User implements UserDetails {
#Id
private String id;
private Map<String, Object> schemalessData;
// getters/setters omitted
}
Basically it gives you a container in which you can put whatever you want, but watch out for serialization/deserialization issues (this may become tricky if you had ObjectIds and DBRefs in your nested document). Also, updating data may become nasty if your data hierarchy becomes too complex.
Still, at some point, you'll realize your data indeed has a schema that can be pinpointed and put into well-defined POJOs.
Update
A late update since people still happen to read this post in 2020: the Jackson annotations JsonAnyGetter and JsonAnySetter let you hide the root of the schemaless-data container so your unknown fields can be sent as top-level fields in your payload. They will still be stored nested in your MongoDB document, but will appear as top-level fields when the ressource is requested through Spring.
#Document(collection = "users")
public class User implements UserDetails {
#Id
private String id;
// add all other expected fields (getters/setters omitted)
private String foo;
private String bar;
// a container for all unexpected fields
private Map<String, Object> schemalessData;
#JsonAnySetter
public void add(String key, Object value) {
if (null == schemalessData) {
schemalessData = new HashMap<>();
}
schemalessData.put(key, value);
}
#JsonAnyGetter
public Map<String, Object> get() {
return schemalessData;
}
// getters/setters omitted
}
I'm getting a ClassCastException (java.lang.Double cannot be cast to java.lang.Long) when retrieving multiple instances of a class in the datastore. The class has many values that are doubles and many that are longs. I'd like to view what is in the datastore and compare with the class properties to see if there is a mismatch. I tried the representationsOfProperty method found near the bottom of https://cloud.google.com/appengine/docs/java/datastore/metadataqueries#Java_Kind_queries, but my queries return null.
I have a class defined similar to the following:
#PersistenceCapable(identityType = IdentityType.APPLICATION)
public class Container
{
#PrimaryKey
#Persistent(valueStrategy = IdGeneratorStrategy.IDENTITY)
private Long containerID = null;
#Persistent
private Long extensionID = null;
#Persistent
private String homeUrl;
#Persistent
private Double containerScore;
}
I copied the code from the GAE page linked above. The only change I made was to convert 'key' to "key" since a string is requested and what is shown in the example isn't a character.
Collection<String> representationsOfProperty(DatastoreService ds,
String kind,
String property)
{
// Start with unrestricted non-keys-only property query
Query q = new Query(Entities.PROPERTY_METADATA_KIND);
// Limit to specified kind and property
q.setFilter(new FilterPredicate("__key__", Query.FilterOperator.EQUAL, Entities.createPropertyKey(kind, property)));
// Get query result
PreparedQuery pq = ds.prepare(q);
Entity propInfo = pq.asSingleEntity();
if( null == propInfo )
{
Collection<String> strs = new ArrayList<String>();
strs.add( "[ERROR: Invalid Query: " + pq.toString() + "]" );
return strs;
}
// Return collection of property representations
return (Collection<String>) propInfo.getProperty("property_representation");
}
I call that method with the following code:
DatastoreService datastore = DatastoreServiceFactory.getDatastoreService();
String prop = "containerID";
Collection<String> reps = representationsOfProperty( datastore, Container.class.toString(), prop );
Unfortunately, propInfo is always null. Any ideas on what I could try? Am I doing something wrong?
I just created a sample project to reproduce your issue. JDO is saving the entities in the Datastore, but not naming the primary key property according to the member variables of the PersistenceCapable class. Check your datastore viewer and grab a random Container entity to see what's happening.
If you need to do metadata queries on the id property in this manner, you should be aware that the ID/Name property is part of the key. Check this interesting talk to learn more about how the Datastore internals work (it's based on BigTable).
I have an entity in my app engine datastore. There's actually only one instance of this entity. I can see it in my admin console. Is it possible to add a new attribute to the entity via the admin console (using gql perhaps)?
Right now it looks something like:
Entity: Foo
Attributes: mName, mAge, mScore
and I'd like to add a new boolean attribute to this entity like "mGraduated" or something like that.
In the worst case I can write some code to delete the entity then save a new one, but yeah was just wondering.
Thanks
-------- Update ---------
Tried adding the new attribute to my class (using java) and upon loading from the datastore I get the following:
java.lang.NullPointerException:
Datastore entity with kind Foo and key Foo(\"Foo\") has a null property named mGraduated.
This property is mapped to com.me.types.Foo.mGraduated, which cannot accept null values.
This is what my entity class looks like, I just added the new attribute (mGraduated), then deployed, then tried loading the single entity from the datastore (which produced the above exception):
#PersistenceCapable
public class Foo
{
#PrimaryKey
private String k;
/** Some old attributes, look like the following. */
#Persistent
#Extension(vendorName = "datanucleus", key = "gae.unindexed", value="true")
private String mName;
...
/** Tried adding the new one. */
#Persistent
#Extension(vendorName = "datanucleus", key = "gae.unindexed", value="true")
private boolean mGraduated;
The only way to implement this is to use Boolean as the type for the new property..
Than in set method you can accept boolean value, that's no issue.
If you want the get method to also return boolean.. you also can, but be sure to check if the value is null and if so.. return default value (e.g. true)
so
private Boolean newProp = null; // can also assing default value .. e.g. true;
public void setNewProp(boolean val)
{
this.newProp = val;
}
public boolean getNewProp()
{
if(this.newProp == null)
return true; // Default value if not set
return this.newProp.booleanValue();
}
I recommend you not to migrate your data in this case - it can be very costly and can deplete your quota easily (read old data, create new, delete old = 3 operations for every entry in you data store)
You can't do this through the admin console, but you shouldn't have to delete the entity. Instead just update it- the Datastore does not enforce schemas for Kinds.
E.g., if Foo is a subclass of db.Model (Python), change your model subclass to include the new property; fetch the model instance (e.g., by its key), update the instance, including setting the value of the new field; and save the modified instance. Since you just have one instance this is easy. With many such instances to update you'd probably want to do this via task queue tasks or via a mapreduce job.
You have declared the new mGraduated field using the primitive type boolean, which cannot be null. The existing entity can't be loaded into the model class because it doesn't have this property. One option is to declare this property using the Boolean class, which can accept a null value.
The Admin Console only knows about properties in existing entities. You cannot use the Admin Console directly to create a new property with a name not used by any existing entities. (This is just a limitation of the Console. App code can do this easily.)
I'm using app engine datastore so I have entity like this.
#PersistenceCapable
public class Author {
#PrimaryKey
#Persistent(valueStrategy = IdGeneratorStrategy.IDENTITY)
#JsonProperty("id")
#JsonSerialize(using = JsonKeySerializer.class)
#JsonDeserialize(using = JsonKeyDeserializer.class)
private Key key;
....
}
When the model is sent to view, it will serialize the Key object as an Id value. Then, if I send data back from view I want to deserialize the Id back to Key object by using JsonKeyDeserializer class.
public class JsonKeyDeserializer extends JsonDeserializer<Key> {
#Override
public Key deserialize(JsonParser jsonParser, DeserializationContext deserializeContext)
throws IOException, JsonProcessingException {
String id = jsonParser.getText();
if (id.isEmpty()) {
return null;
}
// Here is the problem because I have several entities and I can't fix the Author class in this deserializer like this.
// I want to know what class is being deserialized at runtime.
// return KeyFactory.createKey(Author.class.getSimpleName(), Integer.parseInt(id))
}
}
I tried to debug the value in deserialize's parameters but I can't find the way to get the target deserialized class. How can I solve this?
You may have misunderstood the role of KeySerializer/KeyDeserializer: they are used for Java Map keys, and not as generic identifiers in database sense of term "key".
So you probably would need to use regular JsonSerializer/JsonDeserializer instead.
As to type: it is assumed that handlers are constructed for specific types, and no extra type information is passed during serialization or deserialization process: expected type (if handlers are used for different types) must be passed during construction.
When registering general serializers or deserializers, you can do this when implementing Module, as one of the arguments is type for which (de)serializer is requested.
When defining handlers directly for properties (like when using annotations), this information is available on createContextual() callback of interface ContextualSerializer (and -Deserializer), if your handler implements it: BeanProperty is passed to specify property (in this case field with annotation), and you can access its type. This information needs to be stored to be used during (de)serialization.
EDIT: as author pointed out, I actually misread the question: KeySerializer is the class name, not annotation.
As we know, in hibernate, configure the generator of a id to "uuid" , then hibernate will auto generate a UUID value to the id field when saving a new object.If configuring the generator to "assigned", the id must be assigned a value before saving a object.
And I found that if configuring the generator to uuid and assigning the id a value manually, the hibernate will change the value to a new UUID one.
My question is, when the generator is configured as uuid, how to manually assign a value to it?
PS: I use spring HibernateDaoSupport to save.
org.springframework.orm.hibernate3.support.HibernateDaoSupport.save(Ojbect obj)
Thanks!
If you need it only in rare special cases, the simpliest way is to issue INSERT queries in native SQL instead of using save().
Alternatively, you can customize generator to achieve the desired behaviour:
public class FallbackUUIDHexGenerator extends UUIDHexGenerator {
private String entityName;
#Override
public void configure(Type type, Properties params, Dialect d)
throws MappingException {
entityName = params.getProperty(ENTITY_NAME);
super.configure(type, params, d);
}
#Override
public Serializable generate(SessionImplementor session, Object object)
throws HibernateException {
Serializable id = session
.getEntityPersister(entityName, object)
.getIdentifier(object, session);
if (id == null)
return super.generate(session, object);
else
return id;
}
}
and configure Hibernate to use it by setting its fully qualified name as strategy.