How do you search on String fields in google appengine datastore - google-app-engine

I have a google appengine project that where I am storing users.
public void createUser(String loginId, String password) {
Entity e = new Entity("User");
e.setProperty("loginId", loginId);
e.setProperty("password", specialPasswordEncryptor(password));
ds.put(e);
}
Later I want to search for the User Entity by loginId when someone tries to login. But the problem is String properties are not indexed.
In the past I used add filter like this and it worked:
public Entity findUserByLoginId(String loginId) {
Query query = new Query("User").addFilter("loginId",Query.FilterOperator.EQUAL, loginId);
Entity eUser = datastore.prepare(query).asSingleEntity();
return eUser;
}
However addFilter is deprecated. I've tried other ways of doing this but keep running into the problem that String properties are not indexed so I cannot search on loginId.
Can someone point me in the right direction on this?
Thanks

String properties certainly can be indexed, unless they're longer than 1500 characters (javadoc).
To filter, you want to use setFilter(), like this:
Query query = new Query("User").setFilter(
new FilterPredicate("loginId", FilterOperator.EQUAL, loginId));

Related

query by object value inside array on firebase firestore [duplicate]

This is my structure of the firestore database:
Expected result: to get all the jobs, where in the experience array, the lang value is "Swift".
So as per this I should get first 2 documents. 3rd document does not have experience "Swift".
Query jobs = db.collection("Jobs").whereArrayContains("experience.lang","Swift");
jobs.get().addOnSuccessListener(new OnSuccessListener<QuerySnapshot>() {
#Override
public void onSuccess(QuerySnapshot queryDocumentSnapshots) {
//Always the queryDocumentSnapshots size is 0
}
});
Tried most of the answers but none worked out. Is there any way to query data in this structure? The docs only available for normal array. Not available for array of custom object.
Actually it is possible to perform such a query when having a database structure like yours. I have replicated your schema and here are document1, document2, and document3.
Note that you cannot query using partial (incomplete) data. You are using only the lang property to query, which is not correct. You should use an object that contains both properties, lang and years.
Seeing your screenshot, at first glance, the experience array is a list of HashMap objects. But here comes the nicest part, that list can be simply mapped into a list of custom objects. Let's try to map each object from the array to an object of type Experience. The model contains only two properties:
public class Experience {
public String lang, years;
public Experience() {}
public Experience(String lang, String years) {
this.lang = lang;
this.years = years;
}
}
I don't know how you named the class that represents a document, but I named it simply Job. To keep it simple, I have only used two properties:
public class Job {
public String name;
public List<Experience> experience;
//Other prooerties
public Job() {}
}
Now, to perform a search for all documents that contain in the array an object with the lang set to Swift, please follow the next steps. First, create a new object of the Experience class:
Experience firstExperience = new Experience("Swift", "1");
Now you can query like so:
CollectionReference jobsRef = rootRef.collection("Jobs");
jobsRef.whereArrayContains("experience", firstExperience).get().addOnCompleteListener(new OnCompleteListener<QuerySnapshot>() {
#Override
public void onComplete(#NonNull Task<QuerySnapshot> task) {
if (task.isSuccessful()) {
for (QueryDocumentSnapshot document : task.getResult()) {
Job job = document.toObject(Job.class);
Log.d(TAG, job.name);
}
} else {
Log.d(TAG, task.getException().getMessage());
}
}
});
The result in the logcat will be the name of document1 and document2:
firstJob
secondJob
And this is because only those two documents contain in the array an object where the lang is set to Swift.
You can also achieve the same result when using a Map:
Map<String, Object> firstExperience = new HashMap<>();
firstExperience.put("lang", "Swift");
firstExperience.put("years", "1");
So there is no need to duplicate data in this use-case. I have also written an article on the same topic
How to map an array of objects from Cloud Firestore to a List of objects?
Edit:
In your approach it provides the result only if expreience is "1" and lang is "Swift" right?
That's correct, it only searches for one element. However, if you need to query for more than that:
Experience firstExperience = new Experience("Swift", "1");
Experience secondExperience = new Experience("Swift", "4");
//Up to ten
We use another approach, which is actually very simple. I'm talking about Query's whereArrayContainsAny() method:
Creates and returns a new Query with the additional filter that documents must contain the specified field, the value must be an array, and that the array must contain at least one value from the provided list.
And in code should look like this:
jobsRef.whereArrayContainsAny("experience", Arrays.asList(firstExperience, secondExperience)).get().addOnCompleteListener(new OnCompleteListener<QuerySnapshot>() {
#Override
public void onComplete(#NonNull Task<QuerySnapshot> task) {
if (task.isSuccessful()) {
for (QueryDocumentSnapshot document : task.getResult()) {
Job job = document.toObject(Job.class);
Log.d(TAG, job.name);
}
} else {
Log.d(TAG, task.getException().getMessage());
}
}
});
The result in the logcat will be:
firstJob
secondJob
thirdJob
And this is because all three documents contain one or the other object.
Why am I talking about duplicating data in a document it's because the documents have limits. So there are some limits when it comes to how much data you can put into a document. According to the official documentation regarding usage and limits:
Maximum size for a document: 1 MiB (1,048,576 bytes)
As you can see, you are limited to 1 MiB total of data in a single document. So storing duplicated data will only increase the change to reach the limit.
If i send null data of "exprience" and "swift" as "lang" will it be queried?
No, will not work.
Edit2:
whereArrayContainsAny() method works with max 10 objects. If you have 30, then you should save each query.get() of 10 objects into a Task object and then pass them one by one to the to the Tasks's whenAllSuccess(Task... tasks).
You can also pass them directly as a list to Tasks's whenAllSuccess(Collection> tasks) method.
With your current document structure, it's not possible to perform the query you want. Firestore does not allow queries for individual fields of objects in list fields.
What you would have to do is create an additional field in your document that is queryable. For example, you could create a list field with only the list of string languages that are part of the document. With this, you could use an array-contains query to find the documents where a language is mentioned at least once.
For the document shown in your screenshot, you would have a list field called "languages" with values ["Swift", "Kotlin"].

JDO and Cloud SQL update objects with makePersistentAll

I am using google cloud SQL with JDO. When I try to use the JDO PersistenceManager to store new objects with a new key it works fine, however I get an error when I try to update entities already inserted in the db:
com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Duplicate entry '503062001-43661003' for key 'PRIMARY'
The key is indeed duplicated, but I want to update that object.
Is it possible to do this with the PersistentManager.makePersistentAll() method or in an another way that avoids to write the UPDATE query manually?
More details:
The object I am trying to persist is defined like this:
PersistenceCapable(table = "xxx")
public class XXX {
#PrimaryKey
#Index(name = "xxx_id")
private Long userId;
#PrimaryKey
#Index(name = "xxx_idx")
#Column(length = 128)
private String otherId;
...
}
If you want to update the object then you retrieve the object first, and update it (either in the same txn, or detach it and update it whilst detached). All of that would be described in the JDO spec
For example https://db.apache.org/jdo/pm.html and page down to "Update Objects"

Add field to App Engine-hosted database

I'm currently developing a mobile application who uses a Google App Engine-hosted web service.
But i'm facing an issue. I just want to add a field in one my database's table.
App Engine doesn't use classic SQL syntax, but GQL. So i cannot use the ALTER TABLE statement. How can i do this with GQL ? I looked for a solution on the web, but there's not a lot of help.
public MyEntity() {
}
#Id
#GeneratedValue(strategy=GenerationType.IDENTITY)
private Key idStation;
private String name;
private double longitude;
private double latitude;
private java.util.Date dateRefresh = new Date(); //the field i want to add in DB
So, now when i create a "MyEntity" object, it should add the "dateRefresh" field into the database... I create my object like this:
MyEntity station = new MyEntity();
station.setName("test");
station.setLatitude(0);
station.setLongitude(0);
station.setDateRefresh(new Date("01/01/1980"));
DaoFactory.getStationDao().addStation(station);
addStation method:
#Override
public MyEntity addStation(MyEntity station) {
EntityManager em = PersistenceManager.getEntityManagerFactory().createEntityManager();
try {
em.getTransaction().begin();
em.persist(station);
em.getTransaction().commit();
} finally {
if(em.getTransaction().isActive()) em.getTransaction().rollback();
em.close();
}
return station;
}
The field "dateRefresh" is never created into my DB...
Someone to help me please ?
Thanks in advance
Just add another field to your data structure, maybe providing a default clause, and that's all. For example, if you have a UserAccount:
class UserAccount(db.Model):
user = db.UserProperty()
user_id = db.StringProperty()
you may easily add:
class UserAccount(db.Model):
user = db.UserProperty()
user_id = db.StringProperty()
extra_info = db.IntegerProperty(default=0)
timezone = db.StringProperty(default="UTC")
and let it go.
While the datastore kinda mimics tables, data is stored on a per entity basis. There is no schema or table.
All you need to do is update your model class, and new entities will be saved with the structure (fields) of the new entity.
Old entities and indexes, however, are not automatically updated. They still have the same fields as they had when they were originally written to the datastore.
There's two ways to do this. One is to make sure your code can handle situations where your new properties are missing, ie make sure no exceptions are thrown, or handle the exceptions properly when you're missing the properties.
The second way is to write a little function (usu a mapreduce function) to update every entity with appropriate or null values for your new properties.
Note that indexes are not updated unless the entity is written. So if you add a new indexed property, old entities won't show up when you query for the new property. In this case, you must use the second method and update all the entities in the datastore so that they are indexed.

Playframework Siena Filtering and Ordering

This is my first question on any of these websites so pardon my unprofessionalism.
I use playframework with SIENA module (with GAE) and I came accross the following problem:
Given 3 entities:
public class Meeting extends Model{
#Id
public Long id;
public String place;
#Owned
Many<MeetingUser> users;
.
.
.
}
public class User extends Model{
#Id
public Long id;
public String firstName;
public String lastName;
#Owned
Many<MeetingUser> meetings;
.
.
.
}
public class MeetingUser extends Model{
#Id
public Long id;
public Meeting meeting;
public User user;
.
.
.
public User getUser(){
return Model.all(User.class).filter("id", user).get();
}
public Meeting getMeeting(){
return Model.all(Meeting.class).filter("id", meeting).get();
}
}
For instance I am listing a meeting and all their users:
public static void meetingInfo(Long meetingId){
Meeting meeting = Models.all(Meeting.class).filter("id",meetingId);
List<MeetingUser> meetingusers = meeting.asList();
List<User> users = new ArrayList<User>();
for(MeetingUser mu: meetingusers){
users.add(mu.getUser());
}
render(users);
}
This is done(is there any better way here?) however when it comes to filtering (especially dynamic filtering for many many fields) I can not use the Query's filter method on the MeetingUser as I need to filter on a MeetingUser's field's field (firstName). The same problem arise for ordering. I need the solution for both problems.
I hope my problem is clear and I appreciate any kind of help here.
Remember that you are in GAE which is a NoSQL DB.
So you can't do Join request as in RDBMS.
Yet, this is not really the pb you have so this was just to be sure you are aware of it ;)
So if you want to find the person having given firstname in a given meeting, can you try the following:
List<MeetingUser> meetingusers = meeting.users.asQuery().filter("firstname", "XXX");
(you can also order)
Nevertheless, knowing that you can't join, remember that you can't write a query searching for a meeting in which there are users whose firstname is XXX as it would require some joins and it doesn't exist in GAE. In this case, you need to change your model following NoSQL philosophy but this is another subject
regards
Let's try to give a way to do what you want...
Your relation is a Many-to-Many which is always the worst case :)
You want to filter Meeting by User's firstname.
It requires a join request which is not possible in GAE. In this case, you must change your model by denormalizing it (sometimes use redundancy also) and manage the join by yourself. Actually, you must do the job of the RDBMS by yourself. It seems overkill but in fact, it's quite easy. The only drawback is that you must perform several requests to the DB. NoSQL means No Schema (& No Join) so there are a few drawbacks but it allows to scale and to manage huge data load... it depends on your needs :)
The choice you did to create the MeetingUser which is a "joined" table and a kind of denormalization is good in GAE because it allows to manage the join yourself.
Solution:
// fetch users by firstname
List<User> users = users.all().filter("firstName", "John").fetch();
// fetch meetingusers associated to these users (verify the "IN" operator works because I didn't use that for a long time and don't remember if it works with this syntax)
List<MeetingUser> meetingusers = MeetingUser.all().filter("user IN", users);
// now you must fetch the whole meeting because in MeetingUser, only the Meeting ID is stored (other fields are Null or O)
List<Meeting> meetings = new ArrayList<Meeting>()
for(MeetingUsers mu:meetingusers) {
meetings.add(meetingusers.meeting);
}
// use the batch feature to fetch all objects
Meeting.batch(Meeting.class).get(meetings);
// you have your meetings
Hope this helps!

GQL query for <missing>

When you change data models on the app engine to add new properties those entries without a certain property are listed with the value <missing> in the online data viewer.
What I'm wondering is how can I write a query to find those entries?
There is no direct way to query for older entities with missing attribute, but you can design data model upfront to support this. Add a version attribute to each model class. Version should have a default value, which is increased every time model class is changed and deployed. This way you will be able to query entities by version number.
There's no way to query the datastore for entities that don't have a given property. You need to iterate over all the entities and check each one - possibly by using the mapreduce API.
Or you could create a script to stick null in there for all current items that don't have that property using the low level datastore API, so then you can query on null.
I had this issue and that's how I solved it. The rough code would look like:
DatastoreService datastore = DatastoreServiceFactory.getDatastoreService();
Query query = new Query("JDOObjectType");
List<Entity> results = datastore.prepare(query).asList(FetchOptions.Builder.withLimit(9999));
for (Entity lObject : results) {
Object lProperty = lObject.getProperty("YOUR_PROPERTY");
if (lProperty == null) {
lObject.setProperty("YOUR_PROPERTY", null);
datastore.put(lProperty);
}
}
}

Resources