Comparing strings in a JDO query fails when value contains a "Comma" - google-app-engine

I am attempting to check for an existing string using a JDO query, in my attempt to prevent the insertion of a duplicate string.
My query to check for an existing string works fine, unless the two strings I am comparing have a comma in the value. If the commas exists, the comparison bombs using "==".
For example, if I query to see if "Architecture" exists, I get the right result (Horrray!).
If I attempt to see if "Architecture, Engineering, and Drafting" exists, and it does, the query comes back and says an identical value does not exist (Boo!).
The code I'm using is as follows:
Called from the RPC
public void addCommas()
{
final Industry e = new Industry();
e.setIndustryName("Architecture, Engineering, and Drafting");
persist(e);
}
public void addNoCommas()
{
final Industry e = new Industry();
e.setIndustryName("Architecture");
persist(e);
}
Persist Operation
private void persist(Industry industry)
{
if (industryNameExists(industry.getIndustryName()))
{
return;
}
final PersistenceManager pm = PMF.get().getPersistenceManager();
pm.currentTransaction().begin();
try
{
pm.makePersistent(industry);
pm.flush();
pm.currentTransaction().commit();
} catch (final Exception ex)
{
throw new RuntimeException(ex);
} finally
{
if (pm.currentTransaction().isActive())
{
pm.currentTransaction().rollback();
}
pm.close();
}
}
Query
public static boolean industryNameExists(final String industryName)
{
final PersistenceManager pm = PMF.get().getPersistenceManager();
Query q = null;
q = pm.newQuery(Industry.class);
q.setFilter("industryName == industryNameParam");
q.declareParameters(String.class.getName() + " industryNameParam");
final List<Industry> industry = (List<Industry>) q.execute(industryName.getBytes());
boolean exists = !industry.isEmpty();
if (q != null)
{
q.closeAll();
}
pm.close();
return exists;
}
JDO Entity
#PersistenceCapable(detachable = "true")
public class Industry implements StoreCallback
{
#NotNull(message = "Industry Name is required.")
private String industryName;
#Persistent(valueStrategy = IdGeneratorStrategy.IDENTITY)
#PrimaryKey
private Key key;
public Industry()
{
super();
}
public Key getIndustryKey()
{
return key;
}
public String getIndustryName()
{
return industryName;
}
#Override
public void jdoPreStore()
{
if (industryName != null)
{
industryName = industryName.trim();
}
}
public void setIndustryName(final String industryName)
{
this.industryName = industryName;
}
}
Any thoughts on a resolution or pinpointing an oversight would be very much appreciated.
Cheerio.

So you're calling industryNameExists("Architecture, Engineering, and Drafting") and trying to match a JDO with industryName exactly equal "Architecture, Engineering, and Drafting"?
Assuming you don't have any typo or space difference the only thing suspect is the getBytes(). Try the following:
Query q = pm.newQuery(Industry.class, "this.industryName == :industryNameParam");
List<Industry> industry = (List<Industry>) q.execute(industryName);
You can also try variation filters like "this.industryName.equalsIgnoreCase(:industryNameParam)" and "this.industryName.startWith(:industryNameParam)" to troubleshoot.
If it still does not work, try logging the SQL generated for review and compare with a hand-written query that works.

Related

Trouble reading from Storage

Ive had a deeper look at the storage system now, but I just can't get it to work properly. The data seems to be saved, the saved data is definitely not empty and I am still getting a java.lang.NullPointerException.
So here is my Class i want to saved and later read to the Storage.
Band is a trivial Class with name and genre String.
public class Band implements Externalizable{
String name;
String genre;
public Band(String bnd, String gen){
this.name = bnd;
this.genre = gen;
}
public Band() {
}
public String getName(){
return this.name;
}
public String getGenre() { return this.genre; }
public void setName(String name) {
this.name = name;
}
public void setGenre(String genre) {
this.genre = genre;
}
#Override
public String toString(){
String s;
s = this.name;
return s;
}
public int getVersion(){
return 1;
}
#Override
public void externalize(DataOutputStream out) throws IOException {
Util.writeUTF(name, out);
Util.writeUTF(genre, out);
}
#Override
public void internalize(int version, DataInputStream in) throws IOException {
name = Util.readUTF(in);
genre = Util.readUTF(in);
}
public String getObjectId(){
return "Band";
}
}
BandList is just a Class with an ArrayList the Bands are added to.
public class BandList implements Externalizable{
List <Band> bandList;
public List getBandList(){
return this.bandList;
}
public BandList(){
bandList = new ArrayList<Band>();
}
public void setBandList(List<Band> bandList) {
this.bandList = bandList;
}
public int getVersion(){
return 1;
}
#Override
public void externalize(DataOutputStream out) throws IOException {
Util.writeObject(bandList, out);
}
#Override
public void internalize(int version, DataInputStream in) throws IOException {
bandList = (ArrayList<Band>) Util.readObject(in);
}
public String getObjectId(){
return "BandList";
}
}
So, in some other class where the user filled in some TextFields, the band gets added to the BandList and therefore I call saveListsToStorage();
public void saveListsToStorage(){
Storage.getInstance().clearStorage();
Storage.getInstance().writeObject("Bands", bandList);
}
So now, when starting the App, the App is looking for Data in the Storage, if found, it will return the BandList Object stored, otherwise it will return a new BandList Object.
Before that, I regist the Classes with
Util.register("Band", Band.class);
Util.register("BandList", BandList.class);
and finally call this method in the main at the beginning:
public BandList loadSavedBandList() {
String[] temp = Storage.getInstance().listEntries();
for (String s : temp) {
if (s.equals("Bands") == true) {
BandList tempBand = (BandList) Storage.getInstance().readObject("Bands");
return tempBand;
}
}
return new BandList();
}
After saving and then loading it throws the NullPointer exception and I have no clue why. I now used ArrayList instead of LinkedList as Shai told me and I implemented the Externalizable Interface as told in the guide.
Maybe some of you can tell me whats wrong here.
Edit:
Here is the Exception:
java.lang.NullPointerException
at com.rosscode.gehma.Menu_Form.updateBandContainer(Menu_Form.java:95)
at com.rosscode.gehma.Menu_Form.<init>(Menu_Form.java:73)
at com.rosscode.gehma.main.start(main.java:61)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.codename1.impl.javase.Executor$1$1.run(Executor.java:106)
at com.codename1.ui.Display.processSerialCalls(Display.java:1152)
at com.codename1.ui.Display.mainEDTLoop(Display.java:969)
at com.codename1.ui.RunnableWrapper.run(RunnableWrapper.java:120)
at com.codename1.impl.CodenameOneThread.run(CodenameOneThread.java:176)
Edit 2, Line 95:
if(bandList.getBandList().isEmpty() == false) {
for (int i = 0; i < bandList.getBandList().size(); i++) {
Button temp = new Button(bandList.getBandList().get(i).toString());
temp.addActionListener((e) ->
nextStep()
);
listCont.add(temp);
}
menuForm.revalidate();
}
NullPointerExceptions are pretty amazing things, because not only are they easy to figure out, they even show you the exact line where they happen. All you have to do is look at anything left of a . and check if it's null. As soon as you find it, you find the problem.
So let's look at your code. Sadly I don't know which of those lines is line 95 but I'm guessing it's the one starting with if.
That means either bandList is null or bandList.getBandList() returns null. A quick look into getBandList() shows that it won't be null, as you initialize the list in the constructor of your BandList class. Which only leaves bandList.
Sadly you didn't post where you got that from, but I'm gonna assume you just did something like:
BandList bandList = loadSavedBandList();
So let's look into that method. (btw: I say a bit more about where you have to look if you didn't do this at the end of this answer).
If you couldn't find your key, it would return a new BandList, which couldn't be null. So it has to find something, meaning the key exist but the value is null.
So let's take another step and look at how you save things in the first place.
public void saveListsToStorage(){
Storage.getInstance().clearStorage();
Storage.getInstance().writeObject("Bands", bandList);
}
Now "in some other class" doesn't really explain much. But since you read a null, it means you are writing a null here. Which would mean in this "other class", at least in this method, bandList = null.
With the code you gave, it's impossible for me to look deeper into this issue, but it seems like in "that other class", you got a class attribute named bandList but fail to put anything inside it.
Or maybe you saved everything right, but didn't call
BandList bandList = loadSavedBandList();
in some form or another before your if, which would also pretty much explain the problem. I mean... if you never load the bandList, it will obviously be null...

mapreduce fails with message "The request to API call datastore_v3.Put() was too large."

I am running a mapreduce job over 50 million User records.
For each user I read two other Datastore entities and then stream stats for each player to bigquery.
My first dry run (with streaming to bigquery disabled) failed with the following stacktrace.
/_ah/pipeline/handleTask
com.google.appengine.tools.cloudstorage.NonRetriableException: com.google.apphosting.api.ApiProxy$RequestTooLargeException: The request to API call datastore_v3.Put() was too large.
at com.google.appengine.tools.cloudstorage.RetryHelper.doRetry(RetryHelper.java:121)
at com.google.appengine.tools.cloudstorage.RetryHelper.runWithRetries(RetryHelper.java:166)
at com.google.appengine.tools.cloudstorage.RetryHelper.runWithRetries(RetryHelper.java:157)
at com.google.appengine.tools.pipeline.impl.backend.AppEngineBackEnd.tryFiveTimes(AppEngineBackEnd.java:196)
at com.google.appengine.tools.pipeline.impl.backend.AppEngineBackEnd.saveWithJobStateCheck(AppEngineBackEnd.java:236)
I have googled this error and the only thing I find is related to that the Mapper is too big to be serialized but our Mapper has no data at all.
/**
* Adds stats for a player via streaming api.
*/
public class PlayerStatsMapper extends Mapper<Entity, Void, Void> {
private static Logger log = Logger.getLogger(PlayerStatsMapper.class.getName());
private static final long serialVersionUID = 1L;
private String dataset;
private String table;
private transient GbqUtils gbq;
public PlayerStatsMapper(String dataset, String table) {
gbq = Davinci.getComponent(GbqUtils.class);
this.dataset = dataset;
this.table = table;
}
private void readObject(java.io.ObjectInputStream in) throws IOException, ClassNotFoundException {
in.defaultReadObject();
log.info("IOC reinitating due to deserialization.");
gbq = Davinci.getComponent(GbqUtils.class);
}
#Override
public void beginShard() {
}
#Override
public void endShard() {
}
#Override
public void map(Entity value) {
if (!value.getKind().equals("User")) {
log.severe("Expected a User but got a " + value.getKind());
return;
}
User user = new User(1, value);
List<Map<String, Object>> rows = new LinkedList<Map<String, Object>>();
List<PlayerStats> playerStats = readPlayerStats(user.getUserId());
addRankings(user.getUserId(), playerStats);
for (PlayerStats ps : playerStats) {
rows.add(ps.asMap());
}
// if (rows.size() > 0)
// gbq.insert(dataset, table, rows);
}
.... private methods only
}
The maprecuce job is started with this code
MapReduceSettings settings = new MapReduceSettings().setWorkerQueueName("mrworker");
settings.setBucketName(gae.getAppName() + "-playerstats");
// #formatter:off <I, K, V, O, R>
MapReduceSpecification<Entity, Void, Void, Void, Void> spec =
MapReduceSpecification.of("Enque player stats",
new DatastoreInput("User", shardCountMappers),
new PlayerStatsMapper(dataset, "playerstats"),
Marshallers.getVoidMarshaller(),
Marshallers.getVoidMarshaller(),
NoReducer.<Void, Void, Void> create(),
NoOutput.<Void, Void> create(1));
// #formatter:on
String jobId = MapReduceJob.start(spec, settings);
Well I solved this by backing to appengine-mapreduce-0.2.jar which was the one we had used before. The one used above was appengine-mapreduce-0.5.jar which actually turned out not to work for us.
When backing to 0.2 the console _ah/pipiline/list started to work again as well!
Anyone else that have encountered similar problem with 0.5?

Retrieve more than 1000 results Google App Engine Search API

I need to be able to retrieve all records available in the index. Seems like 1000 is the limit. Is there something else I can do?
I was also facing some similar issue in one of my projects so researched over internet and got one idea that instead of using the Search API, I created a workaround scenario. What I did is I have only one attribute in my table which needs to have a pattern based search. I am sharing my code as well here.
Objectify Entity Class
#Entity
public class NewsFeed {
#Id
#Index
private Long feedID;
private String title;
private Set<String> titleKeywords;
// getters and setter
}
Logic for storing the keyword in the same table. I have split all the title words for my entity into keywords, and stored them in a Set Object.
NewsFeed newsFeed = new NewsFeed();
newsFeed.setTitle(title);
newsFeed.setTitleKeywords(getKeywordsSet(newsTitle));
// save entity here
Method for extracting keywords from title(field to be searched)
public Set<String> getKeywordsSet(String title) {
Set<String> keywords = new HashSet<String>();
String titleNews = title.toLowerCase();
String[] array = titleNews.split(" ");
for (int i = 0; i < array.length; i++) {
// replacing all special characters here
String word = array[i].replaceAll("\\W", "");
keywords.add(word);
}
return keywords;
}
Listing all the feeds from our DB, and finally matching the parameter to be searched, by the logic below.
public List<NewsFeed> getFilterJsonArray(String param){
// Listing all the objects of entity
List<NewsFeed> list = newsFeedDao.listOrderedFeeds();
List<NewsFeed> matchedObject = new ArrayList<NewsFeed>();
for (NewsFeed newsFeed : list) {
/**
* main logic for pattern matched keywords
**/
if (isAnElementInSet(newsFeed.getTitleKeywords(), param.toLowerCase())) {
matchedObject.add(newsFeed);
}
}
return matchedObject;
}
public boolean isAnElementInSet(Set<String> keywords, String param) {
String []params = param.split(" ");
if (keywords.size() > 0) {
for(String splittedParam : params){
if (keywords.contains(splittedParam)) {
return true;
} else{
for (String keyword : keywords) {
if(keyword.contains(splittedParam)){
return true;
}
}
return false;
}
}
return true;
}else{
return false;
}
}
I know this cannot be the best solution for searching things but this solution worked very fine for me. I just shared it here so as to get improvements in this logic as well.

Persistence with EJB3 doesn't work

I have a method to save a new object in an EJB bean. This method is called, without error, but nothing changes in the database. I can't understand why.
Here is the code:
#Stateless(name = "Ar", mappedName = "ManagementBean")
public class ManagementBean implements IManagementBeanLocal, IManagementBeanRemote {
...
#Override
public int storeRawSms(String raw, String requestUid, String text, String service, boolean correctlyAnalysed, Date receivedTimestamp,
boolean toBeAnalysed, String phoneNumber) {
// Get phone number, create if it dosn't exist
PhoneNumber pn = getOrCreatePhoneNumberPrivate(phoneNumber);
// Create rawSMS
RawSms rawSms = new RawSms(raw, requestUid, text, service, correctlyAnalysed, receivedTimestamp, toBeAnalysed, pn);
// Store and return result
em.persist(rawSms);
int result = rawSms.getId();
em.flush();
em.clear();
return result;
}
...
And the caller:
#PersistenceContext private EntityManager em;
...
int rawSmsIs = bean.storeRawSms(raw, requestUid, message, service, false, new Date(), true, sender);
Do you have an idea?
I see that you inject a reference to the EntityManager in the client (not sure why), but I don't see it in the session bean (maybe simply because you did not include the line in your message). Is it possible that you forgot to use the annotation #PersistenceContext in your stateless session bean?
Also, be careful: depending on the JPA implementation you are using and the generation strategy for the ids, you should call flush() before calling getId(). Indeed, if you let the DB generate your IDs, then you need a flush() to have this happen before the method returns the value.
Thanks, the prposed solution worked!
I use the container-managed transactions like this:
#Stateless(name = "Ar", mappedName = "ManagementBean")
#TransactionManagement(TransactionManagementType.CONTAINER)
public class ManagementBean implements IManagementBeanLocal, IManagementBeanRemote {
....
#Override
#TransactionAttribute(TransactionAttributeType.REQUIRES_NEW)
public int storeRawSms(String raw, String requestUid, String text, String service, boolean correctlyAnalysed, Date receivedTimestamp, boolean toBeAnalysed, String phoneNumber) {
....
Thanks again!
It seems that your transaction never commited, so try changing transaction management:
#Stateless(name = "Ar", mappedName = "ManagementBean")
#TransactionManagement(TransactionManagementType.BEAN)
public class ManagementBean implements IManagementBeanLocal, IManagementBeanRemote {
#Resource
private UserTransaction utx;
#Override
public int storeRawSms(..) {
try {
utx.begin();
..
em.persist(rawSms);
int result = rawSms.getId();
utx.commit();
}
catch(Exception ex) {
//EXCEPTION HANDLING
utx.rollback();
}
}
}

'Invalid attempt to read when no data is present' - exception happens "sometimes" in Entity Framework

I get the above error sometimes during the read. The exception originates from ASP.NET SqlDataReader whenever you try to read data before calling the Read() method. Since EF does all these internally, I am wondering what else can cause this error. could it be network (or) db connectivity?
thanks
Additional Bounty Info (GenericTypeTea):
I've got the same error after upgrading to EF Code First RC (4.1):
"Invalid attempt to read when no data
is present"
This is the code in question:
using (var context = GetContext())
{
var query = from item in context.Preferences
where item.UserName == userName
where item.PrefName == "TreeState"
select item;
// Error on this line
Preference entity = query.FirstOrDefault();
return entity == null ? null : entity.Value;
}
The table structure is as follows:
Preference
{
Username [varchar(50)]
PrefName [varchar(50)]
Value [varchar(max)] Nullable
}
The table is standalone and has no relationships. This is the DbModelBuilder code:
private void ConfigurePreference(DbModelBuilder builder)
{
builder.Entity<Preference>().HasKey(x => new { x.UserName, x.PrefName });
builder.Entity<Preference>().ToTable("RP_Preference");
}
Exactly the same code works perfectly in CTP5. I'm guessing this is an RC bug, but any ideas of how to fix it would be appreciated.
This error occurs when there is a large amount of data in the RC release. The difference between the RC and CTP5 is that you need to specify the [MaxLength] property that contains a large amount of data.
Are you re-using contexts? I would guess this is happening as a result of something you are doing within GetContext
If GetContext() provides a stale context, in which the DataReader is closed/corrupted, I could see the above happening.
I cannot reproduce your problem on EF4.1 RC1.
POCO:
public class Preference
{
public string UserName { get; set; }
public string PrefName { get; set; }
public string Value { get; set; }
}
Context:
public class PreferenceContext : DbContext
{
public DbSet<Preference> Preferences {get;set;}
public PreferenceContext()
: base("Data Source=localhost;Initial Catalog=_so_question_ef41_rc;Integrated Security=SSPI;") {
}
protected override void OnModelCreating(DbModelBuilder modelBuilder)
{
ConfigurePreference(modelBuilder);
base.OnModelCreating(modelBuilder);
}
private void ConfigurePreference(DbModelBuilder builder)
{
builder.Entity<Preference>().HasKey(x => new { x.UserName, x.PrefName });
builder.Entity<Preference>().ToTable("RP_Preference");
}
}
My little Console App:
class Program
{
static void Main(string[] args)
{
string userName = "Anon";
for (int i = 0; i < 10000; i++)
{
var p = GetPreference(userName);
}
}
private static string GetPreference(string userName)
{
using (var context = new PreferenceContext())
{
var query = from item in context.Preferences
where item.UserName == userName
where item.PrefName == "TreeState"
select item;
// Error on this line
Preference entity = query.FirstOrDefault();
return entity == null ? null : entity.Value;
}
}
}
I do 10,000 reads, and no error. You will need to post more complete code to continue.
Increase the CommandTimeout on the context.
I had the same issue with EF4 - In my case I was (trying to) return the list of entities within the using{} section. This is the same as you are doing in your question:
return entity == null ? null : entity.Value;
} // end using
I moved the return to after the } and it worked.
I think I had the problem because the code was in a function which had already queried the database in another using block, I suspect the table was locking but not reporting the error, ending the using block before the return released the database lock.
Steve

Resources