I am noticing a weird(?) behavior when I am creating a large number of entities (about 1000) in my datastore.
I have a REST method which creates 1000 entities of same kind. I am using
datastore.put(entityObject);
to create entities. When I check the trace timeline I see this:
If I check details of every subsequent put call it shows this:
And there are a ton of these put calls! Even the 'Insights' tab says:
Here is my entity class:
import com.googlecode.objectify.Key;
import com.googlecode.objectify.annotation.Entity;
import com.googlecode.objectify.annotation.Id;
import com.googlecode.objectify.annotation.Index;
import com.sm.task.entity.Task;
#Entity(name = "TMS.Task")
public class TaskEntity {
#Id
private String id;
private String title;
private String description;
#Index
private String status;
public Task toTask() {
return new Task(
id,
title,
description,
status);
}
public Key<TaskEntity> key() {
return Key.create(TaskEntity.class, id);
}
public static Key<TaskEntity> keyFor(String id) {
return Key.create(TaskEntity.class, id);
}
public String getId() {
return id;
}
public void setId(String id) {
this.id = id;
}
public String getTitle() {
return title;
}
public void setTitle(String title) {
this.title = title;
}
public String getDescription() {
return description;
}
public void setDescription(String description) {
this.description = description;
}
public String getStatus() {
return status;
}
public void setStatus(String status) {
this.status = status;
}
}
I didn't find any setting which causes or controls this behavior. Can anyone please help me understand the reason of this issue and how to get rid of it?
Instead of making 1,000 putcalls to store 1,000 entities, you should make 2 put calls with 500 entities in each (just pass a list of entities to be saved). This is way faster and more efficient.
Related
I have a DynamoDB table with a primary key (id : integer) and secondary key (dateTo : String). I've made a Class that utilizes DynamoDBMapper:
#DynamoDBTable(tableName="MyItems"
public class MyItemsMapper {
private int id;
private String dateTo;
private String name;
#DynamoDBHashKey(attributeName="id")
public void setId(int id) { this.id = id; }
public int getId() { return id; }
#DynamoDBAttribute(attributeName="dateTo")
public void setDateTo(String dateTo) { this.dateTo = dateTo; }
public String getDateTo() { return dateTo; }
#DynamoDBAttribute(attributeName="name")
public void setName(String name { this.name = name; }
public String getName() { return name; }
public boolean saveItem(MyItemsMapper item) {
try {
DynamoDBMapper mapper = new DynamoDBMapper(client); //<-- This connects to the DB. This works fine.
item.setId(generateUniqueNumber()); //<-- This generates a unique integer. Also seems to work fine.
mapper.save(item);
logger.info("Successfully saved item. See info below.");
logger.info(item.toString());
return true;
} catch (Exception e) {
logger.error("Exception while trying to save item: " + e.getMessage());
e.printStackTrace();
return false;
}
}
}
I then have a manager class that uses the bean above, like so:
public class MyManager {
public boolean recordItem(
int id,
String dateTo,
String name,
) {
MyItemsMapper myItemsMapper = new MyItemsMapper();
myItemsMapper.setId(id);
myItemsMapper.setDateTo(dateTo);
myItemsMapper.setName(name);
myItemsMapper.saveItem(myItemsMapper);
}
}
I am running the manager class in a JUnit test:
public class MyManagerTest {
#Test
public void saveNewItemTest() {
MyManager myManager = new MyManager();
myManager.recordItem(1234567, "2018-01-01", "Anthony");
}
}
When I use the saveItem method above via my manager by running my JUnit test, I get the following error:
com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBMappingException: MyItemsMapper; no mapping for HASH key
Not really sure what it's pertaining to, as I definitely have a primary key for my table and my secondary key always has a value as well.
How do I get this to work?
More Info:
It's worth noting that I can record data into my DynamoDB table via the Item object. If I do the below, my data gets recorded into the database:
DynamoDB dynamoDB = new DynamoDBClient().connectToDynamoDB(); //<--
Connection. Works fine.
Table table = dynamoDB.getTable("MyItems");
item.withPrimaryKey("id", 1234567);
item.withString("dateTo", "2018-01-01");
item.withString("name", "Anthony");
PutItemOutcome outcome = table.putItem(item);
However, I'm trying to use DynamoDBMapper because I'm reading that it is a more organized, better way to access data.
Im not sure if this is causing the problem, but you are creating the myItemsMapper object, then passing a reference to this object to itself.
I would suggest removing your saveItem method. The MyItemsMapper class should be a plain old java object. Then make MyManager like this
public class MyManager {
public boolean recordItem(
int id,
String dateTo,
String name,
) {
MyItemsMapper myItemsMapper = new MyItemsMapper();
myItemsMapper.setId(id);
myItemsMapper.setDateTo(dateTo);
myItemsMapper.setName(name);
DynamoDBMapper mapper = new DynamoDBMapper(client);
mapper.save(myItemsMapper);
}
}
If you particularly want to keep the saveItem method make it like this
public boolean saveItem() {
try {
DynamoDBMapper mapper = new DynamoDBMapper(client);
mapper.save(this);
logger.info("Successfully saved item. See info below.");
logger.info(this.toString());
return true;
} catch (Exception e) {
logger.error("Exception while trying to save item: " + e.getMessage());
e.printStackTrace();
return false;
}
}
And then in MyManager do
MyItemsMapper myItemsMapper = new MyItemsMapper();
myItemsMapper.setId(id);
myItemsMapper.setDateTo(dateTo);
myItemsMapper.setName(name);
myItemsMapper.saveItem();
I have two entities as below and when i try to persist "Category" the "Tip" object list does not get persisted .I noticed that in my DAO class that I was able to see the category object with tipsForCategory list of size 1 but when i try to retrieve after persisting I am able to see only Category details and tipsForCategory comes as empty list.
#Entity
public class Category {
#GeneratedValue
#Id
public Long id;
#Column
public String categoryName;
#OneToMany(mappedBy = "category",cascade = {CascadeType.ALL})
public List<Tip> tipsForCategory;
public Long getId() { return id; }
public String getCategoryName() {
return categoryName;
}
public void setCategoryName(String categoryName) {
this.categoryName = categoryName.toLowerCase();
}
public void addTip(Tip tip) {
if(!tipsForCategory.contains(tip)) {
tipsForCategory.add(tip);
}
}
public List<Tip> getTipsForCategory() {
return tipsForCategory;
}
}
Code for Tip Entity
#Entity
public class Tip {
#Id
#GeneratedValue(strategy = GenerationType.IDENTITY)
public Key key;
#Column
public String tipDescription;
#ManyToOne(cascade = {CascadeType.ALL})
public Category category;
public String getTipDescription() {
return tipDescription;
}
public void setTipDescription(String tipInformation) {
this.tipDescription = tipInformation;
}
}
Code for persisting in my DAO
#Override
#Transactional
public void save(Category category) {
EntityManager localEntityManager=entityManager.getEntityManagerFactory().createEntityManager();
EntityTransaction transaction=localEntityManager.getTransaction();
try {
transaction.begin();
localEntityManager.persist(category);
localEntityManager.flush();
transaction.commit();
}catch (Exception e) {
e.printStackTrace();
localEntityManager.close();
}
}
My retrieval method is
#Override
public CategoryDTO findCategory(Long categoryId) throws FixitException{
CategoryDTO categoryDTO=null;
Category category=categoryDAO.findById(categoryId);
if(category!=null) {
categoryDTO=new CategoryDTO(category);
}
return categoryDTO;
}
#Override
public List<TipDTO> retrieveTips(Long categoryId) throws FixitException{
List<TipDTO> tips=null;
try {
CategoryDTO category = findCategory(categoryId);
if (category != null) {
tips = category.getTipsForCategory();
}
}
catch(Exception e)
{
throw new FixitException(FixitConstants.TIP_RETRIEVAL_ERROR+categoryId,e.getCause());
}
return tips;
}
Looks like the problem was with lazy fetch I just resolved the same.In my categoryDAO.findById(..) code I had to add an additional line to retrieve the tips as below
#Override
public Category findById(Long categoryId) {
Category category=null;
try {
TypedQuery<Category> findByCategoryId = entityManager.createQuery("Select cat from Category cat where cat.id=:categoryId",Category.class);
category=findByCategoryId.setParameter("categoryId", categoryId).getSingleResult();
}
catch (Exception e)
{
e.printStackTrace();
}
*** int tipsSize=category.getTipsForCategory().size();***
return category;
}
I am trying to get the user's friends list from Facebook.
The problem seems to be the Javabean...
FBUser fbuser = new Gson().fromJson(jsonStr, FBUser.class);
public class FBUser implements Serializable {
private static final long serialVersionUID = -3154429420153433117L;
private String id;
private String name;
private String email;
private Friends friendsList = new Friends();
private FBUser() { }
public String getId() {
return id;
}
public void setId(String id) {
this.id = id;
}
public String getName() {
return name;
}
public void setName(String name) {
this.name = name;
}
public String getEmail() {
return email;
}
public void setEmail(String email) {
this.email = email;
}
public List<Data> getFriendsList() {
return friendsList.getData();
}
public static class Friends implements Serializable {
private static final long serialVersionUID = 6991758772193514527L;
private List<Data> data;
private Friends() { }
public List<Data> getData() {
return data;
}
public void setData(List<Data> data) {
this.data = data;
}
public class Paging implements Serializable {
private static final long serialVersionUID = 1689816298710621080L;
private String next;
private Paging() { }
public String getNext() {
return next;
}
public void setNext(String next) {
this.next = next;
}
}
}
public class Data implements Serializable {
private static final long serialVersionUID = -5008541658519841090L;
private String id;
private String name;
private Data() { }
public String getId() {
return id;
}
public void setId(String id) {
this.id = id;
}
public String getName() {
return name;
}
public void setName(String name) {
this.name = name;
}
}
}
Json:
json: {"id":"10861234","name":"Whatever","email":"whatever\u0040gmail.com","friends":{"data":[{"name":"Someone","id":"10861234"},{"name" ...43"}],"paging":{"next":"https:\/\/graph.facebook.com\/10861234\/friends..."}}}
The fields ID, Name and Email I can retrieve succesfully... but the friendsList is null... =(
Maybe it is the way I am trying to get it from the nested class, any suggestions on that?
There is no friendsList in your JSON (or, there's no friends in your Java class - whichever way you'd like to look at it). Gson silently ignores anything in the JSON that is not present in your classes.
You have a field friends whose value is an object. That object has a field data which is an array of objects and a field paging which is another object.
You need to write Java classes that match that structure. You're ... close.
In your FBUser class change:
private Friends friendsList = new Friends();
to:
private Friends friends = new Friends();
or:
#SerializedName("friends")
private Friends friendsList = new Friends();
Then in your Friends class you need to add:
private Paging paging = new Paging();
Also note that you don't have to initialize these values unless you specifically don't want them to be non-null when using these classes elsewhere.
im trying to build a google app engine projekt with JPA, JAX-RS and JAX-B. My POST and GET Methods work, but my DELETE method doesn't delete the data.
Resource
#DELETE
#Path("card/{id}")
public void deleteCardById (#PathParam ("id") Long id) {
Service.removeCard(id);
}
Service
public static void removeCard(Long id) {
EntityManager em = EMFService.get().createEntityManager();
Card emp = findCard(id);
if (emp != null) {
em.remove(emp);
}
em.close();
}
public static Card findCard(Long id) {
EntityManager em = EMFService.get().createEntityManager();
Card card = em.find(Card.class, id);
em.close();
return card;
}
Entity
#XmlRootElement
#Entity
public class Card {
#Id
#GeneratedValue(strategy = GenerationType.IDENTITY)
Long id;
String begriff;
String tabu1;
String tabu2;
String tabu3;
public Card(String begriff, String tabu1, String tabu2, String tabu3) {
super();
Begriff = begriff;
Tabu1 = tabu1;
Tabu2 = tabu2;
Tabu3 = tabu3;
}
public Card() {
}
public Long getId() {
return id;
}
public void setId(Long id) {
this.id = id;
}
public String getBegriff() {
return Begriff;
}
public void setBegriff(String begriff) {
Begriff = begriff;
}
public String getTabu1() {
return Tabu1;
}
public void setTabu1(String tabu1) {
Tabu1 = tabu1;
}
public String getTabu2() {
return Tabu2;
}
public void setTabu2(String tabu2) {
Tabu2 = tabu2;
}
public String getTabu3() {
return Tabu3;
}
public void setTabu3(String tabu3) {
Tabu3 = tabu3;
}
#Override
public String toString() {
return "Card [Begriff=" + Begriff + ", Tabu1=" + Tabu1 + ", Tabu2="
+ Tabu2 + ", Tabu3=" + Tabu3 + "]";
}
When i Debug the app it gives the correct Object to the remove function. But it just don't remove the data ...
You mean you're using v1 of the GAE JPA plugin, and you don't bother putting a transaction around your remove (so the remove is delayed until the next transaction ... which never happens)?
Obviously you could either put a transaction around the remove, or better still you use v2 of the GAE JPA plugin
I was facing similar issue too. the JPA delete actually deletes the entity in the datastore,but it doesn't delete the entity from the JPA Cache.. You page is actually using the JPA Cached result list to display..
The way I used to resolve the issue is to have the JPA Cache cleared every time after a delete.
Sample Code would be something like this:
EM.getTransaction().begin();
EM.remove(current_record);
EM.getTransaction().commit();
EM.getEntityManagerFactory().getCache().evictAll();
ok i think i should write it like this
*edit the problem was the findCard function, i think because of the secone instance of the EntityManager. I chnaged it without using this method to this and now it works.
public static void removeCard(Long id) {
EntityManager em = EMFService.get().createEntityManager();
EntityTransaction tx = em.getTransaction();
try {
tx.begin();
Card card = em.find(Card.class, id);
if (card != null) {
em.remove(card);
}
tx.commit();
} finally {
if (tx.isActive()) {
tx.rollback();
}
em.close();
}
}
I have some entities that I wish to do an ancestor query on and filter a parameter by the ">" operator.
The entity in question inherits another object(I don't think this should matter). Below are my entity classes:
#Indexed
public class ValidatedObject {
public Long timeCreated=System.currentTimeMillis();
public Long timeUpdated=System.currentTimeMillis();
public Long getTimeUpdated() {
return timeUpdated;
}
public void setTimeUpdated(Long timeUpdated) {
this.timeUpdated = timeUpdated;
}
public Boolean validated=false;
#Unindexed
public String validatedID;
#Unindexed
private Long validatedTime;
#Unindexed
private String creatorID;
public Long getTimeCreated() {
return timeCreated;
}
public void setTimeCreated(Long timeCreated) {
this.timeCreated = timeCreated;
}
public boolean isValidated() {
return validated;
}
public void setValidated(boolean validated) {
this.validated = validated;
}
public String getValidatedID() {
return validatedID;
}
public void setValidatedID(String validatedID) {
this.validatedID = validatedID;
}
public Long getValidatedTime() {
return validatedTime;
}
public void setValidatedTime(Long validatedTime) {
this.validatedTime = validatedTime;
}
public String getCreatorID() {
return creatorID;
}
public void setCreatorID(String creatorID) {
this.creatorID = creatorID;
}
}
#Cached
#Entity
public class PersonnelInfo extends ValidatedObject{
#Id
public String keyName;
#Parent Key<Department> department;
private Long fdID;
#Unindexed
private String userKeyName;
private String firstName;
private String lastName;
#Unindexed
private String address,city,county,state;
#Unindexed
private String cellPhone,homePhone,otherPhone;
public PersonnelInfo(){
}
public PersonnelInfo(String email){
keyName=email;
}
#Override
public Long getTimeUpdated() {
return timeUpdated;
}
#Override
public void setTimeUpdated(Long time) {
timeUpdated=time;
}
}
My query code is as follows:
Query<PersonnelInfo> q = ofy.query(PersonnelInfo.class).ancestor(tmp).filter("timeUpdated >", lastSync);
I am getting the "no matching index found" error everytime. The query works fine without the filter. Some of the entities are missing the "timeUpdated" field because I changed the schema. There are some entities that have been created after the schema change with timeUpdated values and they are not returned. Also I can do a GQL query on the datastore viewer like this:
select * where FROM PersonnelInfo timeUpdated > 0
and I am returned entities, which makes me believe the index is created. What am I doing wrong here? Any help would be appreciated!
You need a multi-property index defined in your datastore-indexes.xml. It should be added to datastore-indexes-auto.xml when you run the query in dev mode, but the result should look like this:
<datastore-index kind="PersonnelInfo" ancestor="true">
<property name="timeUpdated" direction="asc"/>
</datastore-index>