I'm getting started with Google App Engine, and I'm using Objectify. How do I create a root entity in the data store, but err if it already exists? I didn't find anything built in for this (e.g. DatastoreService.put() and therefore ofy().save() will overwrite an existing entity instead of err). The simple technique I am used to is to do this in a transaction:
Err if already exists
Save
However, that is not idempotent; it would err in step 1 if the transaction executes twice. Here is the best I've come up with so far, not in a transaction:
Err if already exists
Save
Fetch
Err if it's not the data we just created
Or, if I don't mind two requests to save the same data both succeeding, I can skip the initial lookup:
Fetch
Report success if it's the same data we are about to create
Err if already exists, but is not the same data we are about to create
Save
That is doable, but it gets a little bulky to accomplish what I thought would be a very simple operation. Is there a better way?
This should guarantee consistent behavior:
final String id = // pick the unique id
final long txnId = // pick a uuid, timestamp, or even just a random number
ofy().transact(new VoidWork() {
public void vrun() {
Thing th = ofy().load().type(thing.class).id(id).now();
if (th != null) {
if (th.getTxnId() == txnId)
return;
else
throw ThingAlreadyExistsException();
}
th = createThing(id, txnId);
ofy().save().entity(th);
}
});
Related
I am using Java Spring Boot and OptaPlanner to generate a timetable with almost 20 constraints. At the initial generation, everything works fine. The score showed by the OptaPlanner logging messages matches the solution received, but when I want to resume the generation, the solution contains a lot of problems (like the constraints are not respected anymore) although the generation starts from where it has stopped and it continues initializing or finding a best solution.
My project is divided into two microservices: one that communicates with the UI and keeps the database, and the other receives data from the first when a request for starting/resuming the generation is done and generates the schedule using OptaPlanner. I use the same request for starting/resuming the generation.
This is how my project works: the UI makes the requests for starting, resuming, stopping the generation and getting the timetable. These requests are handled by the first microservice, which uses WebClient to send new requests to the second microservice. Here, the timetable will be generated after asking for some data from the database.
Here is the method for starting/resuming the generation from the second microservice:
#PostMapping("startSolver")
public ResponseEntity<?> startSolver(#PathVariable String organizationId) {
try {
SolverConfig solverConfig = SolverConfig.createFromXmlResource("solver/timeTableSolverConfig.xml");
SolverFactory<TimeTable> solverFactory = new DefaultSolverFactory<>(solverConfig);
this.solverManager = SolverManager.create(solverFactory);
this.solverManager.solveAndListen(TimeTableService.SINGLETON_TIME_TABLE_ID,
id -> timeTableService.findById(id, UUID.fromString(organizationId)),
timeTable -> timeTableService.updateModifiedLessons(timeTable, organizationId));
return new ResponseEntity<>("Solving has successfully started", HttpStatus.OK);
} catch(OptaPlannerException exception) {
System.out.println("OptaPlanner exception - " + exception.getMessage());
return utils.generateResponse(exception.getMessage(), HttpStatus.CONFLICT);
}
}
-> findById(...) method make a request to the first microservice, expecting to receive all data needed by constraints for generation (lists of planning entities, planning variables and all other useful data)
public TimeTable findById(Long id, UUID organizationId) {
SolverDataDTO solverDataDTO = webClient.get()
.uri("http://localhost:8080/smart-planner/org/{organizationId}/optaplanner-solver/getSolverData",
organizationId)
.retrieve()
.onStatus(HttpStatus::isError, error -> {
LOGGER.error(extractExceptionMessage("findById.fetchFails", "findById()"));
return Mono.error(new OptaPlannerException(
extractExceptionMessage("findById.fetchFails", "")));
})
.bodyToMono(SolverDataDTO.class)
.block();
TimeTable timeTable = new TimeTable();
/.. populating all lists from TimeTable with the one received in solverDataDTO ../
return timeTable;
}
-> updateModifiedLessons(...) method send to the first microservice the list of all generated planning entities with the corresponding planning variables assigned
public void updateModifiedLessons(TimeTable timeTable, String organizationId) {
List<ScheduleSlot> slots = new ArrayList<>(timeTable.getScheduleSlotList());
List<SolverScheduleSlotDTO> solverScheduleSlotDTOs =
scheduleSlotConverter.convertModelsToSolverDTOs(slots);
String executionMessage = webClient.post()
.uri("http://localhost:8080/smart-planner/org/{organizationId}/optaplanner-solver/saveTimeTable",
organizationId)
.header(HttpHeaders.CONTENT_TYPE, MediaType.APPLICATION_JSON_VALUE)
.body(Mono.just(solverScheduleSlotDTOs), SolverScheduleSlotDTO.class)
.retrieve()
.onStatus(HttpStatus::isError, error -> {
LOGGER.error(extractExceptionMessage("saveSlots.savingFails", "updateModifiedLessons()"));
return Mono.error(new OptaPlannerException(
extractExceptionMessage("saveSlots.savingFails", "")));
})
.bodyToMono(String.class)
.block();
}
I would probably start by making sure that the solution you save to the DB after the first run of startSolver() is the same (in terms of Java equality), including the assignments of planning variables to values, as the solution you retrieve via findById() at the beginning of the second run.
We have a requirement to create a kind of user session. Our front end is react and backend is .net core 6 api and db is postgres.
When 1 user clicks on a delete button , he should not be allowed to delete that item when another user is already using that item and performing some actions.
Can you guys suggest me an approach or any kind of service that is available to achieve this. Please help
I would say dont make it too complicated. A simple approach could be to add the properties 'BeingEditedByUserId' and 'ExclusiveEditLockEnd' (datetime) to the entity and check these when performing any action on this entity. When an action is performed on the entity, the id is assigned and a timeslot (for example 10 minutes) would be assigned for this user. If any other user would try to perform an action, you block them. If the timeslot is expired anyone can edit again.
I have had to do something similar with Java (also backed by a postgres db)
There are some pitfalls to avoid with a custom lock implementation, like forgetting to unlock when finished, given that there is not guarantee that a client makes a 'goodbye, unlock the table' call when they finish editing a page, they could simply close the browser tab, or have a power outage... Here is what i decided to do:
Decide if the lock should be implemented in the API or DB?
Is this a distributed/scalable application? Does it run as just a single instance or multiple? If multiple, then you can not (as easily) implement an API lock (you could use something like a shared cache, but that might be more trouble than it is worth)
Is there a record in the DB that could be used as a lock, guaranteed to exist for each editable item in the DB? I would assume so, but if the app is backed by multiple DBs maybe not.
API locking is fairly easy, you just need to handle thread safety as most (if not all) REST/SOAP... implementations are heavily multithreaded.
If you implement at the DB consider looking into a 'Row Level Lock' which allows you to request a lock on a specific row in the DB, which you could use as a write lock.
If you want to implement in the API, consider something like this:
class LockManager
{
private static readonly object writeLock = new();
// the `object` is whatever you want to use as the ID of the resource being locked, probably a UUID/GUID but could be a String too
// the `holder` is an ID of the person/system that owns the lock
Dictionary<object, _lock> locks = new Dictionary<object, _lock>();
_lock acquireLock(object id, String holder)
{
_lock lok = new _lock();
lok.id = id;
lok.holder = holder;
lock (writeLock)
{
if (locks.ContainsKey(id))
{
if (locks[id].release > DateTime.Now)
{
locks.Remove(id);
}
else
{
throw new InvalidOperationException("Resource is already locked, lock held by: " + locks[id].holder);
}
}
lok.allocated = DateTime.Now;
lok.release = lok.allocated.AddMinutes(5);
}
return lok;
}
void releaseLock(object id)
{
lock (writeLock)
{
locks.Remove(id);
}
}
// called by .js code to renew the lock via ajax call if the user is determined to be active
void extendLock(object id)
{
if (locks.ContainsKey(id))
{
lock (writeLock)
{
locks[id].release = DateTime.Now.AddMinutes(5);
}
}
}
}
class _lock
{
public object id;
public String holder;
public DateTime allocated;
public DateTime release;
}
}
This is what i did because it does not depend on the DB or client. And was easy to implement. Also, it does not require configuring any lock timeouts or cleanup tasks to release locked items with expired locks on them, as that is taken care of in the locking step.
The database I am using is a global variable, initialized reading the file sql-repo.db:
const dbFile = "sql-repo.db"
var globalDB *LocalDB
type LocalDB struct {
Path string
handle *sql.DB
}
func InitSqlDB(dbDir string) error {
if globalDB != nil {
return nil
}
db := LocalDB{Path: filepath.Join(dbDir, dbFile)}
var err error
db.handle, err = sql.Open("sqlite3", db.Path)
if err != nil {
return err
}
globalDB = &db
return nil
}
From time to time, I will have an updated version of this database that I can download and store in dbDir.
Ideas that I have:
Use ATTACH DATABASE sql-repo.db AS dbMain to attach a copy of the first database and use it by default.
When I have my new .db file, I attach it as well ATTACH DATABASE sql-repo-new.db AS dbNew
Then I detach dbMain and rename dbNew to dbMain
Simply change the address my globalDB is pointing to:
const newDBFile = "sql-repo-new.db"
func PullNewDB(dbDir string) error {
db := LocalDB{Path: filepath.Join(dbDir, newDBFile)}
var err error
db.handle, err = sql.Open("sqlite3", db.Path)
if err != nil {
return err
}
globalDB = &db
return nil
}
How can I or how should I update my globalDB with the new version/file as I want to avoid any interference if the clients that I have in my code are connected to the DB and are querying it ?
Should I attach a sync.RWMutex to my LocalDB struct and then lock/unlock it when I do the update ?
Or should I use a channel to ask every client to stop querying the DB ?
Thank you for any help / advice / suggestion !
Or you can start a separate go routine or process that syncs your old database with the new file. Do an insert or update on all rows from new to old, then delete on missing rows. If it's all done in a single transaction, all queries will either read all old or new data without ever blocking.
An additional benefit is the separation of concerns, your application code doesn't get clustered with update logic, and in the case, the new file is corrupted, the update transaction errors out and no harm is done.
I currently have an application which uses the activities/places and an AsyncDataProvider.
Right now, everytime the activity loads up - it uses the request factory to retrieve the data (currently not a lot but will get very large coming up here soon) and passes it to the View to update the DataGrid. Before it is updated it is filtered based on a search box.
Right now - I have implemented updating the DataGrid as follows: (this code isn't the prettiest)
private void updateData() {
final AsyncDataProvider<EquipmentTypeProxy> provider = new AsyncDataProvider<EquipmentTypeProxy>() {
#Override
protected void onRangeChanged(HasData<EquipmentTypeProxy> display) {
int start = display.getVisibleRange().getStart();
int end = start + display.getVisibleRange().getLength();
final List<EquipmentTypeProxy> subList = getSubList(start, end);
end = (end >= subList.size()) ? subList.size() : end;
if (subList.size() < DATAGRID_PAGE_SIZE) {
updateRowCount(subList.size(), true);
} else {
updateRowCount(data.size(), true);
}
updateRowData(start, subList);
}
private List<EquipmentTypeProxy> getSubList(int start, int end) {
final List<EquipmentTypeProxy> filteredEquipment;
if (searchString == null || searchString.equals("")) {
if (data.isEmpty() == false && data.size() > (end - start)) {
filteredEquipment = data.subList(start, end);
} else {
filteredEquipment = data;
}
} else {
filteredEquipment = new ArrayList<EquipmentTypeProxy>();
for (final EquipmentTypeProxy equipmentType : data) {
if (equipmentType.getName().contains(searchString)) {
filteredEquipment.add(equipmentType);
}
}
}
return filteredEquipment;
}
};
provider.addDataDisplay(dataGrid);
}
Ultimately - what I would like to do is only load up the necessary data at first (the default page size in this application is 25).
Unfortunately, to my current understanding, with Google App Engine there is no order to any of the Id's (one entry has an ID of 3 the next has an entry of 4203).
What I'm wondering, what is the best way to go about retrieving a subset of data from Google App Engine when using Objectify?
I was looking into using Offset and limit but another stack overflow post (http://stackoverflow.com/questions/9726232/achieve-good-paging-using-objectify) basically said this is inefficient.
The best information I've found is the following link (http://stackoverflow.com/questions/7027202/objectify-paging-with-cursors). The answer here says to use Cursors but also says this is inefficient. I'm also using Request Factory so I will have to store the Cursor in my user Session (if that is incorrect please let me know).
Currently since there isn't likely to be a lot of data (maybe 200 rows total for the next few months) I am just pulling back the entire set to the client as a temporary hack - I know this is the worst way to do it but would like to get input to the best way to do it before wasting my time implementing another hack solution. I am worried currently as it seems every single post i've read on doing this makes it seem like there's not really a solid way to do this.
What i am also thinking about doing - currently my searching / page loading is lightning fast because all the data is already on the client side. I use a KeyUpEvent handler in the search box to filter the data - i don't think there is any way to keep this speed by making a call to the server - is there any accepted solution to this problem?
Thank you very much
Go with Cursors. They are as efficient as it gets - cursor stores the point where last query ended and continues from there. The answer you linked actually does not discuss efficiency of cursors vs offset. (there is a comment that is wrong)
You can use limit with Cursors - it does not affect efficiency.
Also, Cursors can be serialized via cursor.toWebSafeString() and sent to client via RPC. This way you do not need to save them in session. Actually you can also use them as fragment identifier (aka history token in GWT parlance) - this way a certain "page" of your result set can be bookmarked.
(Offset is "inefficient" because it actually loads, and charges you, for all entities upto offset+limit, bit it only returns limit entities)
OTOH, if you already know the query parameters when the page is loaded, then just do the query at page generation time, instead invoking it via RPC. Also, if you have a small set of data (<1000) you could just preload all entity IDs s part of page html.
Say you retrieve a set of records from the datastore (something like: select * from MyClass where reserved='false').
how do i ensure that another user doesn't set the reserved is still false?
I've looked in the Transaction documentation and got shocked from google's solution which is to catch the exception and retry in a loop.
Any solution that I'm missing - it's hard to believe that there's no way to have an atomic operation in this environment.
(btw - i could use 'syncronize' inside the servlet but i think it's not valid as there's no way to ensure that there's only one instance of the servlet object, isn't it? same applies to static variable solution)
Any idea on how to solve?
(here's the google solution:
http://code.google.com/appengine/docs/java/datastore/transactions.html#Entity_Groups
look at:
Key k = KeyFactory.createKey("Employee", "k12345");
Employee e = pm.getObjectById(Employee.class, k);
e.counter += 1;
pm.makePersistent(e);
'This requires a transaction because the value may be updated by another user after this code fetches the object, but before it saves the modified object. Without a transaction, the user's request will use the value of counter prior to the other user's update, and the save will overwrite the new value. With a transaction, the application is told about the other user's update. If the entity is updated during the transaction, then the transaction fails with an exception. The application can repeat the transaction to use the new data'
Horrible solution, isn't it?
You are correct that you cannot use synchronize or a static variable.
You are incorrect that it is impossible to have an atomic action in the App Engine environment. (See what atomic means here) When you do a transaction, it is atomic - either everything happens, or nothing happens. It sounds like what you want is some kind of global locking mechanism. In the RDBMS world, that might be something like "select for update" or setting your transaction isolation level to serialized transactions. Neither one of those types of options are very scalable. Or as you would say, they are both horrible solutions :)
If you really want global locking in app engine, you can do it, but it will be ugly and seriously impair scalability. All you need to do is create some kind of CurrentUser entity, where you store the username of the current user who has a global lock. Before you let a user do anything, you would need to first check that no user is already listed as the CurrentUser, and then write that user's key into the CurrentUser entity. The check and the write would have to be in a transaction. This way, only one user will ever be "Current" and therefore have the global lock.
Do you mean like this:
public void func(Data data2) {
String query = "select from " + objectA.class.getName()
+ " where reserved == false";
List<objectA> Table = (List<objectA>) pm.newQuery(
query).execute();
for (objectA row : Table)
{
Data data1 = row.getData1();
row.setWeight(JUtils.CalcWeight(data1, data2));
}
Collections.sort(Table, new objectA.SortByWeight());
int retries = 0;
int NUM_RETRIES = 10;
for (int i = 0; i < Table.size() ; i++)
{
retries++;
pm.currentTransaction().begin(); // <---- BEGIN
ObjectA obj = pm.getObjectById(Table.get(i).class, Table.get(i).getKey());
if (obj .getReserved() == false) // <--- CHECK if still reserved
obj.setReserved(true);
else
break;
try
{
pm.currentTransaction().commit();
break;
}
catch (JDOCanRetryException ex)
{
if (j == (NUM_RETRIES - 1))
{
throw ex;
}
i--; //so we retry again on the same object
}
}
}