Is this the correct way to catch PermannentTask Failure? (https://cloud.google.com/appengine/articles/deferred)
def do_something_with_key(k):
entity = k.get()
# Do something with entity
entity.put()
k = ndb.Key('MyModel', 123)
try:
deferred.defer(do_something_with_key, k, _countdown=60)
except PermanentTaskFailure:
#catch here
Or do I need to put try/except inside do_something_with_key function
The PermanentTaskFailure exception is typically raised when the task executes or attempts to execute, so you won't catch it when you create the task. Unless, maybe, if you do that from another task execution handler, but in that case it'd be for the enqueueing task, not for the task being enqueued. Or maybe if enqueuing itself has trouble? Not sure - I never got it in such case.
So, at best, I think you might be able to catch it from do_something_with_key(). But you won't be able to catch it for all cases - for example if the task code fails to execute - the exception is caught by the deferred library code itself, see an example in Issue with appengine deferred tasks, execution throws unknown error.
I was able to catch it (again, probably not for all cases), but that was after I switched from the deferred library to directly using the push tasks (which is what the deferred library uses under the hood).
The article you referenced discusses PermanentTaskFailure in the context of your handler code (intentionally) raising the exception to signal to the deferred library that it shouldn't enqueue yet another copy of the task - which is what it does by default if the task execution fails (based on its return code for the request), until the maximum number of retries is reached.
Related
I checked the docs and stackoverflow but didn't find exactly a suiting approach.
E.g. this post seems very close: Dispatch a blocking service in a Reactive REST GET endpoint with Quarkus/Mutiny
However, I don't want so much unneccessary boilerplate code in my service, at best, no service code change at all.
I generally just want to call a service method which uses entity manager and thus is a blocking action, however, want to return a string to the caller immidiately like "query started" or something. I don't need a callback object, it's just a fire and forget approach.
I tried something like this
#NonBlocking
#POST
#Produces(MediaType.TEXT_PLAIN)
#Path("/query")
public Uni<String> triggerQuery() {
return Uni.createFrom()
.item("query started")
.call(() -> service.startLongRunningQuery());
}
But it's not working -> Error message returned to the caller:
You have attempted to perform a blocking operation on a IO thread. This is not allowed, as blocking the IO thread will cause major performance issues with your application. If you want to perform blocking EntityManager operations make sure you are doing it from a worker thread.",
I actually expected quarkus takes care to distribute the tasks accordingly, that is, rest call to io thread and blocking entity manager operations to worker thread.
So I must using it wrong.
UPDATE:
Also tried an proposed workaround that I found in https://github.com/quarkusio/quarkus/issues/11535 changing the method body to
return Uni.createFrom()
.item("query started")
.emitOn(Infrastructure.getDefaultWorkerPool())
.invoke(()-> service.startLongRunningQuery());
Now I don't get an error, but service.startLongRunningQuery() is not invoked, thus no logs and no query is actually sent to db.
Same with (How to call long running blocking void returning method with Mutiny reactive programming?):
return Uni.createFrom()
.item(() ->service.startLongRunningQuery())
.runSubscriptionOn(Infrastructure.getDefaultWorkerPool())
Same with (How to run blocking codes on another thread and make http request return immediately):
ExecutorService executor = Executors.newFixedThreadPool(10, r -> new Thread(r, "CUSTOM_THREAD"));
return Uni.createFrom()
.item(() -> service.startLongRunningQuery())
.runSubscriptionOn(executor);
Any idea why service.startLongRunningQuery() is not called at all and how to achieve fire and forget behaviour, assuming rest call handled via IO thread and service call handled by worker thread?
It depends if you want to return immediately (before your startLongRunningQuery operation is effectively executed), or if you want to wait until the operation completes.
If the first case, use something like:
#Inject EventBus bus;
#NonBlocking
#POST
#Produces(MediaType.TEXT_PLAIN)
#Path("/query")
public void triggerQuery() {
bus.send("some-address", "my payload");
}
#Blocking // Will be called on a worker thread
#ConsumeEvent("some-address")
public void executeQuery(String payload) {
service.startLongRunningQuery();
}
In the second case, you need to execute the query on a worker thread.
#POST
#Produces(MediaType.TEXT_PLAIN)
#Path("/query")
public Uni<String> triggerQuery() {
return Uni.createFrom(() -> service.startLongRunningQuery())
.runSubscriptionOn(Infrastructure.getDefaultWorkerPool());
}
Note that you need RESTEasy Reactive for this to work (and not classic RESTEasy). If you use classic RESTEasy, you would need the quarkus-resteasy-mutiny extension (but I would recommend using RESTEasy Reactive, it will be way more efficient).
Use the EventBus for that https://quarkus.io/guides/reactive-event-bus
Send and forget is the way to go.
In my java code, I am processing huge amount of data. So I moved the code as servlet to Cron Job of App Engine. Some days it works fine. After the amount of the data increases, the cron job is not working and shows the following error message.
2012-09-26 04:18:40.627
'ServletName' 'MethodName': Inside SQLExceptionjava.sql.SQLRecoverableException:
Connection is already in use.
I 2012-09-26 04:18:40.741
This request caused a new process to be started for your application, and thus caused
your application code to be loaded for the first time. This request may thus take
longer and use more CPU than a typical request for your application.
W 2012-09-26 04:18:40.741
A problem was encountered with the process that handled this request, causing it to
exit. This is likely to cause a new process to be used for the next request to your
application. If you see this message frequently, you may be throwing exceptions during
the initialization of your application. (Error code 104)
How to handle this problem?
This exception is typical when a single connection is shared between multiple threads. This will in turn happen when your code does not follow the standard JDBC idiom of acquiring and closing the DB resources in the shortest possible scope in the very same try-finally block like so:
public Entity find(Long id) throws SQLException {
Connection connection = null;
// ...
try {
connection = dataSource.getConnection();
// ...
} finally {
// ...
if (connection != null) try { connection.close(); } catch (SQLException ignore) {}
}
return entity;
}
Your comment on the question,
#TejasArjun i used connection pooling with servlet Init() method.
doesn't give me the impression that you're doing it the right way. This suggests that you're obtaining a DB connection in servlet's init() method and reusing the same one across all HTTP requests in all HTTP sessions. This is absolutely not right. A servlet instance is created/initialized only once during webapp's startup and reused throughout the entire remaining of the application's lifetime. This at least confirms the exception you're facing.
Just rewrite your JDBC code according the standard try-finally idiom as demonstrated above and you should be all set.
See also:
Is it safe to use a static java.sql.Connection instance in a multithreaded system?
I have a similar code below that I'm trying to figure out transaction locking:
DAOT.repeatInTransaction(new Transactable() {
#Override
public void run(DAOT daot)
{
Points points = daot.ofy().find(Points.class, POINTS_ID);
// do something with points
takes_a_very_long_time_delay(); // perhaps 10 secs
daot.ofy().put(points);
}
});
The code above is executed from within a Java servlet. The operation is expected to work for 10 seconds for example. In between that time, I have a test that will invoke another servlet that will delete a Points entity, I was expecting that the delete operation would fail or at least delete the entity after the transaction above has finished.
However the entity was deleted during the period that the above code is executing. In my real application, I added exception handling to throw exception when trying to access or edit a entity that does not exist.
From there, the application is throwing "Entity not found" exception just after I executed the servlet that will delete the Entity in the code above.
Although I am using GAE Transactions already, however I think I am still missing something that's why my test fails.
Code for the delete Transaction from withing the Delete servlet:
DAOT.repeatInTransaction(new Transactable() {
#Override
public void run(DAOT daot)
{
Points points = daot.ofy().find(Points.class, POINTS_ID);
daot.ofy().delete(points);
}
});
How can I ensure that a new operation like a delete for a entity will wait until the current operation is happening on a entity during a transaction?
App Engine uses optimistic concurrency, not locking. That is, a transaction on a group of entities will not prevent other processes from modifying those entities while the transaction runs. Instead, when the transaction attempts to commit, it will check if any modifications were made while the transaction was executing, and if it has, discard any changes and run your function again from the beginning.
I assume you use objectify to work with datastore.
First you need to make sure daot.ofy() returns objectify instance with explicit transaction set (ObjectifyFactory.beginTransaction()) instead of ObjectifyFactory.begin(). Then make sure you use the same objectify instance for both find() and delete() calls (as well as for find()/put pairs).
I have an erlang module with behaviour gen_server.
Now, I have:
init(_Args) ->
erlang:send_after(?PROCESS_STATE_INTERVAL,self(),processState),
{ok, []}.
and
handle_info(processState, _State)->
{ok, NewState} = gen_server:call(self(), {updateLvls}), %works fine, tested
timer:send_after(?PROCESS_STATE_INTERVAL,self(),processState),
{noreply, NewState}.
When I start it with something like {ok, Test}=gen_server:start_link({local,challenge_manager},challenge_manager,[],[]). after a few seconds I get ** exception error: {timeout,{gen_server,call,[<0.329.0>,{updateLvls}]}}
Am I doing something wrong??
You cannot call your own gen_server from within itself. That will result in a dead lock (which is what you see). The server process is busy handling your first request (since you haven't returned yet) and will queue the second request (which is made from the handling of the first), thus dead lock.
To solve this, either create a library function which both handle_call and handle_info uses, or take a look at the reply/2 function which will let you do asynchronous replies (if you return {noreply, ...} from your handle_call function).
Here is my code:
class PublishPhotosHandler(webapp.RequestHandler):
for argument in files_arguments:
taskqueue.add(url='/upload', params={'key': key})
class UploadWorker(webapp.RequestHandler):
def post(self):
key = self.request.get('key')
result = urlfetch.fetch(...)
# how to return there an error, so the task will be retried?
If a task fails to execute (by returning any HTTP status code outside of the range 200–299), App Engine retries the task until it succeeds. By default, the system gradually reduces the retry rate to avoid flooding your application with too many requests, but schedules retry attempts to recur at a maximum of once per hour until the task succeeds.
raising any exception will cause a non-2XX status code, therefore raising any exception will cause the the task to be queued up again and retried.