I am able to create threads however creation of a Timer and scheduling it are eluding me. I am unsure why this is failing...and silently!
Any ideas?
You can use threads in GAE, but there are limitations:
You must create them via GAE provided ThreadManager.
On frontend instances threads can not outlive requests. This means that Timer started from a http request can not run TimerTask if the request already ended - I guess this is why it silently fails.
If you need to start a delayed task, try using DeferredTask (here's an example) with ETA setting.
Related
Maybe you can help me with my problem
I start spark job on google-dataproc through API. This job writes results on the google data storage.
When it will be finished I want to get a callback to my application.
Do you know any way to get it? I don't want to track job status through API each time.
Thanks in advance!
I'll agree that it would be nice if there was to either wait for or get a callback for when operations such as VM creation, cluster creation, job completion, etc finish. Out of curiosity, are you using one of the api clients (like google-cloud-java), or are you using the REST API directly?
In the mean time, there are a couple of workarounds that come to mind:
1) Google Cloud Storage (GCS) callbacks
GCS can trigger callbacks (either Cloud Functions or PubSub notifications) when you create files. You can create an file at the end of your Spark job, which will then trigger a notification. Or, just add a trigger for when you put an output file on GCS.
If you're modifying the job anyway, you could also just have the Spark job call back directly to your application when it's done.
2) Use the gcloud command line tool (probably not the best choice for web servers)
gcloud already waits for jobs to complete. You can either use gcloud dataproc jobs submit spark ... to submit and wait for a new job to finish, or gcloud dataproc jobs wait <jobid> to wait for an in-progress job to finish.
That being said, if you're purely looking for a callback for choosing whether to run another job, consider using Apache Airflow + Cloud Composer.
In general, the more you tell us about what you're trying to accomplish, we can help you better :)
I have a flexible app engine app in which I am running a set of integration tests upon request. I get a sys.exit(1) after 30 seconds every time I run it. I cannot use Task Queue or Deferred library since this is a Flexible(Not Standard) app engine project. Any ideas on how to extend this 30 second deadline?
I also tried to change from auto scaling to manual scaling without any luck :|
In your Flexible environment you can use pub/sub library to create background tasks. You need to create a worker service which listens to a particular queue and you can add tasks in a queue , when the task is ready it will be thrown to worker service and be taken care of by it. Here is the reference you can use to solve it, https://cloud.google.com/python/getting-started/using-pub-sub. Hope it will help :)
I'm using OSB 12c.I have an OSB proxy, which takes 15 minutes to complete each request on an average.
Lets say that I have five request now in running state.
Is there a way to see these running requests just like we can see the requests of bpel in EM console ?
Is there a way to terminate one of the requests without any impact on rest of the running requests ?
Is it possible to kill all request in case point-2 is not possible ?
Thanks !
I don't think so, not without changing things.
If you were open to changing the service to e.g. decompose the request into separate internal JMS messages, you should be able to use JMX to interrogate the MDBs and discover what they're up to. Then again, if you were to switch to JMS you could probably just look at the queue and get an idea about what it's doing just from the number and content of the messages created.
I'm not aware of the ability to cancel individual requests in OSB, sorry.
I think you can not terminate OSB threads directly.
You can configure your Weblogic to deal with stuck threads. (Thread that were running for a certain period of time)
You can configure a Dispatch Policy in your proxys using Work Manager to handle Stuck Threads and minimize impact on server.
It looks like I can create a push-queue that will start backends to process tasks and that I can limit the number of workers to 1. However, is there a way to do this with a pull-queue?
Can App-Engine auto-start a named backend when a pull-queue has tasks and then let it expire when idle and the queue is empty?
It looks like I just need some way to call an arbitrary URL to "notify" it that there are tasks to process but I'm unable to find any documentation on how this can be done.
Use a cron task or a push queue to periodically start the backend. The backend can loop through the tasks (if any) in the pull queue, and then expire.
There isn't a notification system for pull queues, only inspection through queue statistics and empty/non-empty lease results.
First of all you need to decide scalability type you want to use for your module. I think that you should take a look to Basic Scaling (https://developers.google.com/appengine/docs/java/modules/)
Next, to process tasks from pull queue you can use Cron to check queues every several minutes. It will be important to request not basic scaling module, but frontend module, cause cron will start instances. The problem is that you will still need to pay for at least one instance of frontend cause your cron job will not allow it to shutdown.
So the solution could be the following:
Start cron every 1 or 5 minutes and call frontend
Check queue in frontend and issue URLFetch request to basic scaling module if there are tasks in pull queue
Process tasks in queue using basic scaling module
If you use F1 instances for frontend and b2 or greate for other modules it could save you some money.
I have several clients that create new objects. When new object is created I need to start a timer that will change some object properties when time is elapsed (each object can be visible only for defined client groups certain time).
I want to use for this purpuses web-service and wrote a method that starts timer.
For example I need to set timer to 5 minutes. Are there any restrictions for executing time? Will a timer keep my web-service alive?
Perhaps, I don't understand your task completely, but your idea about Web Service usage looks strange to me. Web Services are usually used to process requests from remote clients. I.e. a client calls method of a Web Service and Web Service returns a result to this client.
I think, I got your idea :). If you need to just change data in the DB, I think the better solution is to create a windows service which will ping web service when needed.