I have a gear with JBoss Fuse and Apache Camel consuming a considerable amount of memory that needs to run a nightly task, i.e. it needs to be activated only at a specific time of the day, run its routes, and then shutdown.
Is there a way to accomplish this on OpenShift?
rhc cartridge add cron
In the .openshift/cron/ directory you can place your cron job, make it executable, add with git, commit and push it
Related
My Flink job has a jar file provided by the client, which I can store in /lib folder. Is there a way to reload the updated jar file, without restarting the cluster?
No, that is not possible with the current version (Flink 1.4.0, Dec 2017).
Flink offers savepoints to save the state of an application.
If you want to change the code (or dependencies) of an application, zou have to take a savepoint, update the code/dependencies, and restart the application from the savepoint.
This technique can also be used to scale an application up or down or to migrate it.
I am newbie in Apache Flink and our team is trying to set up an Apache Flink Cluster on Apaches Mesos. We have already installed Apache Mesos & Marathon with 3 Master nodes and 3 Slaves and now we are trying to install Apache Flink without DC/OS as mentioned here https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/mesos.html#mesos-without-dcos.
I have couple of questions over here :
Do we need to download Flink on all the nodes(master and slaves) and configure mesos.master in all nodes?
Or Shall we download flink on only one master node and configure mesos.master over there?
If flink needs to be downloaded on all the nodes then what should be the location of flink directory or if there is any script where I can specify that?
Is running "mesos-appmaster.sh" on master node also responsible for running flink libraries and classes on slaves?
Thanks
Do we need to download Flink on all the nodes(master and slaves) and configure mesos.master in all nodes?
No you don't. Actualy it depends on the way you want to run Flink. In your setup the most convenient way to run Flink would be to run it with Marathon and download binaries during deployment. See this
Or Shall we download flink on only one master node and configure mesos.master over there?
It's up to you. You can run Flink on dedicated server or let Marathon do it for you. If you already have Marathon then it's easier to run Flink with Marathon. On the other hand for debugging purposes and proof of concept I'll recommend standalone version where you can quickly change configuration on local machine and see how it works. Creating docker images or binaries and publishing them in repository and finally deploying Flink on Marathon could have more overhead that will slow you down on development but will keep you safe on production. Flink does not come with support for High Availability (HA) so Marathon is required to provide basic HA support (launch new instance of Flink when agent crash).
If flink needs to be downloaded on all the nodes then what should be the location of flink directory or if there is any script where I can specify that?
Flink does not have to be downloaded on all nodes. It can be downloaded when needed at deployment.
Is running "mesos-appmaster.sh" on master node also responsible for running flink libraries and classes on slaves?
Flink is a scheduler which means that it should start tasks and executors on Mesos when needed.
Even when not using DC/OS, feel free to look at the Apache Flink DC/OS package. At its core, it is a marathon app definition you can deploy on pure Marathon/Mesos. The Flink package (as of today) does not require any DC/OS specific features.
The DC/OS example might also provide useful information.
For example I uploaded JAR with my flow and run it through Apache Flink dashboard. Then I implemented some changes in flow and want to deploy them.
Can anybody explain me step-by-step how to deploy new version of my flow to Apache Flink cluster correctly (without downtime, loosing state, etc.)? I didn't find description of deploy process in official documentation.
What you want to use is the savepoints in Flink.
The steps are as follows:
Prepare the new jar for your job
Save the state of the currently running job using flink savepoint <JobID>
Stop the job
Start the new jar using the just created savepoint flink run -s <pathToSavepoint> <jobJar> ...
See also: https://www.ververica.com/blog/how-apache-flink-enables-new-streaming-applications-part-1
I have a simple question that I can't find information on regarding Apache Camel-Quartz. For Camel-Quartz to work do you have to deploy inside a web container like Tomcat? And hence because the application will always be alive it will know when to run?
I'm asking because if you deploy your Camel application in a stand alone JVM I don't see how the application will be smart enough to understand when to run.
thanks
Quartz is embedded with your Camel application and thus when you start Camel, quartz is also started. And then it knows when to run, as long you keep the Camel application running.
There is no magic in there. Its just java code that runs, and Quartz is also just java code. And it does not require a special server etc. Quartz is just a library (some JAR files) that you run together with your own application.
Quartz just have logic for scheduling jobs (eg its like a big clock-work) that knows what the time is, and when to trigger jobs accordingly to how you tell it to.
Recently I've started using limited staging on my Google App Engine project. The data is still shared between all versions, but behaviour (especially user facing behaviour) is different.
Naturally when I implement something incredibly new it only runs on the latest version of my code and I don't feel like it should be backported to the older versions.
Some of this new functionality requires cron jobs to be run periodically, but I'm hitting a problem. I have to run a cron job to call the latest code, but this is what Google's documentation has to say about the issue:
Cron requests are always sent to the default version of the application.
The default version is the oldest because the first versions of the client code that went out to users weren't future proof and don't know how to select which API version to call.
So my question is, how can I get around this limitation and make a cron job that will call the latest rather than the default version of the application?
You can now specify a version using the target tag.
<target>version-2</target>
You can't change the cron jobs to run on a different version then the default.
Depending on how much time your cron job takes to run you could change your cron job script to to do a URLFetch to "http://latest.appname.appspot.com/cron_job_endpoint".
If you're cron job takes longer then 10 minutes to run, then I would design it in a way that you can chain the different tasks using task queues.