How can I make scheduling in django using djcelery with redis.How can i add starttime and endtime using this?What is the difference between crontab and interval? Can I make scheduling using djcelery?
Related
For anyone with experience with fastai’s distributed training (either within SageMaker or outside it):
Are there any material benefits to using it over PyTorch DDP (which it’s built on top of)?
What would be the easiest way to incorporate this inside a SM training job?
It requires the training script to be run like python -m fastai.launch scriptname.py ...args... so using Script Mode is not immediately straightforward. Pointing to a .sh file for the entry_point in the PyTorch estimator means that SageMaker will not pip install from a provided requirements.txt so the user must control all of this inside their bash script.
Can SM Distributed Data Parallel be used with the fastAI distributed training? Or do we need to utilize Pytorch DDP instead of fastai in order to use SM DDP?
Flink Cluster Details,
Number of nodes : 4
Flink Version : 1.11
Flink Client : RestCluserClient
We are submitting Flink batch job from streaming job using PackagedProgram, but our requirement is to execute only one job at a time, let's say we got 2 events from source so idealy 2 batch job must be triggered(each per event) but only one at a time. To achieve this, we were using client.setDetached(false) (in previous version of flink), but once we have migrated it to 1.11 setDetached(false) API has been removed.
Do we have any idea how to implement this requirement ?
After analysis more on this, I found the solution.
Flink 1.11 API provided Utils class for submitting the job viz,ClientUtils and it has two methods,
ClientUtils.submitJob() -> this method works with detached mode as true
ClientUtils.submitJobAndWaitForExecutionResult() -> this works as detached mode as false.
I've set up a cron.yaml file for my app to run some cronjobs, i.e. every 2 mins, every 15 mins... Is it possible to specify the tasks to start from a specific date? Say I want to start running all cronjobs from 6th december 2017?
The GAE cron Schedule format doesn't have direct support for the functionality you seek.
However it's relatively easy to obtain such functionality, see How to make a cron job run during business hours on Google App Engine cron.yaml?
I am using Flink 1.2.1 running on Docker, with Task Managers distributed across different VMs as part of a Docker Swarm.
Uploading an Apache Beam application using the Flink Web UI and trying to set the parallelism at job submission point doesn't work. Neither does submit a job using the Flink CLI.
It seems like the parallelism doesn't get picked up at client level, it ends up defaulting to 1.
When I set the parallelism programmatically within the Apache Beam code, it works: flinkPipelineOptions.setParallelism(4);
I suspect the root of the problem may be in the org.apache.beam.runners.flink.DefaultParallelismFactory class, as it checks for Flink's GlobalConfiguration, which may not pick up runtime values passed to Flink.
Any ideas on how this could be fixed or worked around? I need to be able to change the parallelism dynamically, so the programmatic approach won't work, nor will setting the Flink configuration at system level.
I am using the following documentation:
https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/parallel.html
https://beam.apache.org/documentation/sdks/javadoc/2.0.0/org/apache/beam/runners/flink/DefaultParallelismFactory.html
This should probably be fixed in the Beam Flink Runner but as a workaround you can try setting the parallelism to -1 programatically. This should make the translation pick up the parallelism that is specified when submitting the job.
I'm making a simple auction application as part of a project.
As is the case with auctions, their timer needs to be counting down (or other kind of time related update) even no one is actively using said application, so I wouldn't be updating them using scripts.
Is there any way to do that from inside the database or otherwise, without the help of an outsider like cron or windows task scheduler?
PostgreSQL has not integrated scheduler now. There are two usual solutions:
cron
pgagent - it is auxiallary application, that can work as scheduler - and it is controlled by SQL commands - pgAdminIII has support for this application http://www.pgadmin.org/docs/dev/pgagent-jobs.html