I'm using Azkaban 3.4.1 and one of my flow has more than 30 dependencies. Some dependencies are takes more longer than another. So, I want to prioritize these flows to started before another flows. (because the running thread is limited)
Currently the number of parallel execution is limited with flow.num.job.threads which is 10 by default. I tried increase that property and make sure the long process started right away, but the cpu get very high, so I am not sure that is a good option.
Using this fork https://github.com/hanip-ss/azkaban/releases/tag/3.4.2.
I can now add job.priority value in job properties file.
Related
I just want to my flink application as much as configurable. And also i want to change the behavior of the process function in the running time instead of stopping the cluster and re-deploy the jar file.
Is there any document for that? Or is it possible to inject process function code into running jar. For instance, from the web ui, i will get the process function input(as a java code) then after submitting the form, I will update the process function behavior.
You can use a BroadcastProcessFunction (or a KeyedBroadcastProcessFunction), and on the broadcast channel, communicate (in some fashion) what the process function is supposed to do.
I've seen this technique used to broadcast javascript code (to be executed by Rhino), commands in a DSL, references to a JAR file to load, etc.
It's old and not well documented, but https://github.com/alpinegizmo/flink-training-exercises/blob/master/src/main/java/com/ververica/flinktraining/solutions/datastream_java/broadcast/TaxiQuerySolution.java is an example of this approach that uses Janino to compile and execute dynamically supplied Java expressions.
We have large data sets in a 4D database. Recently we needed to automate certain tasks to export, import, or process data, which requires some connection between 4D and the unix shell (bash and zsh).
We did fine whenever a 4D function had to call a shell command or shell script, using the LAUNCH EXTERNAL PROCESS instruction. But I am having trouble doing the opposite, starting a 4D function from a shell script. The on-line documentation did not give me any suggestion on how to do this.
How would I run a 4D function at a specific date+time? 4D itself does not seem to provide a way to schedule an action. OS X provides a perfectly good way to start a command at any future time (weekly or monthly). I just do not know 4D well enough to figure out how to call a 4D function from a shell script. If it was MySQL or PostgreSQL then I had no problem, because I could call the mysql or pgsql client, which can work without a GUI. Can I do something similar with 4D?
There are a couple of ways you could accomplish this in 4D. The way I would typically do this, would be to use the web server in 4D. Then I would trigger that service with a GET or POST using whatever scheduler (cron, windows scheduler, etc) that I like. This does require some web server licensing from 4D.
This gives you flexibility to trigger the event at given times or from an external process. Say your data has been cleaned and is now ready to be imported, you don't have to wait until the next import time, it could just be triggered.
Alternatively you could create a 4D method that runs in its own process. You launch the process at database started and then it would loop indefinitely, checking to see if it needed to trigger whatever action you want.
Take a look at the New process, DELAY PROCESS, and While...End While commands. You could launch the new process to run on the server or a client depending your needs.
Basically your On Startup method calls New Process to run your 'scheduler' method. In that method you have a While (some true condition) loop that checks to see if it should run. Then it sleeps (delay process) for a bit so it doesn't hog resources, wakes up again and runs the loop.
If this is something you want to be able to stop programmatically you can use SET PROCESS VARIABLE to set the value of a variable in the scheduler process from outside. Then have the loop check that value to see if it should exit.
To answer your question, if you want call 4D on Mac from shell you should use a plugin from Pluggers (cfr https://www.pluggers.nl/product/scripting-tools/ ) and osascript.
I am working in a large project. There are about minimum 160 specs to be executed with protractor, and it consumes more than one hour to finish automation testing. But issue is my system get sleeps in between it when no action is performed. Is there any way to make system alive until protractor finishes its execution.
So that I could run it without system sleep. I cant increase my sleep time because it had to be run in many sytem. Please let me know how could I handle it ?
I am using chrome for running the automation.
Cheers.
Would it be possible to let me know what OS you are using? So that I can give step by step information. For example: if you are using windows machine then
Goto --> Control Panel\Hardware and Sound\Power Options --> Change the Sleep settings make it to "Never"
It will remain open for your lifetime till you switchoff your computer.
By the way there is no relation this question to Protractor.
We have a COBOL batch program that we are able to execute manually from JCL. We want to automate this process so that it can execute every 15 minutes.
Is there a way to automate the execution of a batch program on the mainframe?
I'm a PC guy and I know in windows I can create a .BAT file and set it up in Task Scheduler to run every 15 minutes. I'm essentially trying to do the same thing on the mainframe.
Is there a way to automate the execution of a batch program on the
mainframe?
Yes.
Many mainframe shops have job schedulers. Control-M from BMC is one, ASG has Zeke, there are others.
Having said that, it sounds like the application in question is written to periodically poll for some event. Mainframes typically have better ways of accomplishing tasks people normally solve via polling. Event monitoring, for example.
Mainframe Scheduling software like Control-M from BMC is one, ASG has Zeke, CA7 from CA and IBM TWS for ZOS formerly OPCA can be used to schedule a job every 15 minutes.
You could add a job for every 15 minute period or have the first step of the job be to add the 1 that will run in the next 15 minutes.
Pros
Operators will be notified of the job failing
Cons
Will end up allot of the same jobs in the schedule
TWS for ZOS (what I am know) you would need to add nearly 96 jobs and set the corresponding times for it
The option I would recommend is using an automation product such System Automation from IBM, Control-O from BMC or OPS from CA.
With any of the above automation products you could setup a started task and get them to start it every 15 minutes. It is much easier say for example using 1 panel in System Automation to set it up to run a start task every 15 minutes
If you wanted to know if it fails you could use the automation products to schedule it in any of the above schedulers.
There are so many solutions to this, it really depends on what you are monitoring. Besides the standard "use a job scheduler like CA7" (with the disadvantage of having so many jobs that run during the day, just kind-of messy).
You could either define an address space (started task) that invokes your COBOL code, and within your COBOL code have it sleep (i.e. wait on a timer) for 15 minutes, wake-up check whatever and go back to sleep. Alternatively, run the job on JES2 but you might have to a little extra so that JES keeps the job active all day!
If this code finds a problem then it can also issue a console message (maybe, you might have to write a little bit of assembler code to do issue a WTO or WTOR), so the the operator either knows (WTO) or knows and has to reply (WTOR) (write to operator with reply).
I'm investigating strange WPF multi threaded application hang that happen randomly on production machines more easily than development ones.
I have added Application.Dispatcher.UnhandledException and CurrentDomain.UnhandledException event handlers to log the every and each exception in order to keep the application from crashing as it is a system shell.
I suspect the reason of the hang is that I'm handling those events and not allowing the application to crash, what make me say this is that when i monitored the application activity when it hanged using windows task manager i noticed CPU load probably caused by background tasks in the application and those background tasks are writing to logs as expected.
The logger class writes the logs to the disk asynchronously using BeginInvoke.
The big problem is every time the application hangs in production machine i find nothing in log files.
Now i have two questions:
What is the fire order of those events: "Application.Dispatcher.UnhandledException and CurrentDomain.UnhandledException".
2.Is there anyway i can manage to be able to know were my code is failing
Note: I use a lot of Thread.Sleep in background tasks as i manage the threads my self and i think i went wrong with this approach and I'm considering rewriting all of my background tasks from scratch.
Any help will be much appreciated.
Order of this event is:
Application.DispatcherUnhandledException
CurrentDomain.UnhandledException
If you want get information from exception where your code is failing, check property StackTrace. You can also attach to your project pdb files and in this case you will get StackTrace with line numbers.