I have a log file that records the start and finish times for processing files. The entries contain strings that look like this:
=============== STARTED PROCESSING FILE filename at Thu Jul 19 00:03:55 2018 EDT===============
=============== FINISHED PROCESSING FILE filename at Thu Jul 19 00:04:05 2018 EDT===============
Initially I came up with a query using _time:
processing.log "FINISHED PROCESSING FILE" OR "STARTED PROCESSING FILE" | rex field=_raw "(?<filename>\S*)" | stats count first(_time) as start last(_time) as finished by filename | eval duration = abs( finished - start)
That seemed to work just fine until I realized that _time of finished and start could be hours apart even though actual processing to 10 seconds (as in the above example. So now I am trying this query:
processing.log "FINISHED PROCESSING FILE" OR "STARTED PROCESSING FILE" | rex field=_raw "(?<filename>\S*) at (?<ptime>.*) EDT" | eval stime=strptime(ptime,'%a %B %d %Y %H:%M:%S')| stats count first(stime) as start last(stime) as finished by filename | eval duration = abs( finished - start)
However it is not providing the desired results of showing the processing duration of each filename. What am I doing wrong/how can I fix this?
Instead of ...| stats count first(stime) as start last(stime) as finished by filename |eval duration = abs( finished - start) try ... | stats count range(stime) as duration by filename
Related
I'm trying to get a service to run at :0/5:00 (every 5min) every time. This process may take more or less than 5min. If it passes the 5min mark, do nothing, and start the next one on the next 5min block.
This all works properly when the process takes less than 5min, but if the process takes more than 5min, it re-starts the service immediately.
From what I've read, the default behavior is to skip the next execution if the service time overflows (see https://unix.stackexchange.com/questions/203254/does-systemd-timer-unit-skip-the-next-run-if-the-process-hasnt-finished-yet). However, I also see a question which seems to indicate that the service will start as soon as the previous execution completes (see https://unix.stackexchange.com/questions/487562/how-to-make-systemd-timer-run-exactly-every-10-minutes-with-a-task-that-lasts-s) which fits with my expreiences.
(For testing purposes, I'm using :0/1:0 so that I don't have to wait 5min every time)
My timer:
[Unit]
Description=Start service on timed schedule.
After=network-online.target
Wants=network-online.target
[Timer]
OnCalendar=*:0/1:0
AccuracySec=1us
[Install]
WantedBy=timers.target
My service:
[Unit]
Description=Do a thing
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
ExecStart=/bin/bash /usr/local/bin/my-script.sh
My script:
#!/bin/bash
PROG="/usr/local/bin/fake-third-party.sh"
exec $PROG &>> /tmp/timer_testing.out
Fake third-party script:
#!/bin/bash
echo "`date` I sleep now"
sleep 76
echo "`date` No more sleep"
So right now, when the 76s ends, it immediately starts the next one.
Thu Jun 25 13:01:00 UTC 2020 I sleep now
Thu Jun 25 13:02:16 UTC 2020 No more sleep
Thu Jun 25 13:02:16 UTC 2020 I sleep now
Thu Jun 25 13:03:32 UTC 2020 No more sleep
Thu Jun 25 13:03:32 UTC 2020 I sleep now
When what I'd be looking for is:
13:01:00 -> sleep
13:02:16 -> wake
13:03:00 -> sleep
13:04:16 -> wake
13:05:00 -> sleep
One possible (aka hacky) solution that is working right now is the following:
Change the my-script.sh to have this at the front:
minutes=$(date +"%-M" | tail -c 2)
seconds=$(date +"%S")
if [[ "$seconds" != "00" || ! ( "$minutes" == "0" || "$minutes" == "5" ) ]]; then
then
exit 0
fi
This does trigger the my-script.sh, but the script never calls exec third-party-script so, it accomplishes the requirement.
However I would like to do this the 'proper' way, if there is one.
Did anyone encounter this behavior also and knows a solution? so_timeout seems the parameter to enlarge but I had no success with this.
In the log files I found caused by pipe closed.
A manual sftp and 'ls *' command took more as 20 minutes to get a listing back. So I guess it is a Camel timeout. Can this set per route?
2020-02-07T15:54:29,624 WARN [com.bank.fuse.filetransfer.config.bankFileTransferManagerLoggingNotifier] (Camel (rabobank-file-transfer-manager-core) thread #4494 - sftp://server.eu/outgoing/attachments) ExchangeFailedEvent | RouteName: SAPSF-ONE-TIME-MIGRATION-18 | OriginatingUri: sftp://server.eu/outgoing/attachments?antInclude=*.pgp&consumer.bridgeErrorHandler=true&delay=20000&inProgressRepository=%23inProgressRepo-SAPSF-ONE-TIME-MIGRATION&knownHostsFile=%2Fhome%2Fjboss%2F.ssh%2Fknown_hosts&move=sent&onCompletionExceptionHandler=%23errorStatusOnCompletionExceptionHandler&password=xxxxxx&privateKeyFile=%2Fhome%2Fjboss%2F.ssh%2Fid_rsa&readLock=none&soTimeout=1800000&streamDownload=true&throwExceptionOnConnectFailed=true&username=account | completedWithoutException: false | toEndpoint: | owner: [SAP] | sm9CI: [CI5328990] | priority: [Low] | BreadcrumbId: ID-system-linux-bank-com-42289-1580217016920-0-5929700 | exception: Cannot change directory to: ..
Maybe soTimeout=1800000 was too short. A manual sftp and ls * command took about 20 minutes.
Since this was a one time action. I resolved it with a manual sftp.
Is there any way to check for how long the NAGIOS TOOL runs? I mean when the tool started running and the time up till now.
Nagios is running in a remote machine, in which I have access (through ssh). I have both credentials for accessing the machine and credentials just to see the Stats from Nagios on this machine
I tried System->Process Info, but I do not have privileges to view such information.
Is there any other way, through terminal?
You can use nagiostats to check the uptime of a Nagios instance. See: https://assets.nagios.com/downloads/nagioscore/docs/nagioscore/3/en/nagiostats.html
[nagios#lanman ~]# /usr/local/nagios/bin/nagiostats -c /usr/local/nagios/etc/nagios.cfg
Nagios Stats 3.0prealpha-05202006
Copyright (c) 2003-2007 Ethan Galstad (www.nagios.org)
Last Modified: 05-20-2006
License: GPL
CURRENT STATUS DATA
------------------------------------------------------
Status File: /usr/local/nagios/var/status.dat
Status File Age: 0d 0h 0m 9s
Status File Version: 3.0prealpha-05202006
Program Running Time: 0d 5h 20m 39s <------------
Nagios PID: 10119
Used/High/Total Command Buffers: 0 / 0 / 64
Used/High/Total Check Result Buffers: 0 / 7 / 512
...
Find the nagios.log file, it's likely in the var directory under the Nagios installation. Then..
grep "Nagios.*starting" nagios.log | tail -1
Grab the epoch time (first field), and convert it to local.
date -d #1580045430
Sun Jan 26 07:30:30 CST 2020
All in one, assuming the nagios.log is in the current directory.
date -d #$(grep "Nagios.*starting" nagios.log | tail -1 | awk '{print $1}' | sed 's/\[//;' | sed 's/\]//;')
Sun Jan 26 07:30:30 CST 2020
When I run Gatling from my command prompt I get a list of simulations like this:
Choose a simulation number: 1,2,3,4
When I type 3 third simulation will run but this sequence is auto-generated.Suppose I want to list them according to my wish like:
3,2,1,4
Is it possible to give user defined sequence for simulations list.If yes how it is possible?
As far as I know there is no possibility in Gatling to provide sequence of simulations. You can achieve this by writing for example bash script. For running Gatling tests in mvn it could look like this
#!/bin/bash
#params
SIMULATION_CLASSES=
#usage
function usage (){
echo "usage: $0 options"
echo "This script run Gatling load tests"
echo ""
echo "OPTIONS:"
echo "Run options:"
echo " -s [*] Simulation classes (comma separated)"
}
#INIT PARAMS
while getopts “s:” OPTION
do
case $OPTION in
s) SIMULATION_CLASSES=$OPTARG;;
?) usage
exit 1;;
esac
done
#checks
if [[ -z $SIMULATION_CLASSES ]]; then
usage
exit 1
fi
#run scenarios
SIMULATION_CLASSES_ARRAY=($(echo $SIMULATION_CLASSES | tr "," "\n"))
for SIMULATION_CLASS in "${SIMULATION_CLASSES_ARRAY[#]}"
do
echo "Run scenario for $SIMULATION_CLASS"
mvn gatling:execute -Dgatling.simulationClass=$SIMULATION_CLASS
done
And sample usage
./campaign.sh -s package.ScenarioClass1,package.ScenarioClass2
If you use the Gatling SBT Plugin (demo project here), you can do, in Bash:
sbt "gatling:testOnly sims.ReadProd02Simulation" "gatling:testOnly sims.ReadProd02Simulation
This first runs only the sceenario ReadProd02Simulation, and then runs ReadProd03Simulation. No Bash script needed.
The output will be first the output from ReadProd02Simulation and then ReadProd03Simulation, like so:
08:01:57 46 ~/dev/ed/gatling-sbt-plugin-demo[master*]$ sbt "gatling:testOnly sims.ReadProd02Simulation" "gatling:testOnly sims.ReadProd02Simulation"
[info] Loading project definition from /home/.../gatling-sbt-plugin-demo/project
[info] Set current project to gatling-sbt-plugin-demo...
Simulation sims.ReadProd02Simulation started...
...
Simulation sims.ReadProd02Simulation completed in 16 seconds
Parsing log file(s)...
Parsing log file(s) done
Generating reports...
======================================================================
- Global Information ----------------------------------------------
> request count 3 (OK=3 KO=0 )
...
...
Reports generated in 0s.
Please open the following file: /home/.../gatling-sbt-plugin-demo/target/gatling/readprod02simulation-1491631335723/index.html
[info] Simulation ReadProd02Simulation successful.
[info] Simulation(s) execution ended.
[success] Total time: 19 s, completed Apr 8, 2017 8:02:33 AM
08:02:36.911 [INFO ] i.g.h.a.HttpEngine - Start warm up
08:02:37.240 [INFO ] i.g.h.a.HttpEngine - Warm up done
Simulation sims.ReadProd03Simulation started...
...
Simulation sims.ReadProd03Simulation completed in 4 seconds
Parsing log file(s)...
Parsing log file(s) done
Generating reports...
======================================================================
---- Global Information ----------------------------------------------
> request count 3 (OK=3 KO=0 )
......
Reports generated in 0s.
Please open the following file: /home/.../gatling-sbt-plugin-demo/target/gatling/readprod03simulation-1491631356198/index.html
[info] Simulation ReadProd03Simulation successful.
[info] Simulation(s) execution ended.
[success] Total time: 9 s, completed Apr 8, 2017 8:02:42 AM
That is, first it runs one sim, then another, and concatenates all output.
But how do you make use of this? Well, you could use Bash and grep the output for exactly two lines matching failed 0 ( 0%) (if you run two simulations) + check the total request counts for both simulations, also via Bash + grep etc.
I'm migrating several Perforce projects to Git. One is failing though at 18% of the process with:
fatal: Unsupported command: Users.Ref "KPLUS"
It looks like git fast-import is trying to execute the text in the file where it should be printed (I think)
The fast-import crash report shows me
fast-import crash report:
fast-import process: 28327
parent process : 28325
at Fri Sep 11 14:34:26 2015
fatal: Unsupported command: Users.Ref "KPLUS"
Most Recent Commands Before Crash
---------------------------------
....
....
commit refs/remotes/p4/master
committer USERNAME <EMAIL> 1175609377 +0100
data <<EOT
* Users.Ref "KPLUS"
Active Branch LRU
-----------------
active_branches = 1 cur, 5 max
pos clock name
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1) 714 refs/remotes/p4/master
Inactive Branches
-----------------
refs/remotes/p4/master:
status : active loaded
tip commit : 307170cc21264c58ab1943c16f1d2943c1a44f45
old tree : 2f45d5c6d9cbe56e5f335f92b21316ad272f3504
cur tree : 2f45d5c6d9cbe56e5f335f92b21316ad272f3504
commit clock: 714
last pack : 0
Marks
-----
-------------------
END OF CRASH REPORT
The text is in a xml file that doesn't seem to be well formatted, but I would assume this shouldn't matter.
Found the cause in the commit messages. There were "EOT" lines in the message causing git-p4 script to interpret this as an End of Transaction. All next lines were interpreted as executable lines. Changing the git-p4 script from using EOT to EOM solved the issue.