I've just started using NSClient++ to monitor a selection of Windows hosts, and have been asked to set up Nagiosgraph for monitoring.
I have a map to graph my memory checks, which should graph an average for 5 mins and one for 30.
# Service type: ntload
# check command: check_nt -H Address -v CPULOAD -l5,70,90,30,70,90
# output: CPU Load 9% (5 min average) 11% (30 min average)
# perfdata: 5 min avg Load=9%;70;80;0;100 30 min avg Load=11%;70;90;0;100
#/perfdata:.*5 min avg Load=(d+)%;(d+);(d+);d+;d+ 30 min avg Load=(d+)%;d+;d+;d+;d+ /
/output:.*?(d+)% .*?(d+)% /
and push #s, [ ntload,
[ avg05min, GAUGE, $1 ],
[ avg30min, GAUGE, $2 ] ];
The 5min check is graphing, however the 30 min check is flat-lining at the bottom.
Your (d+)s should be (\d+). Not sure if this a copy/paste error or the source of the problem though, because as it is now, both your graphs shouldn't be graphing anything.
Related
Hey so I am trying my hand at image classification/transfer learning using the monkey species dataset and the resnet50 with a modified final fc layer to predict just the 10 classes. Eveything is working until I use model.train() and model.eval() then after the first epoch it starts to return nans and the accuracy drops off as you'll see below. I'm curious why is this only when switching to train/eval....?
First I import the model and attach the classifier and freeze the parameters
%%capture
resnet = models.resnet50(pretrained=True)
for param in resnet.parameters():
param.required_grad = False
in_features = resnet.fc.in_features
# Build custom classifier
classifier = nn.Sequential(OrderedDict([('fc1', nn.Linear(in_features, 512)),
('relu', nn.ReLU()),
('drop', nn.Dropout(0.05)),
('fc2', nn.Linear(512, 10)),
]))
# ('output', nn.LogSoftmax(dim=1))
resnet.classifier = classifier
resnet.to(device)
Then setting my loss func, optimizer, and shceduler
# Step : Define criterion and optimizer
criterion = nn.CrossEntropyLoss()
# pass the optimizer to the appended classifier layer
optimizer = torch.optim.SGD(resnet.parameters(), lr=0.01)
# Scheduler
scheduler = torch.optim.lr_scheduler.MultiStepLR(optimizer, milestones=[10], gamma=0.05)
Then setting the training and validation loops
epochs = 20
tr_losses = []
avg_epoch_tr_loss = []
tr_accuracy = []
val_losses = []
avg_epoch_val_loss = []
val_accuracy = []
val_loss_min = np.Inf
resnet.train()
for epoch in range(epochs):
for i, batch in enumerate(train_loader):
# Pull the data and labels from the batch
data, label = batch
# If available push data and label to GPU
if train_on_gpu:
data, label = data.to(device), label.to(device)
# Compute the logit
logit = resnet(data)
# Compte loss
loss = criterion(logit, label)
# Clearing the gradient
resnet.zero_grad()
# Backpropagate the gradients (accumulte the partial derivatives of loss)
loss.backward()
# Apply the updates to the optimizer step in the opposite direction to the gradient
optimizer.step()
# Store the losses of each batch
# loss.item() seperates the loss from comp graph
tr_losses.append(loss.item())
# Detach and store the average accuracy of each batch
tr_accuracy.append(label.eq(logit.argmax(dim=1)).float().mean())
# Print the rolling batch training loss every 20 batches
if i % 40 == 0 and not i == 1:
print(f'Batch No: {i} \tAverage Training Batch Loss: {torch.tensor(tr_losses).mean():.2f}')
# Print the average loss for each epoch
print(f'\nEpoch No: {epoch + 1},Training Loss: {torch.tensor(tr_losses).mean():.2f}')
# Print the average accuracy for each epoch
print(f'Epoch No: {epoch + 1}, Training Accuracy: {torch.tensor(tr_accuracy).mean():.2f}\n')
# Store the avg epoch loss for plotting
avg_epoch_tr_loss.append(torch.tensor(tr_losses).mean())
resnet.eval()
for i, batch in enumerate(val_loader):
# Pull the data and labels from the batch
data, label = batch
# If available push data and label to GPU
if train_on_gpu:
data, label = data.to(device), label.to(device)
# Compute the logits without computing the gradients
with torch.no_grad():
logit = resnet(data)
# Compte loss
loss = criterion(logit, label)
# Store test loss
val_losses.append(loss.item())
# Store the accuracy for each batch
val_accuracy.append(label.eq(logit.argmax(dim=1)).float().mean())
if i % 20 == 0 and not i == 1:
print(f'Batch No: {i+1} \tAverage Val Batch Loss: {torch.tensor(val_losses).mean():.2f}')
# Print the average loss for each epoch
print(f'\nEpoch No: {epoch + 1}, Epoch Val Loss: {torch.tensor(val_losses).mean():.2f}')
# Print the average accuracy for each epoch
print(f'Epoch No: {epoch + 1}, Epoch Val Accuracy: {torch.tensor(val_accuracy).mean():.2f}\n')
# Store the avg epoch loss for plotting
avg_epoch_val_loss.append(torch.tensor(val_losses).mean())
# Checpoininting the model using val loss threshold
if torch.tensor(val_losses).float().mean() <= val_loss_min:
print("Epoch Val Loss Decreased... Saving model")
# save current model
torch.save(resnet.state_dict(), '/content/drive/MyDrive/1. Full Projects/Intel Image Classification/model_state.pt')
val_loss_min = torch.tensor(val_losses).mean()
# Step the scheduler for the next epoch
scheduler.step()
# Print the updated learning rate
print('Learning Rate Set To: {:.5f}'.format(optimizer.state_dict()['param_groups'][0]['lr']),'\n')
The model starts to train but then slowly becomes nan values
Batch No: 0 Average Training Batch Loss: 9.51
Batch No: 40 Average Training Batch Loss: 1.71
Batch No: 80 Average Training Batch Loss: 1.15
Batch No: 120 Average Training Batch Loss: 0.94
Epoch No: 1,Training Loss: 0.83
Epoch No: 1, Training Accuracy: 0.78
Batch No: 1 Average Val Batch Loss: 0.39
Batch No: 21 Average Val Batch Loss: 0.56
Batch No: 41 Average Val Batch Loss: 0.54
Batch No: 61 Average Val Batch Loss: 0.54
Epoch No: 1, Epoch Val Loss: 0.55
Epoch No: 1, Epoch Val Accuracy: 0.81
Epoch Val Loss Decreased... Saving model
Learning Rate Set To: 0.01000
Batch No: 0 Average Training Batch Loss: 0.83
Batch No: 40 Average Training Batch Loss: nan
Batch No: 80 Average Training Batch Loss: nan
I see that resnet.zero_grad() is after logit = resnet(data), which causes the gradient to explode in your case.
Please do it as below:
# Clearing the gradient
optimizer.zero_grad()
logit = resnet(data)
# Compute loss
loss = criterion(logit, label)
I know we can find the max connections by {DBInstanceClassMemory/12582880}(12582880 is the magic number for my AWS resources). However, for db.m4.large (8 GB RAM) I checked online at many places that the maxConnections are 648. However, when I made the calculations, I found
8 * 1000000000/12582880 = 635.7844944877
8 * 1024 * 1024 * 1024 / 12582880 = 682.6684027822
Similarly for db.t2.small
2 * 1000000000/12582880 = 158.9461236219
2 * 1024 * 1024 * 1024 / 12582880 = 170.6671006955
acc to the internet: 150
Please help with finding the correct number. I cannot open MySQL console on the AWS instance due to some restrictions.
This AWS page has a table of containers and max_connections limits (default):
https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraMySQL.Managing.Performance.html
excerpt:
db.t2.small 45
db.t2.medium 90
db.t3.small 45
db.t3.medium 90
db.r3.large 1000
Text says:
the default connection limit is derived using a formula based on the DBInstanceClassMemory value.
How do I decrease the losses by increasing power level?
Here is the code I am using:
https://github.com/maibewakoofhu/Unet
I am changing the power level using:
phy[1].powerLevel = -20.dB;
At noise level 68dB, power level = -20dB all DatagramReq are sent successfully.
At noise level 70dB, power level = -20dB the DatagramReq fails.
Now, increasing the power level to as high as 125dB, still the DatagramReq fails.
I created a simpler version of your simulation to test the SNR and packet-loss relationship:
import org.arl.fjage.RealTimePlatform
import org.arl.unet.sim.channels.BasicAcousticChannel
platform = RealTimePlatform
channel = [
model: BasicAcousticChannel,
carrierFrequency: 25.kHz,
bandwidth: 4096.Hz,
spreading: 2,
temperature: 25.C,
salinity: 35.ppt,
noiseLevel: 73.dB,
waterDepth: 1120.m
]
simulate {
node 'C', address: 31, location: [180.m, 0, -1000.m], web: 8101
node 'A', address: 21, location: [0.m, 0.m, 0.m], web: 8102
}
The web: entries allows us to interact with each of the nodes to explore what is happening. I connect to each of the nodes (http://localhost:8101/shell.html and http://localhost:8102/shell.html) and subscribe phy to see all physical layer events.
Now, from node A, I try broadcasting frames to see (at various power levels) if node C receives them:
> plvl -20
OK
> phy << new TxFrameReq()
AGREE
On node C, you'll see receptions, if successful:
phy >> RxFrameStartNtf:INFORM[type:CONTROL rxTime:3380134843]
phy >> RxFrameNtf:INFORM[type:CONTROL from:21 rxTime:3380134843]
or bad frames if not:
phy >> RxFrameStartNtf:INFORM[type:CONTROL rxTime:3389688843]
phy >> BadFrameNtf:INFORM[type:CONTROL rxTime:3389688843]
Observations:
- At plvl -20 dB, almost all frames fail.
- At plvl -10 dB, almost all frames are successful.
- At plvl -16 dB, I get a frame loss of about 19%.
The transition between all frames failing to all succeeding is expected to be quite sharp, as is typical in reality for stationary noise, as the FEC performance tends to be quite non-linear. So you'll expect big differences in frame loss rate around the transition region (in this example, at around -16 dB).
Do also note the plvl 125 dB isn't valid (range of plvl is given by phy.minPowerLevel to phy.maxPowerLevel, -96 dB to 0 dB by default). So setting that would have not worked:
> plvl 125
phy[1]: WARNING: Parameter powerLevel set to 0.0
phy[2]: WARNING: Parameter powerLevel set to 0.0
phy[3]: WARNING: Parameter powerLevel set to 0.0
phy: WARNING: Parameter signalPowerLevel set to 0.0
Newbie to bash scripting here and could use some help on this if you have time. My customers upload and each has a datestamp in filename like this:
* 20170815041135
* 20170820041135
* 20170823071727
* 20170826040609
* 20170828050704
* 20170830153011
I need to calculate the number of days between each upload then find the average interval of the listed uploads
I can find the date difference between two dates with this command
echo $(( ($(date --date="20170831" +'%s' ) - $(date --date="20170821" +'%s')) / (60*60*24) ))
gives 10
To do multiple dates I've read that I need an array, so here is my range of upload dates in an array.
array=( `20170830153011`,`20170828050704`,`20170826040609`,`20170823071727`,`20170820041135`,`20170815041135` )
I've read I need to loop through the calculation like this
for i in "${array[#]}" do
?
How do I add my array dates into the calculation?
Your datetimes into an array:
timestamps=(
20170815041135
20170820041135
20170823071727
20170826040609
20170828050704
20170830153011
)
Let's now convert those into epoch times:
epochs=()
for timestamp in "${timestamps[#]}"; do
iso8601=$(sed -r 's/(....)(..)(..)(..)(..)(..)/\1-\2-\3T\4:\5:\6/' <<<"$timestamp")
epochs+=( "$(date -d "$iso8601" "+%s")" )
done
printf "%s\n" "${epochs[#]}"
1502784695
1503216695
1503487047
1503734769
1503911224
1504121411
Now we can iterate over them to calculate the differences. Note that bash array indices start at zero:
n=0
sum=0
for ((i=1; i < "${#epochs[#]}"; i++ )); do
((n++, diff=(${epochs[i]} - ${epochs[i-1]}), sum+=diff))
echo "diff $n = $diff seconds = $((diff/86400)) days"
done
echo "average = $((sum/n)) seconds = $((sum/n/86400)) days"
diff 1 = 432000 seconds = 5 days
diff 2 = 270352 seconds = 3 days
diff 3 = 247722 seconds = 2 days
diff 4 = 176455 seconds = 2 days
diff 5 = 210187 seconds = 2 days
average = 267343 seconds = 3 days
Convert the date in seconds from 1970.
Calculate the difference.
I hope that the bash date function knows to take into account the daylight saving time from that date.
I am trying to design a function that will calculate 30 day rolling volatility.
I have a file with 3 columns: date, and daily returns for 2 stocks.
How can I do this? I have a problem in summing the first 30 entries to get my vol.
Edit:
So it will read an excel file, with 3 columns: a date, and daily returns.
daily.ret = read.csv("abc.csv")
e.g. date stock1 stock2
01/01/2000 0.01 0.02
etc etc, with years of data. I want to calculate rolling 30 day annualised vol.
This is my function:
calc_30day_vol = function()
{
stock1 = abc$stock1^2
stock2 = abc$stock1^2
j = 30
approx_days_in_year = length(abc$stock1)/10
vol_1 = 1: length(a1)
vol_2 = 1: length(a2)
for (i in 1 : length(a1))
{
vol_1[j] = sqrt( (approx_days_in_year / 30 ) * rowSums(a1[i:j])
vol_2[j] = sqrt( (approx_days_in_year / 30 ) * rowSums(a2[i:j])
j = j + 1
}
}
So stock1, and stock 2 are the squared daily returns from the excel file, needed to calculate vol. Entries 1-30 for vol_1 and vol_2 are empty since we are calculating 30 day vol. I am trying to use the rowSums function to sum the squared daily returns for the first 30 entries, and then move down the index for each iteration.
So from day 1-30, day 2-31, day 3-32, etc, hence why I have defined "j".
I'm new at R, so apologies if this sounds rather silly.
This should get you started.
First I have to create some data that look like you describe
library(quantmod)
getSymbols(c("SPY", "DIA"), src='yahoo')
m <- merge(ROC(Ad(SPY)), ROC(Ad(DIA)), all=FALSE)[-1, ]
dat <- data.frame(date=format(index(m), "%m/%d/%Y"), coredata(m))
tmpfile <- tempfile()
write.csv(dat, file=tmpfile, row.names=FALSE)
Now I have a csv with data in your very specific format.
Use read.zoo to read csv and then convert to an xts object (there are lots of ways to read data into R. See R Data Import/Export)
r <- as.xts(read.zoo(tmpfile, sep=",", header=TRUE, format="%m/%d/%Y"))
# each column of r has daily log returns for a stock price series
# use `apply` to apply a function to each column.
vols.mat <- apply(r, 2, function(x) {
#use rolling 30 day window to calculate standard deviation.
#annualize by multiplying by square root of time
runSD(x, n=30) * sqrt(252)
})
#`apply` returns a `matrix`; `reclass` to `xts`
vols.xts <- reclass(vols.mat, r) #class as `xts` using attributes of `r`
tail(vols.xts)
# SPY.Adjusted DIA.Adjusted
#2012-06-22 0.1775730 0.1608266
#2012-06-25 0.1832145 0.1640912
#2012-06-26 0.1813581 0.1621459
#2012-06-27 0.1825636 0.1629997
#2012-06-28 0.1824120 0.1630481
#2012-06-29 0.1898351 0.1689990
#Clean-up
unlink(tmpfile)