Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about programming within the scope defined in the help center.
Closed 1 year ago.
Improve this question
I'm trying to interpret a classification report from a neural network that predicts mortality by heart failure as below. I can tell that predicting 0 is more accurate than predicting 1based on the f1-scores. However, I'm not sure why there's a fairly big gap between the precision value and recall value for both 0 and 1. Does this mean that my model has to be tuned or the dataset is biased? I would like to know how to proceed from this. Thanks!
Let us try to understand this further by looking at this example below.
Example - Confusion matrix
The Precision and recall for each label is given below. It is similar to the figures shown in your example.
So in the actual dataset, there are 70 records labelled as 1 and 95 records labelled as 0.
Confusion Matrix
Precision
Precision Formula = True Positives / (True Positives + False Positives)
In case of label 1, Out of the 31 which were classified as 1 by the model, 30 were correct and only 1 was incorrect. Hence its precision score is high which is 96.7%
In case of label 0, Out of the 134 which were classified as 0 by the model, 94 were correct but 40 was incorrect. Hence its precision score is at moderate level of ~70%.
Recall
Recall Formula = True Positives / (True Positives + False Negatives)
In case of label 1, Out of the 70 labels that are actually 1, the model could only identify 30 correctly as 1. Hence its recall score is low at ~43%.
In case of label 0, Out of the 95 labels that are actually 0, the model could identify 94 correctly as 0. Hence its recall score is high at ~99%.
Now to answer your question on whether you should tune the model depends on what is the result that you are looking for. If you are looking for good precision and recall on both labels, then yes should tune them and get a decent f1-score on both of them. But if you are more concerned about recall than precision, then you should let go of precision and improve recall and viceversa.
Related
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 3 years ago.
Improve this question
I am working on algorithm where i can have any number of 16 bit values(For instance i have 1000 16 bit values , and all are sensor data, so no particular series or repetition). I want to stuff all of this data into an 8 or a 10 byte array(each and every value of the 1000 16 bits numbers should be inside the 10 byte array) . The information should be such that i can also easily decode to read each and every value from the 1000 values.
I have thought of using sin function by dividing the values by 100 so every data point would always be in 8 bits(0-1 sin value range) , but that only covers up small range of data and not huge number of values.
Pardon me if i am asking for too much. I am just curious if its possible or not.
The answer to this question is rather obvious with a little knowledge in information sciences. It is not possible to store that much information in so little memory, and the data you are talking about just contains too much information.
Some data, like repetitive data or data which is following some structure (like constantly rising values), contains very little information. The task of compression algorithms is to figure out the structure or repetition and instead of storing the pure data to store the structure or rule how to reproduce the data instead.
In your case, the data is coming from sensors and unless you are willing to lose a massive amount of information, you will not be able to generate a compressed version of it with a compression factor in the magnitude your are talking about (1000 × 2 bytes into 10 bytes). If your sensors more or less produce the same values all the time with just a little jitter, a good compression can be achieved (but for this your question is way to broad to be answered here) but it will probably never be in the range of reducing your 1000 values to 10 bytes.
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 7 years ago.
Improve this question
I am trying to classify the events above as 1 or 0. 1 would be the lower values and 0 would be the higher values. Usually the data is does not look as clean as this. Currently the approach I am taking is to have two different thresholds so that in order to go from 0 to 1 it has to go past the 1 to 0 threshold and it has to be above for 20 sensor values. This threshold is set to the highest value I receive minus ten percent of that value. I dont think a machine learning approach will work because I have too few features to work with and also the implementation has to take up minimal code space. I am hoping someone may be able to point me in the direction of a known algorithm that would apply well to this sort of problem, googling it and checking my other sources isnt producing great results. The current implementation is very effective and the hardware inst going to change.
Currently the approach I am taking is to have two different thresholds so that in order to go from 0 to 1 it has to go past the 1 to 0 threshold and it has to be above for 20 sensor values
Calculate the area on your graph of those 20 sensor values. If the area is greater than a threshold (perhaps half the peak value) assign it as 1, else assign it as 0.
Since your measurements are one unit wide (pixels, or sensor readings) the area ends up being the sum of the 20 sensor values.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
I have 8 data points that form the peak of a partial sine wave. I am trying to fit these to get an equation so I discover the point of the true maximum position (which most likely lies between the data points). The coding will be in C. Does anyone have any info on algorithms or ideally code samples?
Since the data points are all near a maximum, the wave y = A*sin(B*x + C) + D can be approximated as a parabola much like the first 2 terms of cos(x) = (1.0 - x*x/2! + ...).
So find the best fit parabola for the 8 data points and calculate the maximum.
C- Peak detection via quadratic fit
Lots of google examples exist. Example
Provided your sample-values form a "hump", i.e. increasing followed by decreasing samples, you could try viewing the samplevalues as "weights" and compute the "center of gravity":
float cog = 0f;
for (i=0; i<num_samples; ii+) {
cog += i * samples[i];
}
cog /= num_samples;
I've used that in similar cases in the past.
NOTE: This scheme only works if the set of samples used contain a single peak, which the question phrasing certainly made me think was the case. Finding locations of interest can easily be done by monitoring, if sample values are increasing or decreasing, selecting an "interesting" range of samples and computing the peak location as described.
Also note, that if the actual goal is to determine the sine wave phase or frequency of an input signal, it would be a lot better to correlate the signal against reference set of sine-waves (in other words, do a Fourier transform).
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Questions asking for code must demonstrate a minimal understanding of the problem being solved. Include attempted solutions, why they didn't work, and the expected results. See also: Stack Overflow question checklist
Closed 9 years ago.
Improve this question
i need some help with a math task our professor gave us. Any suggestions would help.
The problem is:
There are N cannibals and M missinaries. All missionaries have a strenght attribute, which can be 1 or any positive whole number. Strenght indicates how many cannibals he can fight off.
Basically: there are two sides of the river, there is a 2-slot boat, and you have to transfer all the guys to the other side, without letting the cannibals eat the missionaries.
How would you write a program for this? What would be the transferring-grouping algorythm?
Thanks in anticipation,
Mark.
Model your problem as a states graph.
In here, a state is ({L,R}n,{L,R}m,{L,R}) Where:
First n entries: one for each missionary - where he is: left/right bank of the river
next,m entries: one for each canibal- where he is: left/right bank of the river
Last entry is for the boat
These are your vertices - you should also trim the invalid states - where strength of missionaries is not enough in one (or more) side. It is easy to calculate it for each state.
Your edges are:
E = { (S1,S2) | Can move in one boat ride from S1 to S2 }
All is left to do - use some shortest path algorithm to find the shortest path from: (L,L,....,L) to (R,R,...,R).
You can use BFS for this task, or even bi-directional search - or an informed algorithm (with admissible heuristic) such as A* Algorithm.
PS. The 'graph' is just conceptual, in practice you will have a function next:S->2^S, that given a state - returns all valid successors of this state (states that you can get to them using one edge on the graph from S). This will allow you to "generate the graph" on the fly.
Your next(S) function should be something like (high level pseudo code, without optimizations):
next(S):
let x be the bank where the boat is, and y the other bank
for each person p1 on bank x:
S' = S where boat and p1 moved from x to y
if S' is valid according to strength limitations, yield S'
for each p2 != p1 on bank x:
S' = S where boat and p1 and p2 moved from x to y
if S' is valid according to strength limitations, yield S'
Closed. This question is off-topic. It is not currently accepting answers.
Want to improve this question? Update the question so it's on-topic for Stack Overflow.
Closed 10 years ago.
Improve this question
There are some related questions that I've come across (like this, this, this, and this) but they all deal with fitting data to a known curve. Is there a way to fit given data to an unknown curve? By which I mean, given some data the algorithm will give me a fit which is one function or a sum of functions. I'm programming in C, but I'm at a complete loss on how to use the gsl package to do this. I'm open to using anything that can (ideally) be piped through C. But any help on what direction I should look will be greatly appreciated.
EDIT: This is basically experimental (physics) data that I've collected, so the data will have some trend modified by additive gaussian distributed noise. In general the trend will be non-linear, so I guess that a linear regression fitting method will be unsuitable. As for the ordering, the data is time-ordered, so the curve necessarily has to be fit in that order.
You might be looking for polynomial interpolation, in the field of numerical analysis.
In polynomial interpolation - given a set of points (x,y) - you are trying to find the best polynom that fits these points. One way to do it is using Newton interpolation, which is fairly easy to program.
The field of numerical analysis and interpolations in specifics is widely studied, and you might be able to get some nice upper bound to the error of the polynom.
Note however, because you are looking for a polynom that best fits your data, and the function is not really a polynom - the scale of the error when getting far from your initial training set blasts off.
Also note, your data set is finite, and there are inifnite number (actually, non-enumerable infinity) of functions that can fit the data (exactly or approximately) - so which one out of these is the best might be specific to what you actually are trying to achieve.
If you are looking for a model to fit your data, note that linear regression and polynomial interpolations are at the opposite ends of the scale: polynomial interpolation might be an overfitting to a model, while a linear regression might be underfitting it, what exactly should be used is case specific and varies from one application to the other.
Simple polynomial interpolation example:
Let's say we have (0,1),(1,2),(3,10) as our data.
The table1 we get using newton method is:
0 | 1 | |
1 | 2 | (2-1)/(1-0)=1 |
3 | 9 | (10-2)/(3-1)=4 | (4-1)/(3-0)=1
Now, the polynom we get is the "diagonal" that ends with the last element:
1 + 1*(x-0) + 1*(x-0)(x-1) = 1 + x + x^2 - x = x^2 +1
(and that is a perfect fit indeed to the data we used)
(1) The table is recursively created: The first 2 columns are the x,y values - and each next column is based on the prior one. It is really easy to implement once you get it, the full explanation is in the wikipedia page for newton interpolation.
Another alternative is using linear regression, but multi-dimensional.
The trick here is to artificially generate extra dimensions. You can do so by simply implying some functions on the original data set. A common usage is doing it to generate polynoms to match the data, so in here the function you imply is f(x) = x^i for all i < k (where k is the degree of the polynom you want to get).
For example, the data set (0,2),(2,3) with k = 3 you will get extra 2 dimnsions, and your data set will be: (0,2,4,8),(2,3,9,27).
The linear-regression algorithm will find the values a_0,a_1,...,a_k for the polynom p(x) = a_0 + a_1*x + ... + a_k * x^k that minimized the error for each point in the data comparing to the predicted model (the value of p(x)).
Now, the problem is - when you start increasing the dimension - you are moving from underfitting (of 1 dimensional linear regression) to overfitting (when k==n, you effectively getting polynomial interpolation).
To "chose" what is the best k value - you can use cross-validation, and chose the k that minimized the error according to your cross-validation.
Note that this process can be fully automated, all you need is to iteratively check all k values in the desired range1, and chose the model with the k that minimized the error according to the cross-validation.
(1) The range could be [1,n] - though it will probably be way too time consuming, I'd go for [1,sqrt(n)] or even [1,log(n)] - but it is just a hunch.
You might want to use (Fast) Fourier Transforms to convert data to frequency domain.
With the result of the transform (a set of amplitudes and phases and frequencies) even the most twisted set of data can be represented by several functions (harmonics) of the form:
r * cos(f * t - p)
where r is the harmonic amplitude, f is the frequency an p the phase.
Finally, the unknonwn data curve is the sum of all harmonics.
I have done this in R (you have some examples of it) but I believe C has enough tools to manage it. It is also possible to pipe C and R but don't know much about it. This might be of help.
This method is really good for large chunks of data because it has complexities of:
1) decompose data with Fast Fourier Transforms (FTT) = O(n log n)
2) built the function with the resulting components = O(n)