I am trying to perform Adaboost training stated by Viola and Jones in their paper on rapid object detection. However, I do not understand how to get the threshold values that will classify the faces from non faces for each of the 160k features. Is this a threshold you set manually? or is this based on some kind of maths ?
Can someone please explain the maths to me thanks a lot.
IMO, the best way to describe what happens during threshold assignment of the weak classifiers in every boosting round is a ROC analysis of the weak classifier performance. A great introduction on ROC analysis was written by Tom Fawcett. The full algorithm that does what you want is described in Shappire and Freund`s book, section 3.4.2.
Related
I am implementing a functional face detection algorithm in C using Viola and Jones algorithm. I'm having trouble understanding Adaboost to train a strong classifier.
I can detect all 5 basic haar-features in a single image (162336 in a 24x24 image) I'm pretty sure this is good and working, and my algorithm outputs and array containing all the features sorted.
Then, I started working on Adaboost and here's what I understand. We create a weak classifier (slightly better than random) and we make a linear combination of many weak classifier (approx 200) to get a strong classifier.
What I don't understand is how to create this weak classifier. From what I read online:
Normalize the weights of our training examples (first round 1 by default)
Then get a feature (here's one of my problem, do I have to process each feature of each training example ? (162336 * number of examples) that would be a lot of computing power no ? )
"Apply" this feature to each image to get an optimal treshold and toggle (here's my main problem, I don't understand what "apply" means here, compare it with each feature of the image ? I really don't see what I have to do with it. Then, I don't understand what is the treshold and the toggle and that's where i'm looking for help)
Then many more other things to do
I'm really looking forward your help to make me understand this!
Should have answered my own question faster, but I've forgot about it. It was a project for my computer science school so I can provide answers.
Adaboost is in fact fairly simple when you understand it.
First you need to detect features inside every images in your base (we used 4000 images to have a large set) you can store them if you have enough memory or process them when you need them in you program. For 4000 images with 5 haar features inside we used more than 16Gb of RAM (Code was written in c, but no memory leak, it was arrays of double)
The training algorithm assign a weight to an image. That weight represent the difficulty for the algorithm to make a good prediction (face or no face).
Your training algorithm will be composed of rounds (200 rounds is fine to have 90%+ of good prediction).
At the first round every image possess the same weight because the algorithm never worked on them.
Here is how a round goes:
Find the best haar feature among X (for each type) in each image. To do this, compare each feature to the same one (same type, dimension and position) on every image and see if it is a good or bad prediction. The feature with the best prediction inside the X features is the best one, keep it stored.You will find 5 best features per images because there is 5 type combine them in a single struct and it is your weak classifier
Calculate the weighted error of the classifier. The weighted error is the error of the weak classifier applied to each image while taking in account the weight assigned to each image. In later rounds, the image with a bigger weight (the algorithm made lot of mistakes about this image) will be taken much more into account.
Add the weak classifier to a strong classifier (which is an array of weak classifiers) and its alpha. The alpha is the performance of the weak classifier and is determined with the weighted error. With the alpha, weak classifier which were made at later stage of the algorithm when the training is harder will have more weight in the final prediction of the strong classifier.
Update the weight of each image according to the prediction of the weak classifier you just created. If the classifier is right the weight goes down otherwise it goes up.
At the end of the 200 rounds of training, you will possess a strong classifier composed of 200 weak classifier. To make a prediction about a single image, apply each weak classifier to the image and the majority wins.
I voluntarily simplified the explanation but the majority is here. For more informations look here, it really helped me during my project: http://www.ipol.im/pub/art/2014/104/article.pdf
I suggest every person interested in AI and optimisation to work on a project like that. As a student it made me really interested in AI and made me think a lot about optimisation, which I never did before.
I have a good basis on Evolutionary Algorithms, so now i started to read about Artificial Neural Networks. I come across this tutorial on
http://www.ai-junkie.com/ann/evolved/nnt2.html,
showing how to use a ANN to evolve Tanks that collect mines. It uses a GA to evolve the input weights on each Neuron.
I know i could use GA (without the ANN) to solve the same problem. I already created a Tetris Bot using only GA to optimize the weights in the grid evaluation function (check my blog http://www.bitsrandomicos.blogspot.com.br/).
My question is: what's the conceptual/practical difference between using a ANN + GA in a situation where i could use GA alone? I mean, is my Tetris Bot a ANN?(I don't think so).
There are several related questions about this, but i couldn't find a answer:
Are evolutionary algorithms and neural networks used in the same domains?
When to use Genetic Algorithms vs. when to use Neural Networks?
Thanks!
A genetic algorithm is an optimization algorithm.
An artificial neural network is a function approximator. In order to approximate a function you need an optimization algorithm to adjust the weights. An ANN can be used for supervised learning (classification, regression) or reinforcement learning and some can even be used for unsupervised learning.
In supervised learning a derivative-free optimization algorithm like a genetic algorithm is slower than most of the optimization algorithms that use gradient information. Thus, it only makes sense to evolve neural networks with genetic algorithms in reinforcement learning. This is known as "neuroevolution". The advantage of neural networks like multilayer perceptrons in this setup is that they can approximate any function with arbitrary precision when they have a suffficient number of hidden nodes.
When you create a tetris bot you do not necessarily have to use an ANN as a function approximator. But you need some kind of function approximator to represent your bot's policy. I guess it was just simpler than an ANN. But when you want to create a complex nonlinear policy you could do that e. g. with an ANN.
alfa's answer is perfect. Here is just an image to illustrate what he said:
Meta-Optimizer = None (but could be)
Optimizer = Genetic Algorithm
Problem = Tetris Bot (e.g. ANN)
You use evolutionary algorithm if you yet don't know the answer but you are able to somehow rate candidates and provide meaningful mutations.
Neural network is great if you already have answers (and inputs) and you want to "train the computer" so it can "guess" the answers for unknown inputs. Also, you don't have to think a lot about the problem, the network will figure it out by itself.
Check this "game AI" example: https://synaptic.juancazala.com/#/
(note how simple it is, all you have to do is to give them enough training, you don't have to know a thing about game AI - and once it is good enough all you have to do is to "download" memory and run it when needed)
I'm not an expert, but based on what I know from the field..
An artificial neural network has a basis on neuroscience ultimately. It attempts to simulate/model its behavior through building a neuron-like structures in the algorithm. There is a strong emphasis on the academic nature of the problem than the result. From what I understand, its for this reason that ANN's are not very popular from an engineering standpoint. Statistical basis of machine learning (HMM's and Bayesian networks) produce better results.
In short, so as long as it has a nod towards some underlying neurosciency subject, it can be a ANN, even if it uses some form of GA.
If you use a GA, it is not necessarily an ANN.
I have a typical AI Problem to solve. Customers gonna submit comments about a product. I have to be able to create a program that classify these comment as either good,bad or neutral.
Surely, Neural Network gonna play a great role in it.
Also, I think fuzzy logic can play some role in it. Such as how far a comment is good,bad or neutral!!
Some more ideas about how to solve it??
This problem is usually referred to as Sentiment Analysis. You can check out the wikipedia entry about Sentiment Analysis for a brief review, or Liu Bing's page on sentiment analysis for more detailed resources and tutorials.
You can use some form of supervised learning.
The most important thing for classification is then choosing the right features. "Features" means you extract some values from the review that still capture the essence with respect to the classification task. Things that come to my mind are
number of words
average number of words per sentence
number of words from some set like {crap, shit, damn, viagra, ...}
Then you can use any available machine learning algorithm (neural networks, SVM) and train a classifier given you have enough reviews that are labeled with good/neutral/bad.
Neural networks would certainly work for it, however I would be supicious about introducing new words, and languages. I would go for a Bayes net approach for determining the probability of being in a "good/neutral/bad" state. You should consider cleaning the data [stemming, etc] before putting it through the bayes net.
Additionally: The meta attributes [what ziggy mentioned] are more of an indicator to boost the performance of the approach you take.
EDIT: Bayes-Nets are a form of supervised learning.
I am new to machine learning. I am familiar with SVM , Neural networks and GA. I'd like to know the best technique to learn for classifying pictures and audio. SVM does a decent job but takes a lot of time. Anyone know a faster and better one? Also I'd like to know the fastest library for SVM.
Your question is a good one, and has to do with the state of the art of classification algorithms, as you say, the election of the classifier depends on your data, in the case of images, I can tell you that there is one method called Ada-Boost, read this and this to know more about it, in the other hand, you can find lots of people are doing some researh, for example in Gender Classification of Faces Using Adaboost [Rodrigo Verschae,Javier Ruiz-del-Solar and Mauricio Correa] they say:
"Adaboost-mLBP outperforms all other Adaboost-based methods, as well as baseline methods (SVM, PCA and PCA+SVM)"
Take a look at it.
If your main concern is speed, you should probably take a look at VW and generally at stochastic gradient descent based algorithms for training SVMs.
if the number of features is large in comparison to the number of the trainning examples
then you should go for logistic regression or SVM without kernel
if the number of features is small and the number of training examples is intermediate
then you should use SVN with gaussian kernel
is the number of features is small and the number of training examples is large
use logistic regression or SVM without kernels .
that's according to the stanford ML-class .
For such task you may need to extract features first. Only after that classification is feasible.
I think feature extraction and selection is important.
For image classification, there are a lot of features such as raw pixels, SIFT feature, color, texture,etc. It would be better choose some suitable for your task.
I'm not familiar with audio classication, but there may be some specturm features, like the fourier transform of the signal, MFCC.
The methods used to classify is also important. Besides the methods in the question, KNN is a reasonable choice, too.
Actually, using what feature and method is closely related to the task.
The method mostly depends on problem at hand. There is no method that is always the fastest for any problem. Having said that, you should also keep in mind that once you choose an algorithm for speed, you will start compromising on the accuracy.
For example- since your trying to classify images, there might a lot of features compared to the number of training samples at hand. In such cases, if you go for SVM with kernels, you could end up over fitting with the variance being too high.
So you would want to choose a method that has a high bias and low variance. Using logistic regression or linear SVM are some ways to do it.
You could also use different types of regularizations or techniques such as SVD to remove the features that do not contribute much to your output prediction and have only the most important ones. In other words, choose the features that have little or no correlation between them. Once you do this, you would be able to speed yup your SVM algorithms without sacrificing the accuracy.
Hope it helps.
there are some good techniques in learning machines such as, boosting and adaboost.
One method of classification is the boosting method. This method will iteratively manipulate data which will then be classified by a particular base classifier on each iteration, which in turn will build a classification model. Boosting uses weighting of each data in each iteration where its weight value will change according to the difficulty level of the data to be classified.
While the method adaBoost is one ensamble technique by using loss function exponential function to improve the accuracy of the prediction made.
I think your question is very open ended, and "best classifier for images" will largely depend on the type of image you want to classify. But in general, I suggest you study convulutional neural networks ( CNN ) and transfer learning, currently these are the state of the art techniques for the problem.
check out pre-trained models of cnn based neural networks from pytorch or tensorflow
Related to images I suggest you also study pre-processing of images, pre-processing techniques are very important to highlight some feature of the image and improve the generalization of the classifier.
I've got a classification problem in my hand, which I'd like to address with a machine learning algorithm ( Bayes, or Markovian probably, the question is independent on the classifier to be used). Given a number of training instances, I'm looking for a way to measure the performance of an implemented classificator, with taking data overfitting problem into account.
That is: given N[1..100] training samples, if I run the training algorithm on every one of the samples, and use this very same samples to measure fitness, it might stuck into a data overfitting problem -the classifier will know the exact answers for the training instances, without having much predictive power, rendering the fitness results useless.
An obvious solution would be seperating the hand-tagged samples into training, and test samples; and I'd like to learn about methods selecting the statistically significant samples for training.
White papers, book pointers, and PDFs much appreciated!
You could use 10-fold Cross-validation for this. I believe it's pretty standard approach for classification algorithm performance evaluation.
The basic idea is to divide your learning samples into 10 subsets. Then use one subset for test data and others for train data. Repeat this for each subset and calculate average performance at the end.
As Mr. Brownstone said 10-fold Cross-Validation is probably the best way to go. I recently had to evaluate the performance of a number of different classifiers for this I used Weka. Which has an API and a load of tools that allow you to easily test the performance of lots of different classifiers.