I would like to know how noise can be removed from data (say, radio data that is an array of rows and columns with each data point representing intensity of the radiation in the given frequency and time).The array can contain radio bursts. But many fixed frequency radio noise also exists(RFI=radio frequency intereference).How to remove such noise and bring out only the burst.
I don't mean to be rude, but this question isn't clear at all. Please sharpen it up.
The normal way to remove noise is first to define it exactly and then filter it out. Usually this is done in the frequency domain. For example, if you know the normalized power spectrum P(f) of the noise, build a filter with response
e/(e + P(f))
where e<1 is an attenuation factor.
You can implement the filter digitally using FFT or a convolution kernel.
When you don't know the spectrum of the noise or when it's white, then just use the inverse of the signal band.
Related
to get a moving average like filter on ADC data in microcontrollers, i like to use the following code:
average = average + ((new_value - average)>>2);
it works nicely and super fast and I can adjust the filterstrength just by changing the amount of right-shift.
My question is now: has this filter a name?
cheers :)
Somewhere between Cumulative Moving Average and Weighted Moving Average
In general terms, it can be a specific case of Savitzky-Golay method as well. It is a low pass filter with the following explantion:
A Savitzky–Golay filter is a digital filter that can be applied to a set of digital data points for the purpose of smoothing the data, that is, to increase the precision of the data without distorting the signal tendency. This is achieved, in a process known as convolution, by fitting successive sub-sets of adjacent data points with a low-degree polynomial by the method of linear least squares. When the data points are equally spaced, an analytical solution to the least-squares equations can be found, in the form of a single set of "convolution coefficients" that can be applied to all data sub-sets, to give estimates of the smoothed signal, (or derivatives of the smoothed signal) at the central point of each sub-set.
I read the very interesting and informative article on image noise located here:
http://www.cambridgeincolour.com/tutorials/image-noise-2.htm
One of the key points is noise frequency which can be high or low. I was curious as to how this would be calculated?
I came across another article in which the developer achieved low frequency iage noise by sampling the surrounding pixels and then performing an average. But this requires a separate calculation pass after all the noise has been calculated.
Is this how it's traditionally done or is there a different way to calculate image noise frequency?
Thanks
Look up Perlin noise.
"Salt and pepper noise" is white or black pixels scattered at random over the source. It's a fairly common sort of noise, but not useful for much visually. It has the highest frequency that the image sampling supports.
Perlin noise consists of noise of several frequencies, or "octaves" superimposed on each other. It is useful, with various parameters it looks like wood, or clouds, or swirling lava. It can also be used a bit more subtly to give slightly rugged effects to non-smooth surfaces.
The frequency is simply the distance in pixels at which existence of noise at one pixel bears no relation to the noise at the other pixel (for non-repeating noise) or the "repeat length" for repeating noise.
I have a 2 column vector with times and speeds of a subset of data, like so:
5 40
10 37
15 34
20 39
And so on. I want to get the fourier transform of speeds to get a frequency. How would I go about doing this with a fast fourier transform (fft)?
If my vector name is sampleData, I have tried
fft(sampleData);
but that gives me a vector of real and imaginary numbers. To be able to get sensible data to plot, how would I go about doing this?
Fourier Transform will yield a complex vector, when you fft you get a vector of frequencies, each has a spectral phase. These phases can be extremely important! (they contain most of the information of the time-domain signal, you won't see interference effects without them etc...). If you want to plot the power spectrum, you can
plot(abs(fft(sampleData)));
To complete the story, you'll probably need to fftshift, and also produce a frequency vector. Here's a more elaborate code:
% Assuming 'time' is the 1st col, and 'sampleData' is the 2nd col:
N=length(sampleData);
f=window(#hamming,N)';
dt=mean(diff(time));
df=1/(N*dt); % the frequency resolution (df=1/max_T)
if mod(N,2)==0
f_vec= df*((1:N)-1-N/2); % frequency vector for EVEN length vector
else
f_vec= df*((1:N)-0.5-N/2);
end
fft_data= fftshift(fft(fftshift(sampleData.*f))) ;
plot(f_vec,abs(fft_data))
I would recommend that you back up and think about what you are trying to accomplish, and whether an FFT is an appropriate tool for your situation. You say that you "want to ... get a frequency", but what exactly do you mean by that? Do you know that this data has exactly one frequency component, and want to know what the frequency is? Do you want to know both the frequency and phase of the component? Do you just want to get a rough idea of how many discrete frequency components are present? Are you interested in the spectrum of the noise in your measurement? There are many questions you can ask about "frequencies" in a data set, and whether or not an FFT and/or power spectrum is the best approach to getting an answer depends on the question.
In a comment above you asked "Is there some way to correlate the power spectrum to the time values?" This strikes me as a confused question, but also makes me think that maybe the question you are really trying to answer is "I have a signal whose frequency varies with time, and I want to get an estimate of the frequency vs time". I'm sure I've seen a question along those lines within the past few months here on SO, so I would search for that.
What is a simple way to see if my low-pass filter is working? I'm in the process of designing a low-pass filter and would like to run tests on it in a relatively straight forward manner.
Presently I open up a WAV file and stick all the samples in a array of ints. I then run the array through the low-pass filter to create a new array. What would an easy way to check if the low-pass filter worked?
All of this is done in C.
You can use a broadband signal such as white noise to measure the frequency response:
generate white noise input signal
pass white noise signal through filter
take FFT of output from filter
compute log magnitude of FFT
plot log magnitude
Rather than coding this all up you can just dump the output from the filter to a text file and then do the analysis in e.g. MATLAB or Octave (hint: use periodogram).
Depends on what you want to test. I'm not a DSP expert, but I know there are different things one could measure about your filter (if that's what you mean by testing).
If the filter is linear then all information of the filter can be found in the impulse response. Read about it here: http://en.wikipedia.org/wiki/Linear_filter
E.g. if you take the Fourier transform of the impulse response, you'll get the frequency response. The frequency response easily tells you if the low-pass filter is worth it's name.
Maybe I underestimate your knowledge about DSP, but I recommend you to read the book on this website: http://www.dspguide.com. It's a very accessible book without difficult math. It's available as a real book, but you can also read it online for free.
EDIT: After reading it I'm convinced that every programmer that ever touches an ADC should definitely have read this book first. I discovered that I did a lot of things the difficult way in past projects that I could have done a thousand times better when I had a little bit more knowledge about DSP. Most of the times an unexperienced programmer is doing DSP without knowing it.
Create two monotone signals, one of a low frequency and one of a high frequency. Then run your filter on the two. If it works, then the low frequency signal should be unmodified whereas the high frequency signal will be filtered out.
Like Bart above mentioned.
If it's LTI system, I would insert impulse and record the samples and perform FFT using matlab and plot magnitude.
You ask why?
In time domain, you have to convolute the input x(t) with the impulse response d(t) to get the transfer function which is tedious.
y(t) = x(t) * d(t)
In frequency domain, convolution becomes simple multiplication.
y(s) = x(s) x d(s)
So, transfer function is y(s)/x(s) = d(s).
That's the reason you take FFT of impulse response to see the behavior of the filter.
You should be able to programmatically generate tones (sine waves) of various frequencies, stuff them into the input array, and then compare the signal energy by summing the squared values of the arrays (and dividing by the length, though that's mathematically not necessary here because the signals should be the same length). The ratio of the output energy to the input energy gives you the filter gain. If your LPF is working correctly, the gain should be close to 1 for low frequencies, close to 0.5 at the bandwidth frequency, and close to zero for high frequencies.
A note: There are various (but essentially the same in spirit) definitions of "bandwidth" and "gain". The method I've suggested should be relatively insensitive to the transient response of the filter because it's essentially averaging the intensity of the signal, though you could improve it by ignoring the first T samples of the input, where T is related to the filter bandwidth. Either way, make sure that the signals are long compared to the inverse of the filter bandwidth.
When I check a digital filter, I compute the magnitude response graph for the filter and plot it. I then generate a linear sweeping sine wave in code or using Audacity, and pass the sweeping sine wave through the filter (taking into account that things might get louder, so the sine wave is quiet enough not to clip) . A visual check is usually enough to assert that the filter is doing what I think it should. If you don't know how to compute the magnitude response I suspect there are tools out there that will compute it for you.
Depending on how certain you want to be, you don't even have to do that. You can just process the linear sweep and see that it attenuated the higher frequencies.
I love to work on AI optimization software (Genetic Algorithms, Particle Swarm, Ant Colony, ...). Unfortunately I have run out of interesting problems to solve. What problem would you like to have solved?
This list of NP complete problems should keep you busy for a while...
How about the Hutter Prize?
From the entry on Wikipedia:
The Hutter Prize is a cash prize
funded by Marcus Hutter which rewards
data compression improvements on a
specific 100 MB English text file.
[...]
The goal of the Hutter Prize is to
encourage research in artificial
intelligence (AI). The organizers
believe that text compression and AI
are equivalent problems.
Basically the idea is that in order to make a compressor which is able to compress data most efficiently, the compressor must be, in Marcus Hutter's words, "smarter". For more information on the relation between artificial intelligence and compression, see the Motivation and FAQ sections of the Hutter Prize website.
Does the Netflix Prize count?
I would like my bank balance optimised so that there is as much money as possible left at the end of the month, instead of the other way round.
What about the Go Game ?
Here's an interesting practical problem I came up while tinkering with color quantization and image compression.
The basic idea is that I would like a program to which I give a picture and it reduces the amount of colors is it as much as possible without me noticing it. Since every person has a different sensitivity of the eye (and eyes have different sensitivity of red/green/blue intensities), it should be possible to specify this sensitivity threshold in some way.
In other words, in a truecolor picture, replace every pixel's color with another color so that:
The total count of different colors in a picture would be the smallest possible; and
Every new pixel would have it's color no further from the original color than some user-specified value D.
The D can be defined in different ways, pick your favorite. For example:
Separate red, green and blue components for specifying the maximum possible deviation for each of them (for every pixel you get a rectangular cuboid of valid replacement values);
A real number which would represent the maximum allowable distance in the RGB cube (for every pixel you get a sphere of valid replacement values);
Something inbetween or completely different.
Most efficient solution to a given set of Sudoku puzzles. (excluding brute-force methods)