What is the fastest algorithm for compressing RGBA 32 bit image data? I am working in C, but am happy for examples in other programming languages.
Right now I am using LZ4 but I am considering run length / delta encoding.
Lossless encoding, a mix of real life images and computer generated / clipart images. Alpha channel always exists, but is usually constant.
I ended up just using LZ4. Nothing else was even close to as fast and LZ4 usually got at least 50% size reduction.
Lossy or lossless?
"Real" images or computer graphics?
Do you actually have an alpha channel?
If you need lossless (or semi=lossless) then converting into YUV and compressing that will probably reduce by about 1/2 (after already having it in going to 2bytes/pixel) try Huffyuv
If you have real images then H264 can do very high compression and there are optomised libraries and HW support so it can be very fast.
If you have computer graphics type images with few colours but need to preserve edges, or you actually have an A channel then run length might be good - try splitting the image into per-colour frames first.
LZ4 is LZ77 family which is a few lines of code but I never did it myself but I guess you are right run length or delta code is the fastest and also good for images. There is also snappy algorithm. Recently I tried the exdupe utility to compress my virtual machines. This thing is also incredible fast: http://www.exdupe.com. exdupe seems to use a rzip thing: http://encode.ru/threads/1354-Data-deduplication.
Related
edit: I try to rephrase this as to make this clearer the best I can :)
I need to find a suitable way / choose a suitable compression to store a blob of data (say approx. 900KB) in a ROM where the available amount of free space is only about 700KB. If I compress the blob with some modern compression tool (eg. WinZIP/WinRAR) I can achieve the required compression easily.
The matter here is that the decompression will take place on a very VERY VERY limited hardware where I can't afford to have more than few bytes of RAM available (say no more than 100 bytes, for the sake of it).
I already tried RLE'ing the data... the data hardly compress.
While I'm working trying to change the data blob format so that it could potentially have more redundancy and achieve better compression ratio, I'm at the same time seeking a compression method that will enable me to decompress on my limited hardware. I have a limited knowledge of compression algorithms so I'm seeking suggestions/pointers to continue with my hunt.
Thanks!
Original question was "I need info/pointers on decompression algorithms that can work without using the uncompressed data, as this will be unavailable right after decompression. LZ like approaches would still be preferred."
I'm afraid this is off topic because too broad.
LZW uses a sizable state that is not very different from keeping a slice of uncompressed data. Even if the state is constant and read from ROM, handling it with just registers seems difficult. There are many different algorithms than can use a constant state, but if you really have NO RAM, then only the most basic algorithms can be used.
Look up RLE, run length encoding.
EDIT: OK, no sliding window, but if you can access ROM, 100 bytes of RAM give you quite some possibilities. You want to implement this in assembly, so stick with very simple algorithms. RLE plus a dictionary. Given your requirements, the choice of algorithm should be based on the type of data you need to decompress.
I'm working with radio-to-radio communications where bandwidth is really really precious. It's all done with on the metal C code (no OS, small atmel 8bit microprocessors). So the idea of compression becomes appealing for some large, but rare, transmissions.
I'm no compression expert. I've used the command line tools to shrink files and looked at how much I get. And linked a library or two over the years. But never anything this low level.
In one example, I want to move about 28K over the air between processors. If I just do a simple bzip2 -9 on a representative file, I get about 65% of the original size.
But I'm curious if I can do better though. I am (naively?) under the impression that most basic compression formats must be some declaration of metadata up front, that describes how to inflate a bitstream that follows. What I don't know is how much space that metadata itself takes up. I histogram'ed said same file, and a number of other ones, and found that due to the nature of what's being transmitted, the histogram is almost always about the same. So I'm curious if I could hard code these frequencies in my code so that that was no longer dynamic, but also wasn't transmitted as part of the packet.
For example, my understanding of a huffman encoding is that usually there's a "dictionary" up front, followed by a bitstream. And that if a compressor does it by blocks, each block will have its own dictionary.
On top of this, it's a small processor, with a small footprint, I'd like to keep whatever I do small, simple, and straightforward.
So I guess the basic question is, what, if any, basic compression algorithm would you implement in this kind of environment/scenario. Especially taking into account, that you can basically precompile a representative histogram of the bytes per transmission.
What you are suggesting, providing preset frequency data, would help very little. Or more likely it would hurt, since you will take a hit by not using optimal codes. As an example, only about 80 bytes at the start of a deflate block is needed to represent the literal/length and distance Huffman codes. A slight increase in the, say, 18 KB of your compressed data could easily cancel that.
With zlib, you could use a representative one of your 28K messages as a dictionary in which to search for matching strings. This could help the compression quite a bit, if there are many common strings in your messages. See deflateSetDictionary() and inflateSetDictionary().
One of the big deals in Silverlight v4 is audio/video capture... but I haven't found an example yet that does what I want to do. So:
How do you capture audio/video with Silverlight (from a webcam), and then save it as a compressed format (WMV or MP4)? The idea here is to upload it after compression.
Have already looked at this blog post for the capture piece, but need to find a way to compress audio/video for upload.
Silverlight does not support video encoding and more likely this won't be implemented at least by Microsoft. To transmit video over network, some people use "pseudo-MJPEG" codec by compressing individual frames as regular JPEG images. Some people even improved that idea by dividing frames into fixed block (say 8x8), and only transmits changed blocks (with lossy comparison).
If you're a veteran programmer and enjoy coding, here is another slightly improved version of "psuedo-MJPEG" idea:
Divide current frame into fixed 8x8 block
Apply RGB -> YCbCr color space conversion for each block
Down sample Cb and Cr plane by half
Apply DCT to YCbCr
Quantize DCT coefficients with a quantization matrix
Compare this DCT coefficients with previous frame's block. This way you make "perceptually lossy" comparison for each consecutive frames.
Use a bit-wise range-coder and encode a flag for unchanged blocks
For changed blocks, transmit DCT coefficient by modeling them (you can use JPEG's standard zig-zag pattern and zero-run model) and encode them with range coder.
This is more or less a standard JPEG algorithm actually. But, actual advantages over standard JPEG are:
Perceptually lossy comparison for blocks
Stronger compression due to both small overhead and stronger entropy coder (range coder)
Another option could be pay for 3rd party software (sorry, I don't know any free software). I find that product. I didn't used it at all. But, I believe it could be useful for you.
Where Can I find algorithm details for holistic word recognition? I need to build a simple OCR system in hardware (FPGAs actually), and the scientific journals seems so abstract?
Are there any open source (open core) codes for holistic word recognition?
Thanks
For an algorithm that is quite suitable for FPGA implementation (embarrassingly parallel) you might look at:
http://en.wikipedia.org/wiki/Cross-correlation
It is fast, and easily implemented.
The only thing is: it recognizes a shape (in your case some text) DEPENDENT of the rotation and size / stretch / skew etc. But if that isn't a problem, it can be very fast and is quite robust. You should only watch out for interpretation problems with characters that are similar (like o and c).
I used it to find default texts on scanned forms to obtain bearings where Region of Interests are and searching in those images (6M pixels) only took around 15 ms with our implementation on a Core2 CPU in a single thread.
I'm a physicist that normally deals with large amounts of numerical data generated using C programs. Typically, I store everything as columns in ASCII files, but this had led to massively large files. Given that I am limited in space, this is an issue and I'd like to be a little smarter about the whole thing. So ...
Is there a better format than ASCII? Should I be using binary files, or perhaps a custom format some library?
Should I be compressing each file individually, or the entire directory? In either case, what format should I use?
Thanks a lot!
In your shoes, I would consider the standard scientific data formats, which are much less space- and time-consuming than ASCII, but (while maybe not quite as bit-efficient as pure, machine-dependent binary formats) still offer standard documented and portable, fast libraries to ease the reading and writing of the data.
If you store data in pure binary form, the metadata is crucial to make any sense out of the data again (are these numbers single or double precision, or integers and of what length, what are the arrays' dimensions, etc, etc), and issues with archiving and retrieving paired data/metadata pairs can, and in practice do, occasionally make perfectly good datasets unusable -- a real pity and waste.
CDF, in particular, is "a self-describing data format for the storage and manipulation of scalar and multidimensional data in a platform- and discipline-independent fashion" with many libraries and utilities to go with it. As alternatives, you might also consider NetCDF and HDF -- I'm less familiar with those (and such tradeoffs as flexibility vs size vs speed issues) but, seeing how widely they're used by scientists in many fields, I suspect any of the three formats could give you very acceptable results.
If you need the files for a longer time, they are important experimental data that prove somethings for you or so, don't use binary formats. You will not be able to read them when your architecture changes. dangerous. stick to text (yes ascii) files.
Choose a compression format that fits your needs. Is compression time an issue? Usually not, but check that for yourself. Is decompression time an issue? Usually yes, if you want to do data analysis on it. Under these conditions I'd go for bzip2. This is quite common nowadays, well tested, foolproof. I'd do files individually, since the larger your file, the larger the probability of losses. (Bits flip etc).
A terabyte disk is a hundred bucks. Hard to run out of space these days. Sure, storing the data in binary saves space. But there's a cost, you'll have a lot less choices to get the data out of the file again.
Check what your operating system can do. Windows supports automatic compression on folders for example, the file content get zipped by the file system without you having to do anything at all. Compression rates should compete well with raw binary data.
There's a lot of info you didn't include, but should think about:
1.) Are you storing integers or floats? What is the typical range of the numbers?
For example: storing small comma-separated integers in ascii, such as "1,2,4,2,1" will average 2-bytes per datum, but storing them as binary would require 4-bytes per datum.
If your integers are typically 3 digits, then comma-separated vs binary won't matter much.
On the other hand, storing doubles (8-byte values) will almost certainly be smaller in binary format.
2.) How do you need to access these values? If you are not concerned about access time, compress away! On the other hand, if you need speedy, random access then compression will probably hinder you.
3.) Are some values frequently repeated? Then you may consider a Huffman encoding or a table of "short-cut" values.