process a large file via multithreading - c

Question

There is a quite large file (>10G) on the disk, each line inside the fie is composed of a line-number and a person's name, like this:
1 Jane
2 Perk
3 Sime
4 Perk
.. ..
I have to read this large file, and find the frequency of each name, finally output the results in descending order of each name's frequency, like this:
Perk 2
Jane 1
Sime 1
As the interviewer requested, the above job should be done as efficiently as possible, and multithreading is allowed. And my solution is something like this:
Because the file is too large, I partition the file into several small files, each small file is about 100M, via lseek I can locate the begin and the end of each small file (beg, end);
For these small files, there is a shared hash-map using person's name as key and how many times it shows so far as value;
For each small file, there is a single thread go through it, every time the thread encounters a person's name, it will increment its corresponding value in the shared hash-map;
When all threads finish, I think it's time to sort the hash-map according to the value field.
But because there might be too many names in that file, so the sorting would be slow. I didn't come up with a good idea about how to output the names in descending order.
Hope anyone can help me with the above problem, give me a better solution on how to do the job via multithreading and the sorting stuff.

Answer 1

Using a map-reduce approach could be a good idea for your problem. That approach would consist of two steps:
Map: read chunks of data from the file and create a thread to process that data
Reduce: the main thread waits for all other threads to finish and then it combines the results from each individual thread.
The advantage of this solution is that you would not need locking between the threads, since each one of them would operate on a different chunk of data. Using a shared data structure, as you are proposing, could be a solution too, but you may have some overhead due to contention for locking.
You need to do the sorting part at the reduce step, when the data from all the threads is available. But you might want to do some work during the map step, so that it is easier (quicker) to finish the complete sort at the reduce step.
If you prefer to avoid the sequential sorting at the end, you could use some custom data structure. I would use a map (something like a red-black tree or a hash table) for quickly finding a name. Moreover, I would use a heap in order to keep the order of frequencies among names. Of course, you would need to have parallel versions of those data structures. Depending on how coarse the parallelization is, you may have locking contention problems or not.

Answer 2

If I asked that as an interview question using the word "efficiently" I would expect an answer something like "cut -f 2 -d ' ' < file | sort | uniq -c" because efficiency is most often about not wasting time solving an already solved problem. Actually, this is a good idea, I'll add something like this to our interview questions.
Your bottleneck will be the disk so all kinds of multithreading is overdesigning the solution (which would also count against "efficiency"). Splitting your reads like this will either make things slower if there are rotating disks or at least make the buffer cache more confused and less likely to kick in a drop-behind algorithm. Bad idea, don't do it.

Answer 3

I don't think multithreading is a good idea. The "slow" part of the program is reading from disk, and multithreading the read from disk won't make it faster. It will only make it much more complex (for each chunk you have to find the first "full" line, for example, and you have to coordinate the various threads, and you have to lock the shared hash map each time you access it). You could work with "local" hash map and then merge them at the end (when all the threads finish (at the end of the 10gb) the partial hash maps are merged). Now you don't need to sync the access to the shared map.
I think that sorting the resulting hash map will be the easiest part, if the full hash map can be kept in memory :-) You simply copy it in a malloc(ed) block of memory and qsort it by its counter.

Answer 4

Your (2) and (4) steps in the solution make it essentially sequential (the second introduces locking to keep the hash-map consistent, and the last one, where you're attempting to sort all the data).
One-step sorting of the hash-map at the end is a little strange, you should use an incremental sorting technique, like heapsort (locking of the data structure required) or mergesort (sort parts of the "histogram" file, but avoid merging everything "in one main thread at the end" - try to create the sorting network and mix the contents of the output file at each step of the sorting).
Multi-threaded reads might be an issue, but with modern SSD drives and aggressive read caching multi-threading is not the main slowdown factor. It's all about synchronizing the results sorting process.
Here's a sample of mergesort's parallelization: http://dzmitryhuba.blogspot.com/2010/10/parallel-merge-sort.html
Once again, as I have said, some sorting network might help to allow efficient parallel sort, but not the straightforward "wait-for-all-subthreads-and-sort-their-results". Maybe, bitonic sort in case you have a lot of processors.

Answer 5

The interviewer's original question states "...and multithreading is allowed". The phrasing of this question might be a little ambiguous, however the spirit of the question is obvious: the interviewer is asking the candidate to write a program to solve the problem, and to analyse/justify the use (or not) of multithreading within the proposed solution. It is a simple question to test the candidate's ability to think around a large-scale problem and explain algorithmic choices they make, making sure the candidate hasn't just regurgitated something from an internet website without understanding it.
Given this, this particular interview question can be efficiently solved in O(n log n) (asymptotically speaking) whether multithreading is used or not, and multi-threading can additionally be used to logarithmically accelerate the actual execution time.
Solution Overview
If you were asked the OP's question by a top-flight company, the following approach would show that you really understood the problem and the issues involved. Here we propose a two stage approach:
The file is first partitioned and read into memory.
A special version of Merge Sort is used on the partitions that simultaneously tallies the frequency of each name as the file is being sorted.
As an example, let us consider a file with 32 names, each one letter long, and each with an initial frequency count of one. The above strategy can be visualised as follows:
1. File: ARBIKJLOSNUITDBSCPBNJDTLGMGHQMRH 32 Names
2. A|R|B|I|K|J|L|O|S|N|U|I|T|D|B|S|C|P|B|N|J|D|T|L|G|M|G|H|Q|M|R|H 32 Partitions
1|1|1|1|1|1|1|1|1|1|1|1|1|1|1|1|1|1|1|1|1|1|1|1|1|1|1|1|1|1|1|1 with counts
3. AR BI JK LO NS IU DT BS CP BN DJ LT GM GH MQ HR Merge #1
11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 11 and tally
4. ABRI JKLO INSU BDST BCNP DJLT GHM HMQR Merge #2
1111 1111 1111 1111 1111 1111 211 1111 and tally
5. ABIJKLOR BDINSTU BCDJLNPT GHMQR Merge #3
11111111 1111211 11111111 22211 and tally
6. ABDIJKLNORSTU BCDGHJLMNPQRT Merge #4
1212111111211 1112211211111 and tally
7. ABCDGHIJKLMNOPQRSTU Merge #5
1322111312132113121 and tally
So, if we read the final list in memory from start to finish, it yields the sorted list:
A|B|C|D|G|H|I|J|K|L|M|N|O|P|Q|R|S|T|U
-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-
1|3|2|2|1|1|1|3|1|2|1|3|2|1|1|3|1|2|1 = 32 Name instances (== original file).
Why the Solution is Efficient
Whether a hash table is used (as the original poster suggested), and whether multi-threading is used or not, any solution to this question cannot be solved more efficiently than O(n log n) because a sort must be performed. Given this restriction, there are two strategies that can be employed:
Read data from disk, use hash table to manage name/frequency totals, then sort the hash table contents (original poster's suggested method)
Read data from disk, initialise each name with its frequency total from the file, then merge sort the names simultaneously summing all the totals for each name (this solution).
Solution (1) requires the hash table to be sorted after all data has been read in. Solution (2) performs its frequency tallying as it is sorting, thus the overhead of the hash table has been removed. Without considering multithreading at all, we can already see that even with the most efficient hash table implementation for Solution (1), Solution (2) is already more efficient as it doesn't have the overhead of the hash table at all.
Constraints on Multithreading
In both Solution (1) and Solution (2), assuming the most efficient hash table implementation ever devised is being used for Solution (1), both algorithms perform the same asymptotically in O(n log n); it's simply that the ordering of their operations is slightly different. However, while multithreading Solution (1) actually slows its execution down, multithreading Solution (2) will gain substantial improvements in speed. How is this possible?
If we multithread Solution (1), either in the reading from disk or in the sort afterwards, we hit a problem of contention on the hash table as all threads try to access the hash table simultaneously. Especially for writing to the table, this contention could cripple the execution time of Solution (1) so much so that running it without multithreading would actually give a faster execution time.
For multithreading to give execution time speed ups, it is necessary to make sure that each block of work that each thread performs is independent of every other thread. This will allow all threads to run at maximum speed with no contention on shared resources and to complete the job much faster. Solution (2) does exactly this removing the hash table altogether and employing Merge Sort, a Divide and Conquer algorithm that allows a problem to be broken into sub-problems that are independent of each other.
Multithreading and Partitioning to Further Improve Execution Times
In order to multithread the merge sort, the file can be divided into partitions and a new thread created to merge each consecutive pair of partitions. As names in the file are variable length, the file must be scanned serially from start to finish in order to be able to do the partitioning; random access on the file cannot be used. However, as any solution must scan the file contents at least once anyway, allowing only serial access to the file still yields an optimal solution.
What kind of speed-up in execution times can be expected from multithreading Solution (2)? The analysis of this algorithm is quite tricky given its simplicity, and as been the subject of various white papers. However, splitting the file into n partitions will allow the program to execute (n / log(n)) times quicker than on a single CPU with no partitioning of the file. Simply put, if a single processor takes 1 hour to process a 640GB file, then splitting the file into 64 10GB chunks and executing on a machine with 32 CPUs will allow the program to complete in around 6 minutes, a 10 fold increase (ignoring disk overheads).