Understanding the mapper and reducer in a HIVE database - database

Question

I am learning about big data in my class and right now we are learning about HIVE. We learned about the mappers and reducers today, but honestly it went way over my head. Could someone explain to me what the mapper and the reducer does in each step? or atleast point me to some good readings? Thanks in advance

Answer 1

Lets try to understand map-reduce flow from above diagram which i downloaded from internet.
We are going to discuss word count problem in hadoop , which is also known as hello world in Hadoop .
Word Count is a program where we find the occurrence of each word from the file.
Lets try to understand
Step 1) :
Input file : we need some data on which we will be running the word count program , to run this program on cluster , first step should be to put this file on hadoop , which can be done via various ways easiest way is to use hadoop shell commands :
like put or copyFromLocal can be used :
Step 2)
Map Reduce talk in terms of key value pair , which means mapper will get input in the form of key and value pair, they will do the required processing then they will produce intermediate result in the form of key value pair ,which would be input for reducer to further work on that and finally reducer will also write their output in key value pair.
But we know mapper execute just after main driver program ,so who is providing input to mapper in the form of key value pair,input format does this thing for you .
InputFormat is the class which does two major things:
1) Input split ( Your number of instances of mapper driven by this input split or number of mapper is driven by input split , by default your one split size is equivalent to one block if you go by default configuration, but you may change the split size as per your need .
so if you are playing with lets say 512 mb data and your block size is 64 mb so about 8 input split size will be used , 8 mapper instances would run or 8 mappers would be used )
2) Breaking the data in key value pair ( record reader is the class which does this thing at the back end)
Now what would be key and value for a mapper , that would be driven by the file input format you use , for instance for TextInputFormat which is the mostly used inputformat. it sends longWritable(equivalent to long) as a key and Text(string) as a value to mapper
Your mapper class would work on 1 split , in class you have a map function which would work on a single line at a time so as we can see from the above diagram single line would go the map function
for example it send : "Apple orange Mango" to map function
3) Mapper
In mapper we get line as an input so now we need to write our logic .
we break the line into words based on delimited so now we have single single word in one line
As we know map works on key value pair .
we can take that work as a key and value as 1
why we have taken word as key not the other way round , because next phase is
Shuffling and Sorting phase : In this phase framework will make the group based on similar keys , or all the different keys will come together during shuffling phase and they will be sorted on the basis of keys.
Now Lets again revise :
Initially we had one file which was sent to different different mapper based on input splits , then in mapper class in map function we got one line as an input ,so built our logic with respect to one line , all the lines will work in a similar way for one instance and finally all instance would be able to work parallel way like this.
Now lets say you have 10 mappers running , now in map reduce your number of reducer is always less than mapper .
so if we 10 mappers were used so most likely 2-3 reducers would be used .
Shuffling and sorting phase we have seen all the similar keys will club together .
First of all on which basis it would be decided that which mapper data will go to which reducer.
In out case 10 mappers data has to divide in 2 reducers ,so on which basis it would decide .
There is a component called Partitioner which will decide which mapper output will go to which reducer based on hash partitioning and using modulo operator on that .
so if we are using hashing so this is 100% sure that all the same keys would go to same reducer.
We don't have to bother about anything as framework has been designed to do so efficiently , but yes as it has been written in java so we do have all the flexibility to play with different components as per need like customizing key,custom partitioner ,custom comparator and so on .
4)
Reducer : Now reducer will get keys and list of its value in its input something like this
Apple,<1,1,1,1)>
Now in reducer we write logic what exactly we want to do , for our case we want to do word count so simply we have to count the values .
That was also the reason we took 1 as value initially in Map phase because simply we had to count .
Output : final output would be written to hdfs by reducer Again in key value pair.