Apache Flink: Why do reduce or groupReduce transformations not operate in parallel? - apache-flink

Question

For example:
DataSet<Tuple1<Long>> input = env.fromElements(1,2,3,4,5,6,7,8,9);
DataSet<Tuple1<Long>> sum = input.reduce(new ReduceFunction()<Tuple1<Long>,Tuple1<Long>>{
public Tuple1<Long> reduce(Tuple1<Long> value1,Tuple1<Long> value2){
return new Tuple1<>(value1.f0 + value2.f0);
}
}
If the above reduce transform is not a parallel operation, do I need to use additional two transformation 'partitionByHash' and 'mapPartition' as below:
DataSet<Tuple1<Long>> input = env.fromElements(1,2,3,4,5,6,7,8,9);
DataSet<Tuple1<Long>> sum = input.partitionByHash(0).mapPartition(new MapPartitionFunction()<Tuple1<Long>,Tuple1<Long>>{
public void map(Iterable<Tuple1<Long>> values,Collector<Tuple1<Long>> out){
long sum = getSum(values);
out.collect(new Tuple1(sum));
}
}).reduce(new ReduceFunction()<Tuple1<Long>,Tuple1<Long>>{
public Tuple1<Long> reduce(Tuple1<Long> value1,Tuple1<Long> value2){
return new Tuple1<>(value1.f0 + value2.f0);
}
}
and why the result of reduce transform is still an instance of DataSet but not an instance of Tuple1<Long>

Answer 1

Both, reduce and reduceGroup are group-wise operations and are applied on groups of records. If you do not specify a grouping key using groupBy, all records of the data set belong to the same group. Therefore, there is only a single group and the final result of reduce and reduceGroup cannot be computed in parallel.
If the reduce transformation is combinable (which is true for any ReduceFunction and all combinable GroupReduceFunctions), Flink can apply combiners in parallel.

Answer 2

Two answers, to your two questions:
(1) Why is reduce() not parallel
Fabian gave a good explanation. The operations are parallel if applied by key. Otherwise only pre-aggregation is parallel.
In your second example, you make it parallel by introducing a key. Instead of your complex workaround with "mapPartition()", you can also simply write (Java 8 Style)
DataSet<Tuple1<Long>> input = ...;
input.groupBy(0).reduce( (a, b) -> new Tuple1<>(a.f0 + b.f0);
Note however that your input data is so small that there will be only one parallel task anyways. You can see parallel pre-aggregation if you use larger input, such as:
ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
env.setParallelism(10);
DataSet<Long> input = env.generateSequence(1, 100000000);
DataSet<Long> sum = input.reduce ( (a, b) -> a + b );
(2) Why is the result of a reduce() operation still a DataSet ?
A DataSet is still a lazy representation of X in the cluster. You can continue to use that data in the parallel program without triggering some computation and fetching the result data back from the distributed workers to the driver program. That allows you to write larger programs that run entirely on the distributed workers and are lazily executed. No data ever fetched to the client and re-distributed to the parallel workers.
Especially in iterative programs, this is very powerful, as the entire loops work without ever involving the client and needing to re-deploy operators.
You can always get the "X" by calling "dataSet.collext().get(0);" - which makes it explicit that something should be executed and fetched.