Flink How to Write DataSet As Parquet files in S3? - apache-flink

Question

How to write DataSet as Parquet files in s3 bucket using Flink. Is there any direct function like spark : DF.write.parquet("write in parquet")
Please help me how to write flink Dataset in parquet format.
I am stuck when trying to convert my DataSet to (Void,GenericRecord)
DataSet<Tuple2<Void,GenericRecord>> df = allEvents.flatMap(new FlatMapFunction<Tuple2<LongWritable, Text>, Tuple2<Void, GenericRecord>>() {
#Override
public void flatMap(Tuple2<LongWritable, Text> longWritableTextTuple2, Collector<Tuple2<Void, GenericRecord>> collector) throws Exception {
JsonAvroConverter converter = new JsonAvroConverter();
Schema schema = new Schema.Parser().parse(new File("test.avsc"));
try {
GenericRecord record = converter.convertToGenericDataRecord(longWritableTextTuple2.f1.toString().getBytes(), schema);
collector.collect( new Tuple2<Void,GenericRecord>(null,record));
}
catch (Exception e) {
System.out.println("error in converting to avro")
}
}
});
Job job = Job.getInstance();
HadoopOutputFormat parquetFormat = new HadoopOutputFormat<Void, GenericRecord>(new AvroParquetOutputFormat(), job);
FileOutputFormat.setOutputPath(job, new Path(outputPath));
df.output(parquetFormat);
env.execute();
Please help me with what I am doing wrong. I am getting Exception and this
code is not working.

Answer 1

It's a little more complicated than that with Spark. The only way I was able to read and write Parquet data in Flink is through Hadoop & MapReduce compatibility. You need hadoop-mapreduce-client-core and flink-hadoop-compatibility in Your dependencies.
Then You need to create a proper HadoopOutoutFormat. You need to do something like this:
val job = Job.getInstance()
val hadoopOutFormat = new hadoop.mapreduce.HadoopOutputFormat[Void, SomeType](new AvroParquetOutputFormat(), job)
FileOutputFormat.setOutputPath(job, [somePath])
And then You can do:
dataStream.writeUsingOutputFormat(hadoopOutFormat)

Answer 2

You didn't say which exception you are getting but here is a complete example on how to achieve this.
The main points are:
Use org.apache.flink.api.java.hadoop.mapreduce.HadoopOutputFormat
From dependency org.apache.flink:flink-hadoop-compatibility_2.11:1.11.0
HadoopOutputFormat is an adapter that allows you to use output formats developed for Hadoop
You need a DataSet<Tuple2<Void,IndexedRecord>>, because hadoop's OutputFormat<K,V> works with key-value pairs, the key we are not interested in so we use Void for the key type, and the value needs to be an Avro's IndexedRecord or GenericRecord.
Use org.apache.parquet.avro.AvroParquetOutputFormat<IndexedRecord>
From dependency org.apache.parquet:parquet-avro:1.11.1
This hadoop's OutputFormat produces Parquet
This inherits from org.apache.parquet.hadoop.FileOutputFormat<Void, IndexedRecord>
Create your own subclass of IndexedRecord
You can't use new GenericData.Record(schema) because a record like this is no serializable java.io.NotSerializableException: org.apache.avro.Schema$Field is not serializable and Flink requires it to be serializable.
You still need to provide a getSchema() method, but you can either return null or return a Schema that you hold in a static member (so that it doesn't need to be serialized, and you avoid the java.io.NotSerializableException: org.apache.avro.Schema$Field is not serializable)
The source code
import org.apache.avro.Schema;
import org.apache.avro.generic.IndexedRecord;
import org.apache.commons.lang3.NotImplementedException;
import org.apache.flink.api.common.functions.MapFunction;
import org.apache.flink.api.java.DataSet;
import org.apache.flink.api.java.ExecutionEnvironment;
import org.apache.flink.api.java.hadoop.mapreduce.HadoopOutputFormat;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.configuration.Configuration;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.parquet.avro.AvroParquetOutputFormat;
import org.apache.parquet.hadoop.metadata.CompressionCodecName;
import java.io.IOException;
import java.io.Serializable;
import java.util.ArrayList;
import java.util.stream.Collectors;
import java.util.stream.IntStream;
import java.util.stream.Stream;
public class MyParquetTest implements Serializable {
public static void main(String[] args) throws Exception {
new MyParquetTest().start();
}
private void start() throws Exception {
final ExecutionEnvironment env = ExecutionEnvironment.createLocalEnvironment();
Configuration parameters = new Configuration();
Stream<String> stringStream = IntStream.range(1, 100).mapToObj(n -> String.format("Entry %d", n));
DataSet<String> text = env.fromCollection(stringStream.collect(Collectors.toCollection(ArrayList::new)));
Job job = Job.getInstance();
HadoopOutputFormat<Void, IndexedRecord> hadoopOutputFormat = new HadoopOutputFormat<>(new AvroParquetOutputFormat<IndexedRecord>(), job);
FileOutputFormat.setCompressOutput(job, true);
FileOutputFormat.setOutputCompressorClass(job, CompressionCodecName.SNAPPY.getHadoopCompressionCodecClass());
FileOutputFormat.setOutputPath(job, new org.apache.hadoop.fs.Path("./my-parquet"));
final Schema schema = new Schema.Parser().parse(MyRecord.class.getClassLoader().getResourceAsStream("schema.avsc"));
AvroParquetOutputFormat.setSchema(job, schema);
DataSet<Tuple2<Void, IndexedRecord>> text2 = text.map(new MapFunction<String, Tuple2<Void, IndexedRecord>>() {
#Override
public Tuple2<Void, IndexedRecord> map(String value) throws Exception {
return Tuple2.of(null, new MyRecord(value));
// IndexedRecord record = new GenericData.Record(schema); // won't work becuase Schema$Field is not serializable
// record.put(0, value);
// return Tuple2.of(null, record);
}
});
text2.output(hadoopOutputFormat);
env.execute("Flink Batch Java API Skeleton");
}
public static class MyRecord implements IndexedRecord {
private static Schema schema;
static {
try {
schema = new Schema.Parser().parse(MyRecord.class.getClassLoader().getResourceAsStream("schema.avsc"));
} catch (IOException e) {
e.printStackTrace();
}
}
private final String value;
public MyRecord(String value) {
this.value= value;
}
#Override
public void put(int i, Object v) {
throw new NotImplementedException("You can't update this IndexedRecord");
}
#Override
public Object get(int i) {
return this.value;
}
#Override
public Schema getSchema() {
return schema; // or just return null and remove the schema member
}
}
}
The schema.avsc is simply
{
"name": "aa",
"type": "record",
"fields": [
{"name": "value", "type": "string"}
]
}
and the dependencies:
implementation "org.apache.flink:flink-java:${flinkVersion}"
implementation "org.apache.flink:flink-avro:${flinkVersion}"
implementation "org.apache.flink:flink-streaming-java_${scalaBinaryVersion}:${flinkVersion}"
implementation "org.apache.flink:flink-hadoop-compatibility_${scalaBinaryVersion}:${flinkVersion}"
implementation "org.apache.parquet:parquet-avro:1.11.1"
implementation "org.apache.hadoop:hadoop-client:2.8.3"

Answer 3

You'll create a Flink OutputFormat via new HadoopOutputFormat(parquetOutputFormat, job), and then pass that to DataSet.output(xxx).
The job comes from...
import org.apache.hadoop.mapreduce.Job;
...
Job job = Job.getInstance();
The parquetOutputFormat is created via:
import org.apache.parquet.hadoop.ParquetOutputFormat;
...
ParquetOutputFormat<MyOutputType> parquetOutputFormat = new ParquetOutputFormat<>();
See https://javadoc.io/doc/org.apache.parquet/parquet-hadoop/1.10.1/org/apache/parquet/hadoop/ParquetOutputFormat.html