How to set presto.s3.xxx properties when running flink from an IDE? - apache-flink

Question

I'm able to successfully run my flink job which saves to S3 using ./bin/flink run ....
For this to work, I had to copy the flink-s3-fs-presto jar to my $FLINK_HOME/lib folder and I also had to configure my S3 connection details in my flink-conf.yaml:
You need to configure both s3.access-key and s3.secret-key in Flink’s
flink-conf.yaml:
s3.access-key: your-access-key
s3.secret-key: your-secret-key
Source: flink aws docs
I also had to set a property s3.endpoint because I'm using S3 from IBM Cloud.
This all works fine when I run using ./bin/flink run.
However, when I try to run my job from my IDE (IntelliJ), I get the following error:
org.apache.flink.runtime.client.JobExecutionException: Cannot initialize task 'DataSink (TextOutputFormat (s3://xxxx/folder) - UTF-8)': Unable to load credentials from service endpoint
I've set an environment variable in the IDE run job, FLINK_CONF_DIR to point to my flink-conf.yaml and I can see that my configuration properties are picked up:
11:04:39,487 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: s3.access-key, ****
11:04:39,487 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: s3.secret-key, ****
11:04:39,487 INFO org.apache.flink.configuration.GlobalConfiguration - Loading configuration property: s3.endpoint, s3-api.us-geo.objectstorage.softlayer.net
However, I'm getting an error that suggests these properties are not getting passed to the presto library when I run from my IDE:
Caused by: org.apache.flink.fs.s3presto.shaded.com.amazonaws.SdkClientException: Unable to load credentials from service endpoint
Also, to validate this theory, if I step through the code when running from the IDE, I can see that my endpoint property is not applied:
... and drilling into the Hadoop configuration, I can see that the flink configuration is an empty map:
Digging a bit deeper, I can see that the org.apache.flink.core.fs.FileSystem#getUnguardedFileSystem() is creating a new empty Configuration:
// this "default" initialization makes sure that the FileSystem class works
// even when not configured with an explicit Flink configuration, like on
// JobManager or TaskManager setup
if (FS_FACTORIES.isEmpty()) {
initialize(new Configuration());
}
How can I configure the s3.access-key, s3.secret-key and s3.endpoint properties when running from an IDE?

Answer 1

Simply call
FileSystem.initialize(GlobalConfiguration.loadConfiguration(System.getenv("FLINK_CONF_DIR")));
before
env.execute()
will solve the issue.
Remember you still have to put your secret key and access key in flink-conf.yaml.

Answer 2

create core-site.xml
<configuration>
<property>
<name>fs.s3.impl</name>
<value>org.apache.hadoop.fs.s3a.S3AFileSystem</value>
</property>
<property>
<name>fs.s3a.access.key</name>
<value>xxxx</value>
</property>
<property>
<name>fs.s3a.secret.key</name>
<value>xxxxx</value>
</property>
</configuration>
and add
Map par = new HashMap();par.put("fs.hdfs.hadoopconf", "path to core-site.xml";
ParameterTool pt = ParameterTool.fromMap(par);
env.getConfig().setGlobalJobParameters(pt);