flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Metzger <rmetz...@apache.org>
Subject Re: AWS Client Builder with default credentials
Date Fri, 21 Feb 2020 10:58:00 GMT
There are multiple ways of passing configuration parameters to your user
defined code in Flink

a)  use getRuntimeContext().getUserCodeClassLoader().getResource() to load
a config file from your user code jar or the classpath.
b)  use getRuntimeContext().getExecutionConfig().getGlobalJobParameters() to
access a configuration object serialized from the main method.
you can pass a custom object to the job parameters, or use Flink's
"Configuration" object in your main method:

final StreamExecutionEnvironment env =

Configuration config = new Configuration();
config.setString("foo", "bar");

c) Load the flink-conf.yaml:

Configuration conf = GlobalConfiguration.loadConfiguration();

I'm not 100% sure if this approach works, as it is not intended to be used
in user code (I believe).

Let me know if this helps!


On Thu, Feb 20, 2020 at 1:50 PM Chesnay Schepler <chesnay@apache.org> wrote:

> First things first, we do not intend for users to use anything in the S3
> filesystem modules except the filesystems itself,
> meaning that you're somewhat treading on unsupported ground here.
> Nevertheless, the S3 modules contain a large variety of AWS-provided
> CerentialsProvider implementations,
> that can derive credentials from environment variables, system properties,
> files on the classpath and many more.
> Ultimately though, you're kind of asking us how to use AWS APIs, for which
> I would direct you to the AWS documentation.
> On 20/02/2020 13:16, David Magalhães wrote:
> I'm using
> org.apache.flink.fs.s3base.shaded.com.amazonaws.client.builder.AwsClientBuilder
> to create a S3 client to copy objects and delete object inside
> a TwoPhaseCommitSinkFunction.
> Shouldn't be another way to set up configurations without put them
> hardcoded ? Something like core-site.xml or flink-conf.yaml ?
> Right now I need to have them hardcoded like this.
> AmazonS3ClientBuilder.standard
>       .withPathStyleAccessEnabled(true)
>       .withEndpointConfiguration(
>         new EndpointConfiguration("http://minio:9000", "us-east-1")
>       )
>       .withCredentials(
>         new AWSStaticCredentialsProvider(new BasicAWSCredentials("minio",
> "minio123"))
>       )
>       .build
> Thanks

View raw message