flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Till Rohrmann <trohrm...@apache.org>
Subject Re: StreamingFileSink cannot get AWS S3 credentials
Date Wed, 16 Jan 2019 10:42:48 GMT
I haven't configured this myself but I would guess that you need to set the
parameters defined here under S3A Authentication methods [1]. If the
environment variables don't work, then I would try to set the
authentication properties.

[1]
https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html#S3A

Cheers,
Till

On Wed, Jan 16, 2019 at 11:09 AM Vinay Patil <vinay18.patil@gmail.com>
wrote:

> Hi Till,
>
> Can you please let us know the configurations that we need to set for
> Profile based credential provider in flink-conf.yaml
>
> Exporting AWS_PROFILE property on EMR did not work.
>
> Regards,
> Vinay Patil
>
>
> On Wed, Jan 16, 2019 at 3:05 PM Till Rohrmann <trohrmann@apache.org>
> wrote:
>
>> The old BucketingSink was using Hadoop's S3 filesystem directly whereas
>> the new StreamingFileSink uses Flink's own FileSystem which need to be
>> configured via the flink-conf.yaml.
>>
>> Cheers,
>> Till
>>
>> On Wed, Jan 16, 2019 at 10:31 AM Vinay Patil <vinay18.patil@gmail.com>
>> wrote:
>>
>>> Hi Till,
>>>
>>> We are not providing `fs.s3a.access.key: access_key`,
>>> `fs.s3a.secret.key: secret_key` in flink-conf.yaml as we are using Profile
>>> based credentials provider. The older BucketingSink code is able to get the
>>> credentials and write to S3. We are facing this issue only with
>>> StreamingFileSink. We tried adding fs.s3a.impl to core-site.xml when the
>>> default configurations were not working.
>>>
>>> Regards,
>>> Vinay Patil
>>>
>>>
>>> On Wed, Jan 16, 2019 at 2:55 PM Till Rohrmann <trohrmann@apache.org>
>>> wrote:
>>>
>>>> Hi Vinay,
>>>>
>>>> Flink's file systems are self contained and won't respect the
>>>> core-site.xml if I'm not mistaken. Instead you have to set the credentials
>>>> in the flink configuration flink-conf.yaml via `fs.s3a.access.key:
>>>> access_key`, `fs.s3a.secret.key: secret_key` and so on [1]. Have you tried
>>>> this out?
>>>>
>>>> This has been fixed with Flink 1.6.2 and 1.7.0 [2].
>>>>
>>>> [1]
>>>> https://ci.apache.org/projects/flink/flink-docs-stable/ops/filesystems.html#built-in-file-systems
>>>> [2] https://issues.apache.org/jira/browse/FLINK-10383
>>>>
>>>> Cheers,
>>>> Till
>>>>
>>>> On Wed, Jan 16, 2019 at 10:10 AM Kostas Kloudas <
>>>> k.kloudas@da-platform.com> wrote:
>>>>
>>>>> Hi Taher,
>>>>>
>>>>> So you are using the same configuration files and everything and the
>>>>> only thing you change is the "s3://" to "s3a://" and the sink cannot
find
>>>>> the credentials?
>>>>> Could you please provide the logs of the Task Managers?
>>>>>
>>>>> Cheers,
>>>>> Kostas
>>>>>
>>>>> On Wed, Jan 16, 2019 at 9:13 AM Dawid Wysakowicz <
>>>>> dwysakowicz@apache.org> wrote:
>>>>>
>>>>>> Forgot to cc ;)
>>>>>> On 16/01/2019 08:51, Vinay Patil wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Can someone please help on this issue. We have even tried to set
>>>>>> fs.s3a.impl in core-site.xml, still its not working.
>>>>>>
>>>>>> Regards,
>>>>>> Vinay Patil
>>>>>>
>>>>>>
>>>>>> On Fri, Jan 11, 2019 at 5:03 PM Taher Koitawala [via Apache Flink
>>>>>> User Mailing List archive.] <ml+s2336050n25464h36@n4.nabble.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi All,
>>>>>>>          We have implemented S3 sink in the following way:
>>>>>>>
>>>>>>> StreamingFileSink sink= StreamingFileSink.forBulkFormat(new
>>>>>>> Path("s3a://mybucket/myfolder/output/"),
>>>>>>> ParquetAvroWriters.forGenericRecord(schema))
>>>>>>> .withBucketCheckInterval(50l).withBucketAssigner(new
>>>>>>> CustomBucketAssigner()).build();
>>>>>>>
>>>>>>> The problem we are facing is that StreamingFileSink is initializing
>>>>>>> S3AFileSystem class to write to s3 and is not able to find the
s3
>>>>>>> credentials to write data, However other flink application on
the
>>>>>>> same cluster use "s3://" paths are able to write data to the
same s3 bucket
>>>>>>> and folders, we are only facing this issue with StreamingFileSink.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Taher Koitawala
>>>>>>> GS Lab Pune
>>>>>>> +91 8407979163
>>>>>>>
>>>>>>>
>>>>>>> ------------------------------
>>>>>>> If you reply to this email, your message will be added to the
>>>>>>> discussion below:
>>>>>>>
>>>>>>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/StreamingFileSink-cannot-get-AWS-S3-credentials-tp25464.html
>>>>>>> To start a new topic under Apache Flink User Mailing List archive.,
>>>>>>> email ml+s2336050n1h83@n4.nabble.com
>>>>>>> To unsubscribe from Apache Flink User Mailing List archive.,
click
>>>>>>> here
>>>>>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=dmluYXkxOC5wYXRpbEBnbWFpbC5jb218MXwxODExMDE2NjAx>
>>>>>>> .
>>>>>>> NAML
>>>>>>> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>>>>>>>
>>>>>>

Mime
View raw message