flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephan Ewen <se...@apache.org>
Subject Re: S3/S3A support
Date Tue, 11 Oct 2016 16:01:11 GMT
Hi!

The "truncate()" functionality is only needed for the rolling/bucketing
sink. The core checkpoint functionality does not need any truncate()
behavior...

Best,
Stephan


On Tue, Oct 11, 2016 at 5:22 PM, Vijay Srinivasaraghavan <
vijikarthi@yahoo.com.invalid> wrote:

> Thanks Stephan. My understanding is checkpoint uses truncate API but S3A
> does not support it. Will this have any impact?
> Some of the known S3A client limitations are captured in Hortonworks site
> https://hortonworks.github.io/hdp-aws/s3-s3aclient/index.html and
> wondering if that has any impact on Flink deployment using S3?
> RegardsVijay
>
>
>
>     On Tuesday, October 11, 2016 1:46 AM, Stephan Ewen <sewen@apache.org>
> wrote:
>
>
>  Hi!
> In 1.2-SNAPSHOT, we recently fixed issues due to the "eventual
> consistency" nature of S3. The fix is not in v1.1 - that is the only known
> issue I can think of.
> It results in occasional (seldom) periods of heavy restart retries, until
> all files are visible to all participants.
> If you run into that issue, may be worthwhile to look at Flink
> 1.2-SNAPSHOT.
> Best,
> Stephan
>
> On Tue, Oct 11, 2016 at 12:13 AM, Vijay Srinivasaraghavan
> <vijikarthi@yahoo.com.invalid> wrote:
>
> Hello,
> Per documentation (https://ci.apache.org/ projects/flink/flink-docs-
> master/setup/aws.html), it looks like S3/S3A FS implementation is supported
> using standard Hadoop S3 FS client APIs.
> In the absence of using standard HCFS and going with S3/S3A,
> 1) Are there any known limitations/issues?
> 2) Does checkpoint/savepoint works properly?
> Regards
> Vijay
>
>
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message