flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fabian Hueske <fhue...@gmail.com>
Subject Re: Issue with writeAsText() to S3 bucket
Date Fri, 25 Oct 2019 13:04:10 GMT
Hi Michael,

One reason might be that S3's file listing command is only eventually
consistent.
It might take some time until the file appears and is listed.

Best, Fabian

Am Mi., 23. Okt. 2019 um 22:41 Uhr schrieb Nguyen, Michael <
Michael.Nguyen79@t-mobile.com>:

> Hello all,
>
>
>
> I am running into issues at the moment trying to print my DataStreams to
> an S3 bucket using writeAsText(“s3://bucket/result.json”) in my Flink job.
> I used print() on the same DataStream and I see the output I am looking for
> in standard output. I first confirm that my datastream has data by looking
> at the standard output, then I cancel my Flink job. After cancelling the
> job, result.json only gets created in my S3 bucket some of the time. It
> does not always gets created, but I confirmed that I see my data in
> standard output.
>
>
>
> I understand writeAsText() should be used for debugging purposes only
> according to Flink’s documentation, but I’m just curious as to why I can’t
> get writeAsText() to always work every time I cancel my job.
>
>
>
> Thank you for your help,
>
> Michael
>

Mime
View raw message