flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fabian Hueske <fhue...@gmail.com>
Subject Re: Generate _SUCCESS (map-reduce style) when folder has been written
Date Tue, 20 Dec 2016 15:41:20 GMT
Great to hear!

Do you mean that the behavior of Flink's HadoopOutputFormat is not
consistent with Hadoop's behavior?
If that's the case, could you open a JIRA ticket to report this and maybe
also contribute your changes back?

Thanks a lot,
Fabian

2016-12-20 16:37 GMT+01:00 Gwenhael Pasquiers <
gwenhael.pasquiers@ericsson.com>:

> Thanks, it is working properly now.
>
> NB : Had to delete the folder by code because Hadoop’s OuputFormats will
> only overwrite file by file, not the whole folder.
>
>
>
> *From:* Fabian Hueske [mailto:fhueske@gmail.com]
> *Sent:* mardi 20 décembre 2016 14:21
> *To:* user@flink.apache.org
> *Subject:* Re: Generate _SUCCESS (map-reduce style) when folder has been
> written
>
>
>
> Hi Gwenhael,
>
> The _SUCCESS files were originally generated by Hadoop for successful
> jobs. AFAIK, Spark leverages Hadoop's Input and OutputFormats and seems to
> have followed this approach as well to be compatible.
>
> You could use Flink's HadoopOutputFormat which is a wrapper for Hadoop
> OutputFormats (both mapred and mapreduce APIs).
> The wrapper does also produce the _SUCCESS files. In fact, you might be
> able to use exactly the same OutputFormat as your Spark job.
>
> Best,
>
> Fabian
>
>
>
> 2016-12-20 14:00 GMT+01:00 Gwenhael Pasquiers <
> gwenhael.pasquiers@ericsson.com>:
>
> Hi,
>
>
>
> Sorry if it’s already been asked but is there an embedded way for flink to
> generate a _SUCCESS file in the folders it’s been writing into (using the
> write method with OutputFormat) ?
>
>
>
> We are replacing a spark job that was generating those files (and further
> operations rely on it).
>
>
>
> Best regards,
>
>
>
> Gwenhaël PASQUIERS
>
>
>

Mime
View raw message