Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id A1508200BE7 for ; Tue, 20 Dec 2016 16:42:01 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 9FE0B160B29; Tue, 20 Dec 2016 15:42:01 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id C15C1160B12 for ; Tue, 20 Dec 2016 16:42:00 +0100 (CET) Received: (qmail 11773 invoked by uid 500); 20 Dec 2016 15:41:59 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flink.apache.org Delivered-To: mailing list user@flink.apache.org Received: (qmail 11759 invoked by uid 99); 20 Dec 2016 15:41:59 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Dec 2016 15:41:59 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 25609180255 for ; Tue, 20 Dec 2016 15:41:59 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.38 X-Spam-Level: ** X-Spam-Status: No, score=2.38 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RCVD_IN_SORBS_SPAM=0.5, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id dBzJv7XvE4Rk for ; Tue, 20 Dec 2016 15:41:57 +0000 (UTC) Received: from mail-wm0-f42.google.com (mail-wm0-f42.google.com [74.125.82.42]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 9761B5FC61 for ; Tue, 20 Dec 2016 15:41:57 +0000 (UTC) Received: by mail-wm0-f42.google.com with SMTP id t79so133801099wmt.0 for ; Tue, 20 Dec 2016 07:41:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=+628PU5198JU3cNtTsXB4VZTdIe4glqqXolUMukLrm0=; b=ZDr6qQsu1bMZBEZHhRrzOEBO83oMDPFEhDoCB2Hgc8oOhrW/9mI9oKQ1MX/oiUM7yn RwXC7rF11PJXMEdH21reVeEEZnj4/C+lFQyQa+5nS6HZ0ZqRn3V6imYdiO4GqK/L/f/G u+LEs7xvDKIOHjXPiLeGUIRywt2gWH7b29W/v7LEqpwwCpqVf3ziQJk4Zh92GFuX5eFt JCk35ZFhPN+lcjDPucr9+yzwfNwKFZnBm+GE0ZYnzNwDG/fT0jFflllHHG/07Ogwq39X SDYuYhbMJJML6dWEBhTUA/fJYlT7XeJum9Rj3Cwsc88xOao+qFaUmql4z3YVLELXC7Yy ScmA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=+628PU5198JU3cNtTsXB4VZTdIe4glqqXolUMukLrm0=; b=P2sFewC10gvkECc3lurEnwPws64THsVG4YJUey3GfCh9eqx41MerMZea2HztvjD+7g 353f55neHSFGISstGLb6cr6MnHWgomBMXutjEVszehYbBUnQsgfq7R0j0Gb2783MfVNE Jveocafcvz0y7Oz//2x8UEYubbvKDWh1MbetuAOzlGpVB+eaS/sb8nzxhFEVYAqAex9d yqbjcca6j6AtCct+VqMBsqOahAEaHLRQoRPSiYkERldaqio3FaYhTL4PKBs5RcdBhheY YLnxtUH2kFwK1ZgGlcKz4FYQ6VO5QFdtY/s/7uiTN7JOf2ki5SSiHqOqMO7nr9Esg1cE l5/A== X-Gm-Message-State: AIkVDXIBWEOE+B7f/mUcwaVyOISVBaFM1x9NI/kWEDq6tQGrSyfrC/35hb2rHo82ggGjHEkOlzHop33rYBJqZg== X-Received: by 10.28.32.150 with SMTP id g144mr2572605wmg.46.1482248510960; Tue, 20 Dec 2016 07:41:50 -0800 (PST) MIME-Version: 1.0 Received: by 10.194.162.129 with HTTP; Tue, 20 Dec 2016 07:41:20 -0800 (PST) In-Reply-To: References: From: Fabian Hueske Date: Tue, 20 Dec 2016 16:41:20 +0100 Message-ID: Subject: Re: Generate _SUCCESS (map-reduce style) when folder has been written To: user@flink.apache.org Content-Type: multipart/alternative; boundary=001a113c8b58f6f9f1054418dfdf archived-at: Tue, 20 Dec 2016 15:42:01 -0000 --001a113c8b58f6f9f1054418dfdf Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Great to hear! Do you mean that the behavior of Flink's HadoopOutputFormat is not consistent with Hadoop's behavior? If that's the case, could you open a JIRA ticket to report this and maybe also contribute your changes back? Thanks a lot, Fabian 2016-12-20 16:37 GMT+01:00 Gwenhael Pasquiers < gwenhael.pasquiers@ericsson.com>: > Thanks, it is working properly now. > > NB : Had to delete the folder by code because Hadoop=E2=80=99s OuputForma= ts will > only overwrite file by file, not the whole folder. > > > > *From:* Fabian Hueske [mailto:fhueske@gmail.com] > *Sent:* mardi 20 d=C3=A9cembre 2016 14:21 > *To:* user@flink.apache.org > *Subject:* Re: Generate _SUCCESS (map-reduce style) when folder has been > written > > > > Hi Gwenhael, > > The _SUCCESS files were originally generated by Hadoop for successful > jobs. AFAIK, Spark leverages Hadoop's Input and OutputFormats and seems t= o > have followed this approach as well to be compatible. > > You could use Flink's HadoopOutputFormat which is a wrapper for Hadoop > OutputFormats (both mapred and mapreduce APIs). > The wrapper does also produce the _SUCCESS files. In fact, you might be > able to use exactly the same OutputFormat as your Spark job. > > Best, > > Fabian > > > > 2016-12-20 14:00 GMT+01:00 Gwenhael Pasquiers < > gwenhael.pasquiers@ericsson.com>: > > Hi, > > > > Sorry if it=E2=80=99s already been asked but is there an embedded way for= flink to > generate a _SUCCESS file in the folders it=E2=80=99s been writing into (u= sing the > write method with OutputFormat) ? > > > > We are replacing a spark job that was generating those files (and further > operations rely on it). > > > > Best regards, > > > > Gwenha=C3=ABl PASQUIERS > > > --001a113c8b58f6f9f1054418dfdf Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Great to hear!

Do you me= an that the behavior of Flink's HadoopOutputFormat is not consistent wi= th Hadoop's behavior?
If that's the case, could you open a= JIRA ticket to report this and maybe also contribute your changes back?
Thanks a lot,
Fabian

2016-12-20 16:37 GMT+01:00 Gwenhael Pasqui= ers <gwenhael.pasquiers@ericsson.com>:

Thanks, it is working properly now.<= u>

NB : Had to delete the folder by cod= e because Hadoop=E2=80=99s OuputFormats will only overwrite file by file, n= ot the whole folder.

=C2=A0

From: = Fabian Hueske [mailto:fhueske@gmail.com]
Sent: mardi 20 d=C3=A9cembre 2016 14:21
To: user@= flink.apache.org
Subject: Re: Generate _SUCCESS (map-reduce style) when folder has be= en written

=C2=A0

Hi Gwenhael,

The _SUCCESS files we= re originally generated by Hadoop for successful jobs. AFAIK, Spark leverag= es Hadoop's Input and OutputFormats and seems to have followed this app= roach as well to be compatible.

You could use Flink&#= 39;s HadoopOutputFormat which is a wrapper for Hadoop OutputFormats (both m= apred and mapreduce APIs).
The wrapper does also produce the _SUCCESS files. In fact, you might be abl= e to use exactly the same OutputFormat as your Spark job.

Best,

Fabian

=C2=A0

2016-12-20 14:00 GMT+01:00 Gwenhael Pasquiers <gwenhael= .pasquiers@ericsson.com>:

Hi,

=C2=A0

Sorry if it=E2=80=99s already b= een asked=C2=A0but is there an embedded way for flink to generate a _SUCCES= S file in the folders it=E2=80=99s been writing into (using the write metho= d with OutputFormat) ?

=C2=A0

We are replacing a spark job th= at was generating those files (and further operations rely on it).

=C2=A0

Best regards,<= /u>

=C2=A0

Gwenha=C3=ABl PASQUIERS<= u>

=C2=A0


--001a113c8b58f6f9f1054418dfdf--