Return-Path: X-Original-To: apmail-flink-user-archive@minotaur.apache.org Delivered-To: apmail-flink-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id DF18318710 for ; Tue, 30 Jun 2015 14:51:32 +0000 (UTC) Received: (qmail 64443 invoked by uid 500); 30 Jun 2015 14:51:32 -0000 Delivered-To: apmail-flink-user-archive@flink.apache.org Received: (qmail 64373 invoked by uid 500); 30 Jun 2015 14:51:32 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flink.apache.org Delivered-To: mailing list user@flink.apache.org Received: (qmail 64363 invoked by uid 99); 30 Jun 2015 14:51:32 -0000 Received: from mail-relay.apache.org (HELO mail-relay.apache.org) (140.211.11.15) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 30 Jun 2015 14:51:32 +0000 Received: from mail-vn0-f41.google.com (mail-vn0-f41.google.com [209.85.216.41]) by mail-relay.apache.org (ASF Mail Server at mail-relay.apache.org) with ESMTPSA id 1C4961A0255 for ; Tue, 30 Jun 2015 14:51:32 +0000 (UTC) Received: by vnbg190 with SMTP id g190so1929095vnb.2 for ; Tue, 30 Jun 2015 07:51:31 -0700 (PDT) X-Gm-Message-State: ALoCoQlsMt9nIQTuuMB/orSDhZ6Ps+KXyAmQHpXsQE783SOH3eDuov/ZAwxEG0Oi5AiVzzzNjOzM X-Received: by 10.52.135.4 with SMTP id po4mr20265245vdb.78.1435675891118; Tue, 30 Jun 2015 07:51:31 -0700 (PDT) MIME-Version: 1.0 Received: by 10.31.233.194 with HTTP; Tue, 30 Jun 2015 07:51:11 -0700 (PDT) In-Reply-To: <55929DEA.3080608@informatik.hu-berlin.de> References: <559268FC.7030203@informatik.hu-berlin.de> <55926B2D.4010601@informatik.hu-berlin.de> <55929DEA.3080608@informatik.hu-berlin.de> From: Maximilian Michels Date: Tue, 30 Jun 2015 16:51:11 +0200 Message-ID: Subject: Re: writeAsCsv not writing anything on HDFS when WriteMode set to OVERWRITE To: "user@flink.apache.org" Content-Type: multipart/alternative; boundary=bcaec52d52dd80cbef0519bd5701 --bcaec52d52dd80cbef0519bd5701 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable HI Mihail, Thank you for your question. Do you have a short example that reproduces the problem? It is hard to find the cause without an error message or some example code. I wonder how your loop works without WriteMode.OVERWRITE because it should throw an exception in this case. Or do you change the file names on every write? Cheers, Max On Tue, Jun 30, 2015 at 3:47 PM, Mihail Vieru wrote: > I think my problem is related to a loop in my job. > > Before the loop, the writeAsCsv method works fine, even in overwrite mode= . > > In the loop, in the first iteration, it writes an empty folder containing > empty files to HDFS. Even though the DataSet it is supposed to write > contains elements. > > Needless to say, this doesn't occur in a local execution environment, whe= n > writing to the local file system. > > > I would appreciate any input on this. > > Best, > Mihail > > > > On 30.06.2015 12:10, Mihail Vieru wrote: > > Hi Till, > > thank you for your reply. > > I have the following code snippet: > > *intermediateGraph.getVertices().writeAsCsv(tempGraphOutputPath, "\n", > ";", WriteMode.OVERWRITE);* > > When I remove the WriteMode parameter, it works. So I can reason that the > DataSet contains data elements. > > Cheers, > Mihail > > > On 30.06.2015 12:06, Till Rohrmann wrote: > > Hi Mihail, > > have you checked that the DataSet you want to write to HDFS actually > contains data elements? You can try calling collect which retrieves the > data to your client to see what=E2=80=99s in there. > > Cheers, > Till > =E2=80=8B > > On Tue, Jun 30, 2015 at 12:01 PM, Mihail Vieru < > vieru@informatik.hu-berlin.de> wrote: > >> Hi, >> >> the writeAsCsv method is not writing anything to HDFS (version 1.2.1) >> when the WriteMode is set to OVERWRITE. >> A file is created but it's empty. And no trace of errors in the Flink or >> Hadoop logs on all nodes in the cluster. >> >> What could cause this issue? I really really need this feature.. >> >> Best, >> Mihail >> > > > > --bcaec52d52dd80cbef0519bd5701 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
HI Mihail,

Thank you for your = question. Do you have a short example that reproduces the problem? It is ha= rd to find the cause without an error message or some example code.

=
I wonder how your loop works without WriteMode.OVERWRITE because= it should throw an exception in this case. Or do you change the file names= on every write?

Cheers,
Max

On Tue, Jun 30, 2015 = at 3:47 PM, Mihail Vieru <vieru@informatik.hu-berlin.de>= ; wrote:
=20 =20 =20
I think my problem is related to a loop in my job.

Before the loop, the writeAsCsv method works fine, even in overwrite mode.

In the loop, in the first iteration, it writes an empty folder containing empty files to HDFS. Even though the DataSet it is supposed to write contains elements.

Needless to say, this doesn't occur in a local execution environment, when writing to the local file system.


I would appreciate any input on this.

Best,
Mihail



On 30.06.2015 12:10, Mihail Vieru wrote:
=20 Hi Till,

thank you for your reply.

I have the following code snippet:

intermediateGraph.getVertices().writeAsCsv(tempGraphOutputPath, "\n", ";", WriteMode.OVERWRITE);

When I remove the WriteMode parameter, it works. So I can reason that the DataSet contains data elements.

Cheers,
Mihail


On 30.06.2015 12:06, Till Rohrmann wrote:

Hi Mihail,

have you checked that the DataSet you want to write to HDFS actually contains data elements? You can try calling collect which retrieves the data to your client to see what=E2=80=99s= in there.

Cheers,
Till

=E2=80=8B

On Tue, Jun 30, 2015 at 12:01 PM, Mihail Vieru <vieru@informatik.hu-berlin.de>= ; wrote:
Hi,

the writeAsCsv method is not writing anything to HDFS (version 1.2.1) when the WriteMode is set to OVERWRITE.
A file is created but it's empty. And no trace of errors in the Flink or Hadoop logs on all nodes in the cluster.

What could cause this issue? I really really need this feature..

Best,
Mihail




--bcaec52d52dd80cbef0519bd5701--