Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
MIME-Version: 1.0
In-Reply-To: <D50BCEFC-D9C8-4E09-9B04-A0EDAE34BC95@data-artisans.com>
References: <CAAtbvNVftTo-yx8RFTuO72gWHR2=c9F6ds86MKSAnMSnen7TMw@mail.gmail.com>
 <53C1FDC4-7ACA-4A26-B9AF-72F273B9880B@data-artisans.com> <C24F8FA0-E762-42AC-8775-DA0DD30A9BE0@apache.org>
 <D50BCEFC-D9C8-4E09-9B04-A0EDAE34BC95@data-artisans.com>
From: Isuru Suriarachchi <isurues@gmail.com>
Date: Thu, 12 Oct 2017 14:57:01 -0400
Message-ID: <CAAtbvNUtGvjoX5FLPwbg4Z0BQ00MmDRJOp2Jw+E=zRMvmm9SDA@mail.gmail.com>
Subject: Re: Writing to an HDFS file from a Flink stream job
To: Piotr Nowojski <piotr@data-artisans.com>
Cc: Aljoscha Krettek <aljoscha@apache.org>, user <user@flink.apache.org>
Content-Type: multipart/alternative; boundary="94eb2c05d4740985e2055b5e1b52"
archived-at: Thu, 12 Oct 2017 18:57:07 -0000

--94eb2c05d4740985e2055b5e1b52
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Thanks for all your directions. BucketingSink worked.

Isuru

On Thu, Oct 12, 2017 at 9:05 AM, Piotr Nowojski <piotr@data-artisans.com>
wrote:

> I think the issue might be that writeAsText (TextOutputFormat) doesn=E2=
=80=99t
> flush the data anywhere (only on close, which in streaming doesn=E2=80=99=
t happen).
> You would need to use custom output format, but as Aljoscha pointed out
> BucketingSink makes more sense for streaming applications.
>
> Piotrek
>
> On 12 Oct 2017, at 14:58, Aljoscha Krettek <aljoscha@apache.org> wrote:
>
> Hi Isuru,
>
> What is the source in your job and is the job terminating at some point o=
r
> running continuously?
>
> In general, the writeAsText()/writeAsCsv() methods should not be used
> because they don't work well in an infinite streaming job that might have
> failures and recovery. I.e. what does that mean for the file, if you have
> recovery. For writing to files you would use the BucketingSink:
> https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/
> connectors/filesystem_sink.html
>
> Best,
> Aljoscha
>
> On 12. Oct 2017, at 14:55, Piotr Nowojski <piotr@data-artisans.com> wrote=
:
>
> Hi,
>
> Maybe this is an access rights issue? Could you try to create and write t=
o
> same file (same directory) in some other way (manually?), using the same
> user and the same machine as would Flink job do?
>
> Maybe there will be some hint in hdfs logs?
>
> Piotrek
>
> On 12 Oct 2017, at 00:19, Isuru Suriarachchi <isurues@gmail.com> wrote:
>
> Hi all,
>
> I'm just trying to use an HDFS file as the sink for my flink stream job. =
I
> use the following line to do so.
>
> stream.writeAsText("hdfs://hadoop-master:9000/user/isuru/foo");
>
>
> I have not set "fs.hdfs.hadoopconf" in my flink configuration as it shoul=
d
> work with the full hdfs file name according to [1].
>
> However, it doesn't work as expected. File foo is created on hdfs. But
> that file is empty. But I don't see any error logs too on Flink side. Whe=
n
> I used a normal file sink using a "file:///.." url, it works fine and
> data is there in the file.
>
> Do I need any other configuration to get his working?
>
> Thanks,
> Isuru
>
> [1] https://ci.apache.org/projects/flink/flink-docs-
> release-1.3/setup/config.html#hdfs
>
>
>
>
>

--94eb2c05d4740985e2055b5e1b52
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Thanks for all your directions.=C2=A0<span style=3D"font-s=
ize:12.8px">BucketingSink worked.</span><div><span style=3D"font-size:12.8p=
x"><br></span></div><div><span style=3D"font-size:12.8px">Isuru</span></div=
></div><div class=3D"gmail_extra"><br><div class=3D"gmail_quote">On Thu, Oc=
t 12, 2017 at 9:05 AM, Piotr Nowojski <span dir=3D"ltr">&lt;<a href=3D"mail=
to:piotr@data-artisans.com" target=3D"_blank">piotr@data-artisans.com</a>&g=
t;</span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0=
 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style=3D"word-wrap:=
break-word"><div>I think the issue might be that writeAsText (TextOutputFor=
mat) doesn=E2=80=99t flush the data anywhere (only on close, which in strea=
ming doesn=E2=80=99t happen). You would need to use custom output format, b=
ut as Aljoscha pointed out BucketingSink makes more sense for streaming app=
lications.</div><div><br></div><div>Piotrek</div><div><div class=3D"h5"><br=
><div><blockquote type=3D"cite"><div>On 12 Oct 2017, at 14:58, Aljoscha Kre=
ttek &lt;<a href=3D"mailto:aljoscha@apache.org" target=3D"_blank">aljoscha@=
apache.org</a>&gt; wrote:</div><br class=3D"m_3772568616471054594Apple-inte=
rchange-newline"><div><div style=3D"word-wrap:break-word;line-break:after-w=
hite-space">Hi Isuru,<div><br></div><div>What is the source in your job and=
 is the job terminating at some point or running continuously?</div><div><b=
r></div><div>In general, the writeAsText()/writeAsCsv() methods should not =
be used because they don&#39;t work well in an infinite streaming job that =
might have failures and recovery. I.e. what does that mean for the file, if=
 you have recovery. For writing to files you would use the BucketingSink:=
=C2=A0<a href=3D"https://ci.apache.org/projects/flink/flink-docs-release-1.=
4/dev/connectors/filesystem_sink.html" target=3D"_blank">https://ci.<wbr>ap=
ache.org/projects/flink/<wbr>flink-docs-release-1.4/dev/<wbr>connectors/fil=
esystem_sink.<wbr>html</a></div><div><br></div><div>Best,</div><div>Aljosch=
a</div><div></div><div><br><blockquote type=3D"cite"><div>On 12. Oct 2017, =
at 14:55, Piotr Nowojski &lt;<a href=3D"mailto:piotr@data-artisans.com" tar=
get=3D"_blank">piotr@data-artisans.com</a>&gt; wrote:</div><br class=3D"m_3=
772568616471054594Apple-interchange-newline"><div><div style=3D"word-wrap:b=
reak-word"><div>Hi,</div><div><br></div><div>Maybe this is an access rights=
 issue? Could you try to create and write to same file (same directory) in =
some other way (manually?), using the same user and the same machine as wou=
ld Flink job do?</div><div><br></div><div>Maybe there will be some hint in =
hdfs logs?</div><div><br></div><div>Piotrek</div><br><div><blockquote type=
=3D"cite"><div>On 12 Oct 2017, at 00:19, Isuru Suriarachchi &lt;<a href=3D"=
mailto:isurues@gmail.com" target=3D"_blank">isurues@gmail.com</a>&gt; wrote=
:</div><br class=3D"m_3772568616471054594Apple-interchange-newline"><div><d=
iv dir=3D"ltr">Hi all,<div><br></div><div>I&#39;m just trying to use an HDF=
S file as the sink for my flink stream job. I use the following line to do =
so.</div><div><br></div><div><pre style=3D"font-family:Menlo;font-size:9pt"=
>stream.writeAsText(<span style=3D"color:rgb(0,128,0);font-weight:bold">&qu=
ot;<a>hdfs://<wbr>hadoop-master:9000/user/isuru/<wbr>foo</a>&quot;</span>);=
</pre></div><div><br></div><div>I have not set &quot;fs.hdfs.hadoopconf&quo=
t; in my flink configuration as it should work with the full hdfs file name=
 according to [1].=C2=A0</div><div><br></div><div>However, it doesn&#39;t w=
ork as expected. File foo is created on hdfs. But that file is empty. But I=
 don&#39;t see any error logs too on Flink side. When I used a normal file =
sink using a &quot;<a>file:///</a>..&quot; url, it works fine and data is t=
here in the file.</div><div><br></div><div>Do I need any other configuratio=
n to get his working?</div><div><br></div><div>Thanks,</div><div>Isuru</div=
><div><br></div><div>[1]=C2=A0<a href=3D"https://ci.apache.org/projects/fli=
nk/flink-docs-release-1.3/setup/config.html#hdfs" target=3D"_blank">https:/=
/ci.apache.org/<wbr>projects/flink/flink-docs-<wbr>release-1.3/setup/config=
.html#<wbr>hdfs</a></div></div>
</div></blockquote></div><br></div></div></blockquote></div><br></div></div=
></blockquote></div><br></div></div></div></blockquote></div><br></div>

--94eb2c05d4740985e2055b5e1b52--