flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Metzger <rmetz...@apache.org>
Subject Re: HDFS append
Date Thu, 04 Dec 2014 21:03:03 GMT

I think there is no support for appending to HDFS files in Flink yet.
HDFS supports it, but there are some adjustments in the system required
(not deleting / creating directories before writing; exposing the append()
methods in the FS abstractions).

I'm planning to work on the FS abstractions in the next week, if I have
enough time, I can also look into adding support for append().

Another approach could be adding support for recursively reading
directories with the input formats. Vasia asked for this feature a few days
ago on the mailing list. If we would have that feature, you could just
write to a directory and read the parent directory (with all the dirs for
the appends).


On Thu, Dec 4, 2014 at 5:59 PM, Flavio Pompermaier <pompermaier@okkam.it>

> Hi guys,
> how can I efficiently appends data (as plain strings or also avro records)
> to  HDFS using Flink?
> Do I need to use Flume or can I avoid it?
> Thanks in advance,
> Flavio

View raw message