flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Flavio Pompermaier <pomperma...@okkam.it>
Subject Re: HDFS append
Date Tue, 09 Dec 2014 10:59:21 GMT
Any news about this Robert?

Thanks in advance,

On Thu, Dec 4, 2014 at 10:03 PM, Robert Metzger <rmetzger@apache.org> wrote:

> Hi,
> I think there is no support for appending to HDFS files in Flink yet.
> HDFS supports it, but there are some adjustments in the system required
> (not deleting / creating directories before writing; exposing the append()
> methods in the FS abstractions).
> I'm planning to work on the FS abstractions in the next week, if I have
> enough time, I can also look into adding support for append().
> Another approach could be adding support for recursively reading
> directories with the input formats. Vasia asked for this feature a few days
> ago on the mailing list. If we would have that feature, you could just
> write to a directory and read the parent directory (with all the dirs for
> the appends).
> Best,
> Robert
> On Thu, Dec 4, 2014 at 5:59 PM, Flavio Pompermaier <pompermaier@okkam.it>
> wrote:
>> Hi guys,
>> how can I efficiently appends data (as plain strings or also avro
>> records) to  HDFS using Flink?
>> Do I need to use Flume or can I avoid it?
>> Thanks in advance,
>> Flavio

View raw message