flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Metzger <rmetz...@apache.org>
Subject Re: HDFS append
Date Tue, 09 Dec 2014 13:03:19 GMT
It seems that Vasia started working on adding support for recursive
reading: https://issues.apache.org/jira/browse/FLINK-1307.
I'm still occupied with refactoring the YARN client, the HDFS refactoring
is next on my list.

On Tue, Dec 9, 2014 at 11:59 AM, Flavio Pompermaier <pompermaier@okkam.it>
wrote:

> Any news about this Robert?
>
> Thanks in advance,
> Flavio
>
> On Thu, Dec 4, 2014 at 10:03 PM, Robert Metzger <rmetzger@apache.org>
> wrote:
>
>> Hi,
>>
>> I think there is no support for appending to HDFS files in Flink yet.
>> HDFS supports it, but there are some adjustments in the system required
>> (not deleting / creating directories before writing; exposing the append()
>> methods in the FS abstractions).
>>
>> I'm planning to work on the FS abstractions in the next week, if I have
>> enough time, I can also look into adding support for append().
>>
>> Another approach could be adding support for recursively reading
>> directories with the input formats. Vasia asked for this feature a few days
>> ago on the mailing list. If we would have that feature, you could just
>> write to a directory and read the parent directory (with all the dirs for
>> the appends).
>>
>> Best,
>> Robert
>>
>> On Thu, Dec 4, 2014 at 5:59 PM, Flavio Pompermaier <pompermaier@okkam.it>
>> wrote:
>>
>>> Hi guys,
>>> how can I efficiently appends data (as plain strings or also avro
>>> records) to  HDFS using Flink?
>>> Do I need to use Flume or can I avoid it?
>>>
>>> Thanks in advance,
>>> Flavio
>>>
>>>
>>
>

Mime
View raw message