flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Radu Tudoran <radu.tudo...@huawei.com>
Subject RE: output writer
Date Thu, 30 Jul 2015 10:53:55 GMT
Re-hi,

I have double –checked and actually there is an OutputFormat interface in flink which can
be extended.
I believe that for this kind of specific formats as mentioned by Michele, each can develop
the appropriate format.
On the other hand, having more outputformats I believe is something that could be contributed.
We should identify a couple of common formats. The first one that comes in my mind is to have
something for writing to memory (e.g. memory buffer)



Dr. Radu Tudoran
Research Engineer
IT R&D Division

[cid:image007.jpg@01CD52EB.AD060EE0]
HUAWEI TECHNOLOGIES Duesseldorf GmbH
European Research Center
Riesstrasse 25, 80992 München

E-mail: radu.tudoran@huawei.com
Mobile: +49 15209084330
Telephone: +49 891588344173

HUAWEI TECHNOLOGIES Duesseldorf GmbH
Hansaallee 205, 40549 Düsseldorf, Germany, www.huawei.com<http://www.huawei.com/>
Registered Office: Düsseldorf, Register Court Düsseldorf, HRB 56063,
Managing Director: Jingwen TAO, Wanzhou MENG, Lifang CHEN
Sitz der Gesellschaft: Düsseldorf, Amtsgericht Düsseldorf, HRB 56063,
Geschäftsführer: Jingwen TAO, Wanzhou MENG, Lifang CHEN

From: Fabian Hueske [mailto:fhueske@gmail.com]
Sent: Thursday, July 30, 2015 11:34 AM
To: user@flink.apache.org
Subject: Re: output writer

Hi Michele, hi Radu
Flink does not have such an OutputFormat, but I agree, it would be a valuable addition.
Radu's approach looks like the way to go to implement this feature.
@Radu, is there a way to contribute your OutputFormat to Flink?
Cheers, Fabian

2015-07-30 10:24 GMT+02:00 Radu Tudoran <radu.tudoran@huawei.com<mailto:radu.tudoran@huawei.com>>:
Hi,

My 2 cents ... based on something similar that I have tried.
I have created an own implementation for OutputFormat where you define your own logic for
what happens in the "writerecord function". This logic would consist in making a distinction
between the ids and write each to the appropriate file

Might be that other solutions exist


Dr. Radu Tudoran
Research Engineer
IT R&D Division


HUAWEI TECHNOLOGIES Duesseldorf GmbH
European Research Center
Riesstrasse 25, 80992 München

E-mail: radu.tudoran@huawei.com<mailto:radu.tudoran@huawei.com>
Mobile: +49 15209084330<tel:%2B49%2015209084330>
Telephone: +49 891588344173<tel:%2B49%20891588344173>

HUAWEI TECHNOLOGIES Duesseldorf GmbH
Hansaallee 205, 40549 Düsseldorf, Germany, www.huawei.com<http://www.huawei.com>
Registered Office: Düsseldorf, Register Court Düsseldorf, HRB 56063,
Managing Director: Jingwen TAO, Wanzhou MENG, Lifang CHEN
Sitz der Gesellschaft: Düsseldorf, Amtsgericht Düsseldorf, HRB 56063,
Geschäftsführer: Jingwen TAO, Wanzhou MENG, Lifang CHEN

-----Original Message-----
From: Michele Bertoni [mailto:michele1.bertoni@mail.polimi.it<mailto:michele1.bertoni@mail.polimi.it>]
Sent: Thursday, July 30, 2015 10:15 AM
To: user@flink.apache.org<mailto:user@flink.apache.org>
Subject: output writer

Hi everybody,
I have a question about the writer
I have to save my dataset in different files according to a field of the tuples

let’s assume I have a groupId in the tuple, I need to store each group in a different file,
with a custom name: any idea on how i can do that?


thanks!
Michele

Mime
View raw message