flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fabian Hueske <fhue...@apache.org>
Subject Re: WriteAsText bug or bad name?
Date Wed, 29 Oct 2014 21:34:05 GMT
You can set the DOP of the data sink to 1 [1].
There is also a config parameter whether to create a directory or not in
case of DOP=1. If I remember correctly, the default is to NOT create
a folder for DOP=1.

[1]
http://flink.incubator.apache.org/docs/0.7-incubating/programming_guide.html#parallel-execution

Best, Fabian

2014-10-29 22:22 GMT+01:00 Flavio Pompermaier <pompermaier@okkam.it>:

> Would it be that difficult to change the behaviour for file:/// and create
> a single file?or is there a way to do that?
> On Oct 29, 2014 9:52 PM, "Márton Balassi" <balassi.marton@gmail.com>
> wrote:
>
>> Dear Flavio,
>>
>> Yes, the writeAsText() merthod really creates a folder which contains a
>> file for each execution thread, so your threads do not block each other and
>> the execution can use multiple cores on your machine. You can see similar
>> results if you try it with env.execute() from an IDE.
>>
>> There are filesystems, HDFS to mention the most prominent one which can
>> transparently treat such folder structure as a single file and then it
>> would behave as you expect. I hope this answers your question.
>>
>> Best,
>>
>> Marton
>>
>> On Wed, Oct 29, 2014 at 8:31 PM, Flavio Pompermaier <pompermaier@okkam.it
>> > wrote:
>>
>>> Hi to all,
>>> running the example at
>>> http://flink.incubator.apache.org/docs/0.7-incubating/local_execution.html
>>> I was thinking that the writeAsText on a local file was creating a text
>>> file on my local filesystem..instead it creates something similar to a
>>> sequence file (within a folder).
>>> This is something misleading I think...or the API name is wrong or this
>>> is a bug (IMHO).
>>> Btw..how can I modify the following program to write results in a single
>>> text file on my local filesystem?
>>>
>>> public static void main(String[] args) throws Exception {
>>>  ExecutionEnvironment env = ExecutionEnvironment.createLocalEnvironment();
>>>  DataSet<String> data = env.readTextFile("file:///tmp/res.txt");
>>>  data.filter(new FilterFunction<String>() {
>>>    public boolean filter(String value) {
>>>     return value.startsWith("http://");
>>>    }
>>>   }).writeAsText("file:///tmp/res.txt");
>>>   env.execute();}
>>>
>>> Best,
>>> Flavio
>>>
>>>
>>>
>>

Mime
View raw message