flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Flavio Pompermaier <pomperma...@okkam.it>
Subject Re: WriteAsText bug or bad name?
Date Wed, 29 Oct 2014 21:22:28 GMT
Would it be that difficult to change the behaviour for file:/// and create
a single file?or is there a way to do that?
On Oct 29, 2014 9:52 PM, "Márton Balassi" <balassi.marton@gmail.com> wrote:

> Dear Flavio,
>
> Yes, the writeAsText() merthod really creates a folder which contains a
> file for each execution thread, so your threads do not block each other and
> the execution can use multiple cores on your machine. You can see similar
> results if you try it with env.execute() from an IDE.
>
> There are filesystems, HDFS to mention the most prominent one which can
> transparently treat such folder structure as a single file and then it
> would behave as you expect. I hope this answers your question.
>
> Best,
>
> Marton
>
> On Wed, Oct 29, 2014 at 8:31 PM, Flavio Pompermaier <pompermaier@okkam.it>
> wrote:
>
>> Hi to all,
>> running the example at
>> http://flink.incubator.apache.org/docs/0.7-incubating/local_execution.html
>> I was thinking that the writeAsText on a local file was creating a text
>> file on my local filesystem..instead it creates something similar to a
>> sequence file (within a folder).
>> This is something misleading I think...or the API name is wrong or this
>> is a bug (IMHO).
>> Btw..how can I modify the following program to write results in a single
>> text file on my local filesystem?
>>
>> public static void main(String[] args) throws Exception {
>>  ExecutionEnvironment env = ExecutionEnvironment.createLocalEnvironment();
>>  DataSet<String> data = env.readTextFile("file:///tmp/res.txt");
>>  data.filter(new FilterFunction<String>() {
>>    public boolean filter(String value) {
>>     return value.startsWith("http://");
>>    }
>>   }).writeAsText("file:///tmp/res.txt");
>>   env.execute();}
>>
>> Best,
>> Flavio
>>
>>
>>
>

Mime
View raw message