flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fabian Hueske <fhue...@apache.org>
Subject Re: Cluster execution of an example program ("Word count") and a problem related to the modificated example
Date Mon, 07 Jul 2014 10:29:29 GMT
I thought about why it had worked for small input files.

Obviously, all input split have been read from the (local) worker that is
on the same machine as the master (otherwise the remote worker would have
raised a FileNotFoundException). Since the input splits were very small,
reading the first input split might have been faster than the request of
the remote worker to the JobManager to assign an input split. Therefore,
the local worker could requested the next input split before the remote
worker and read the full file.

Once the input was larger, reading the first split took longer than the
request of the remote worker which got a split assigned and failed to read
the file.

2014-07-07 12:16 GMT+02:00 Stephan Ewen <sewen@apache.org>:

> Hi!
> Okay, good to hear you solved the issue. HDFS is a good way to go in large
> setups, though shared filesystems / SANs that are mounted on all machines
> work as well (using Amazon EBS is an example for that).
> Stephan

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message