hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Schmidtke <ro.schmid...@gmail.com>
Subject Shuffle buffer size in presence of small partitions
Date Mon, 31 Jul 2017 13:36:40 GMT
Hi all,

I just ran into an issue, which likely resulted from my not very
intelligent configuration, but nonetheless I'd like to share this with the
community. This is all on Hadoop 2.7.3.

In my setup, each reducer roughly fetched 65K from each mapper's spill
file. I disabled transferTo during shuffle, because I wanted to have a look
at the file system statistics, which miss mmap calls, which is what
transferTo sometimes defaults to. I left the shuffle buffer size at 128K
(not knowing about the parameter at the time). This had the effect that I
observed roughly 100% more data being read during shuffle, since 128K were
read for each 65K needed.

I added a quick fix to Hadoop which chooses the minimum of the partition
size and the shuffle buffer size:
Benchmarking this version against transferTo.allowed=true yields the same
runtime and roughly 10% more reads in YARN during the shuffle phase
(compared to previous 100%).
Maybe this is something that should be added to Hadoop? Or do users have to
be more clever about their job configurations? I'd be happy to open a PR if
this is deemed useful.

Anyway, thanks for the attention!


My GPG Key ID: 336E2680

View raw message