hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Allen Wittenauer <awittena...@linkedin.com>
Subject Do we shoot ourselves by using all task slots?
Date Fri, 28 May 2010 18:37:41 GMT

	I've been thinking (which is always a dangerous thing) about data locality lately.  

	If we look at file systems, there is this idea of 'reserved space'.  This space is used for
a variety of reasons, including to reduce fragmentation on busy file systems.  This allows
the file system driver to make smarter decisions of block placement and helping the overall

	At LinkedIn, we're about to build a new grid with a few hundred nodes.  I'm beginning to
wonder if it wouldn't make sense to actually 'hold back' some task slots from usage with this
same concept in mind.  Let's take a grid that is full:  all of the task slots are in use.
 When a task ends, the scheduler has to make a decision as to which task gets used for any
available task slots.  If we assume a fairly FIFO view of the world (default scheduler, capacity,
maybe fair share?), it pulls the next task off the stack and pushes it to the task slot. 
If only one task slot is free, locality doesn't enter into the picture at all.  In essence,
we've fragmented our execution.

	If we were to leave even 1 slot 'always' free (and therefore sacrificing execution speed
by 1 slot), the scheduler could potentially make sure the task is host or rack local.  If
it can't, no loss--it wouldn't have been local anyway.  Obviously reserving more slots as
'always' free increases our likelihood of being local.  It just comes down to how much of
a tradeoff it is worth.

	I guess the real question comes down to how much of an impact does data locality really have.
 I know in the case of the bigger grids at Yahoo!, the ops team suspected (but never did the
homework to verify) that our grids and their usage so massive that the data locality rarely
happened, especially for "popular" data.  I can't help but wonder if the situation would have
been better if we would have kept x% (say .005%?) of the grid free based upon the speculation

View raw message