hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Dunning <tdunn...@maprtech.com>
Subject Re: How frequently can I set status?
Date Thu, 23 Dec 2010 19:41:37 GMT
It is reasonable to update counters often, but I think you are right to
limit the number status updates.

On Thu, Dec 23, 2010 at 11:15 AM, W.P. McNeill <billmcn@gmail.com> wrote:

> I have a loop that runs over a large number of iterations (order of
> 100,000)
> very quickly.  It is nice to do context.setStatus() with an indication of
> where I am in the loop.  Currently I'm only calling setStatus() every
> 10,000
> iterations because I don't want to overwhelm the task trackers with lots of
> status messages.  Is this something I should be worried, about or is Hadoop
> designed to handle a high volume of status messages?  If so, I'll just call
> setStatus() every iteration.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message