hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "W.P. McNeill" <bill...@gmail.com>
Subject How frequently can I set status?
Date Thu, 23 Dec 2010 19:15:17 GMT
I have a loop that runs over a large number of iterations (order of 100,000)
very quickly.  It is nice to do context.setStatus() with an indication of
where I am in the loop.  Currently I'm only calling setStatus() every 10,000
iterations because I don't want to overwhelm the task trackers with lots of
status messages.  Is this something I should be worried, about or is Hadoop
designed to handle a high volume of status messages?  If so, I'll just call
setStatus() every iteration.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message