hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dick King (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-323) Improve the way job history files are managed
Date Thu, 26 Aug 2010 00:12:37 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12902719#action_12902719

Dick King commented on MAPREDUCE-323:

It is not used and I'll remove it.

Kind of a minor point ... saves about four executed lines on an error 
path that normally never happens ... but I'll make the change.

Because when we create a new subdirectory within the new runnable within 
moveToDone(JobID), the thread waits until enough time passes that that 
subdirectory will never have any entries added again, and then it writes 
the index.  That ties up a tread, so we need an additional one to move 
the mail.

Indeed it isn't.

I left it in the parameter chain because future code changes may use it. 
   In particular we might place a ceiling on how many jobs there could 
ever come to be in one subdirectory, and that would take a JobID to enforce.

Actually, I have it backwards.  We're indexing on every call, whether it 
needs it or not, which is bad.  I'll fix this.

Yeah, when I abstracted out buildIndex I didn't delete enough code from 
the inline.

I make the 5 minute checkpoints because there is a small exposure to 
some history logs not getting indexed after a job tracker crash.  This 
measure reduces the exposure.

I made the busy wait loop 30 seconds, rather than one second, and on 
every pass to reduce the load and to make this code run only as often as 
it needs to.  However, I therefore increased the thread pool size to THREE:

1 to be in the loop waiting for the hour to end,

1 to be obsolete because the hour already ended during its half minute 
but it doesn't realize it yet, and

1 to copy a history file.

That's three.  If we're in the usual case where there is only one 
instance busy-waiting, then two instances might flow into the copying 
code.  This is harmless but not useful [since the whole copying code is 
run with the lock taken].

> Improve the way job history files are managed
> ---------------------------------------------
>                 Key: MAPREDUCE-323
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-323
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: jobtracker
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Amar Kamat
>            Assignee: Dick King
>            Priority: Critical
>         Attachments: MR323--2010-08-20--1533.patch
> Today all the jobhistory files are dumped in one _job-history_ folder. This can cause
problems when there is a need to search the history folder (job-recovery etc). It would be
nice if we group all the jobs under a _user_ folder. So all the jobs for user _amar_ will
go in _history-folder/amar/_. Jobs can be categorized using various features like _jobid,
date, jobname_ etc but using _username_ will make the search much more efficient and also
will not result into namespace explosion. 

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message