hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sharad Agarwal (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-786) Jobtracker history should be written aysnchronously to the filesystem
Date Thu, 23 Jul 2009 10:31:14 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12734542#action_12734542
] 

Sharad Agarwal commented on MAPREDUCE-786:
------------------------------------------

I think we should cleanup the way history is written. Providing cleaner API to plugin the
async. Perhaps a separate JIRA for it. Something like:
{code}
public interface Event {
  public Map<String, String> getData();
}

public interface EventReader {
   public Iterator<Event> read();
   public void close();
}

public interface EventWriter {
  public void write(Event event);
  public void flush();
  public void close();
}
{code}

Async writer should be able to wrap to an EventWriter object:
{code}
class AsyncEventWriter implements EventWriter {
  AsyncEventWriter(EventWriter writer){
  }
}
{code}

> Jobtracker history should be written aysnchronously to the filesystem
> ---------------------------------------------------------------------
>
>                 Key: MAPREDUCE-786
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-786
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: jobtracker
>            Reporter: Sharad Agarwal
>            Assignee: Sharad Agarwal
>
> Jobtracker lock is held while writing the history events. This makes the jobtracker slow
on flushes, especially when history is written to HDFS. History events should be written asynchronously
to avoid this problem.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message