hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Luke Lu (JIRA)" <j...@apache.org>
Subject [jira] Created: (MAPREDUCE-2025) Improve job counter wire protocol with incremental schema and delta values
Date Sat, 21 Aug 2010 01:33:16 GMT
Improve job counter wire protocol with incremental schema and delta values
--------------------------------------------------------------------------

                 Key: MAPREDUCE-2025
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2025
             Project: Hadoop Map/Reduce
          Issue Type: Improvement
          Components: client, jobtracker, task
    Affects Versions: 0.20.2
            Reporter: Luke Lu
            Assignee: Luke Lu
             Fix For: 0.22.0


The current job counter protocol is quite verbose (string key and vlong value pairs) especially
when keys are long. By using an incremental schema (key to key id mapping) and data (vint
key id to vlong value delta) protocol, we can significantly lower the counter overhead and
allow us to up (from the current 1 update/minute) the counter update frequency. This jira
tracks the user counter (for apps or tools (pig etc.) built on top of mr framework) improvement.
Framework (mr/hdfs etc) counter improvement is tracked by MAPREDUCE-901 (a more common case
with more optimization opportunities, as the schema is fixed.)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message