chukwa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jerome Boulon (JIRA)" <j...@apache.org>
Subject [jira] Created: (CHUKWA-146) All hadoop logs should use a different RecordType
Date Fri, 17 Apr 2009 00:22:15 GMT
All hadoop logs should use a different RecordType
-------------------------------------------------

                 Key: CHUKWA-146
                 URL: https://issues.apache.org/jira/browse/CHUKWA-146
             Project: Hadoop Chukwa
          Issue Type: Improvement
            Reporter: Jerome Boulon
            Priority: Critical


All hadoop logs are using the same RecordType, so only 1 Reducer is used to process all log
files (other than DN,NN,Audit).
This cause a SKU issue at the M/R level.
So all hadoop logs should use a different RecordType.

Note: 
- using the cluster information in the ChukwaRecordPartitioner will also help.
- using a predefine list of recordType/reducer association will also help by avoiding to have
2 log RecordType going to the same reducer,
the dynamic affectation ( ( hashCode() & Integer.MAX_VALUE) % numReduceTasks) could be
used at a fallback mechanism


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message