chukwa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jerome Boulon (JIRA)" <>
Subject [jira] Created: (CHUKWA-146) All hadoop logs should use a different RecordType
Date Fri, 17 Apr 2009 00:22:15 GMT
All hadoop logs should use a different RecordType

                 Key: CHUKWA-146
             Project: Hadoop Chukwa
          Issue Type: Improvement
            Reporter: Jerome Boulon
            Priority: Critical

All hadoop logs are using the same RecordType, so only 1 Reducer is used to process all log
files (other than DN,NN,Audit).
This cause a SKU issue at the M/R level.
So all hadoop logs should use a different RecordType.

- using the cluster information in the ChukwaRecordPartitioner will also help.
- using a predefine list of recordType/reducer association will also help by avoiding to have
2 log RecordType going to the same reducer,
the dynamic affectation ( ( hashCode() & Integer.MAX_VALUE) % numReduceTasks) could be
used at a fallback mechanism

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message