hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amogh Vasekar <am...@yahoo-inc.com>
Subject Re: Need Suggestion: Tuning MR performance by changing parameters in Hadoop project and JVM
Date Wed, 02 Jun 2010 08:37:15 GMT
You might want to check https://issues.apache.org/jira/browse/HADOOP-4179
And  http://hadoop.apache.org/common/docs/current/vaidya.html


On 6/2/10 1:24 PM, "WANG Shicai" <Evan_65@yahoo.cn> wrote:


This message is a little long. I beg your patient.

Our team would like to tune MR performance by changing parameters in Hadoop project and JVM
according to the MR Job status and result.

First, classify MR jobs into several kinds. Then monitor cpu, memory, etc. in a MR job, structing
the data from the monitor and input it into HBase. The crucial step is to build a model or
models to analysis the data. Finally, acquire the proposal for tuning MR jobs, such as increase
the memory for the job or reduce it, etc.

However, I am a developer in HBase subproject and not so acquainted with MR jobs. I need some
suggestion about the following aspects:
* Is this plan feasible or not? why?
* Is there any one or team doing the above before?
* Which processes in a MR job we ought to monitor more carefully?
* Which parameters in that processes we ought to care?
* What can we refer for the model building?
* Also, any other suggestion about our plan will be welcome.
Thank you a lot!!!



View raw message