hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lance Norskog <goks...@gmail.com>
Subject Re: Estimating Time required to compute M/Rjob
Date Sun, 17 Apr 2011 23:57:25 GMT
ROC Convex Hull is an analysis technique for optimizing parameters for
given outputs.

For example, if a classification technique has tuning knobs, ROCCH
will find the settings that give a desired failure rate.

On Sun, Apr 17, 2011 at 12:07 PM, Matthew Foley <mattf@yahoo-inc.com> wrote:
> Since general M/R jobs vary over a huge (Turing problem equivalent!) range of behaviors,
a more tractable problem might be to characterize the descriptive parameters needed to answer
the question: "If the following problem P runs in T0 amount of time on a certain benchmark
platform B0, how long T1 will it take to run on a differently configured real-world platform
B1 ?"
>
> Or are you only dealing with one particular M/R job?  If so, the above is a good way
to look at it: first identify the controlling parameters, then analyze how they co-vary with
execution time.  Now you've reduced it to a question that can be answered by a series of
"make hypothesis" / "do experiment" steps :-)  Pick a parameter you think is a likely candidate,
and make a series of measurements of execution time for different values of the parameter.
 Repeat until you've fully characterized the problem space.
>
> Good luck,
> --Matt
>
> On Apr 16, 2011, at 6:39 AM, Sonal Goyal wrote:
>
> What is your MR job doing? What is the amount of data it is processing? What
> kind of a cluster do you have? Would you be able to share some details about
> what you are trying to do?
>
> If you are looking for metrics, you could look at the Terasort run ..
>
> Thanks and Regards,
> Sonal
> <https://github.com/sonalgoyal/hiho>Hadoop ETL and Data
> Integration<https://github.com/sonalgoyal/hiho>
> Nube Technologies <http://www.nubetech.co>
>
> <http://in.linkedin.com/in/sonalgoyal>
>
>
>
>
>
> On Sat, Apr 16, 2011 at 3:31 PM, real great..
> <greatness.hardness@gmail.com>wrote:
>
>> Hi,
>> As a part of my final year BE final project I want to estimate the time
>> required by a M/R job given an application and a base file system.
>> Can you folks please help me by posting some thoughts on this issue or
>> posting some links here.
>>
>> --
>> Regards,
>> R.V.
>>
>
>



-- 
Lance Norskog
goksron@gmail.com

Mime
View raw message