Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of goksron@gmail.com designates
 209.85.216.48 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=CtPJU50F5Isw3t7Lki7DN4jQAdc5aKCLS4Xkw0CZqzMwd/2klzR8htCn71iQj1X0JW
         8gVq7H0qpaIFvdATPtHTBpwNx0UxsNPCh424zC5Kgy+OnSrOMIeTOkkUksbU59F5Epjm
         AIHv1zZCMDm9jPED552v1FmRF3d/7xZ3g0uxk=
MIME-Version: 1.0
In-Reply-To: <5612E552-2214-40A2-A54C-3C452A3891F5@yahoo-inc.com>
References: <BANLkTimu+-SxrRQytLBRoeFc7E9WKC1wcA@mail.gmail.com>
	<BANLkTin6LvNb1X6S36XO7MCRY6Jqiif=yQ@mail.gmail.com>
	<5612E552-2214-40A2-A54C-3C452A3891F5@yahoo-inc.com>
Date: Sun, 17 Apr 2011 16:57:25 -0700
Message-ID: <BANLkTimb_b_WhgQS-DRMLUiG+x1nO8Hy=A@mail.gmail.com>
Subject: Re: Estimating Time required to compute M/Rjob
From: Lance Norskog <goksron@gmail.com>
To: common-user@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

ROC Convex Hull is an analysis technique for optimizing parameters for
given outputs.

For example, if a classification technique has tuning knobs, ROCCH
will find the settings that give a desired failure rate.

On Sun, Apr 17, 2011 at 12:07 PM, Matthew Foley <mattf@yahoo-inc.com> wrote=
:
> Since general M/R jobs vary over a huge (Turing problem equivalent!) rang=
e of behaviors, a more tractable problem might be to characterize the descr=
iptive parameters needed to answer the question: "If the following problem =
P runs in T0 amount of time on a certain benchmark platform B0, how long T1=
 will it take to run on a differently configured real-world platform B1 ?"
>
> Or are you only dealing with one particular M/R job? =C2=A0If so, the abo=
ve is a good way to look at it: first identify the controlling parameters, =
then analyze how they co-vary with execution time. =C2=A0Now you've reduced=
 it to a question that can be answered by a series of "make hypothesis" / "=
do experiment" steps :-) =C2=A0Pick a parameter you think is a likely candi=
date, and make a series of measurements of execution time for different val=
ues of the parameter. =C2=A0Repeat until you've fully characterized the pro=
blem space.
>
> Good luck,
> --Matt
>
> On Apr 16, 2011, at 6:39 AM, Sonal Goyal wrote:
>
> What is your MR job doing? What is the amount of data it is processing? W=
hat
> kind of a cluster do you have? Would you be able to share some details ab=
out
> what you are trying to do?
>
> If you are looking for metrics, you could look at the Terasort run ..
>
> Thanks and Regards,
> Sonal
> <https://github.com/sonalgoyal/hiho>Hadoop ETL and Data
> Integration<https://github.com/sonalgoyal/hiho>
> Nube Technologies <http://www.nubetech.co>
>
> <http://in.linkedin.com/in/sonalgoyal>
>
>
>
>
>
> On Sat, Apr 16, 2011 at 3:31 PM, real great..
> <greatness.hardness@gmail.com>wrote:
>
>> Hi,
>> As a part of my final year BE final project I want to estimate the time
>> required by a M/R job given an application and a base file system.
>> Can you folks please help me by posting some thoughts on this issue or
>> posting some links here.
>>
>> --
>> Regards,
>> R.V.
>>
>
>


--=20
Lance Norskog
goksron@gmail.com