Mailing-List: contact common-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-dev@hadoop.apache.org
Received-SPF: neutral (nike.apache.org: 216.145.54.173 is neither permitted
 nor denied by domain of evans@yahoo-inc.com)
From: Robert Evans <evans@yahoo-inc.com>
To: "mapreduce-dev@hadoop.apache.org" <mapreduce-dev@hadoop.apache.org>,
        "common-dev@hadoop.apache.org" <common-dev@hadoop.apache.org>
CC: "cdh-dev@cloudera.org" <cdh-dev@cloudera.org>
Date: Mon, 5 Dec 2011 11:14:44 -0800
Subject: Re: Automatically Documenting Apache Hadoop Configuration
Thread-Topic: Automatically Documenting Apache Hadoop Configuration
Thread-Index: AcyzdyI22zu2IF70TAuhNDrw23CL7AACwMYD
Message-ID: <CB027444.3003A%evans@yahoo-inc.com>
In-Reply-To: <A94BAF60-6EDD-4675-8B77-28D8BE101F06@cloudera.com>
Accept-Language: en-US
Content-Language: en
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_CB0274443003Aevansyahooinccom_"
MIME-Version: 1.0

--_000_CB0274443003Aevansyahooinccom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

>From my work on yarn trying to document the configs there and to standardiz=
e them, writing anything that is going to automatically detect config value=
s through static analysis is going to be very difficult.  This is because m=
ost of the configs in yarn are now built up using static string concatenati=
on.

public static String BASE =3D "yarn.base.";
public static String CONF =3D BASE+"config";

I am not sure that there is a good way around this short of using a full ja=
va parser to trace out all method calls, and try to resolve the parameters.=
  I know this is possible, just not that simple to do.

I am +1 for anything that will clean up configs and improve the documentati=
on of them.  Even if we have to rewire or rewrite a lot of the Configuratio=
n class to make things work properly.

--Bobby Evans

On 12/5/11 11:54 AM, "Harsh J" <harsh@cloudera.com> wrote:

Praveen,

(Inline.)

On 05-Dec-2011, at 10:14 PM, Praveen Sripati wrote:

> Hi,
>
> Recently there was a query about the Hadoop framework being tolerant for
> map/reduce task failure towards the job completion. And the solution was =
to
> set the 'mapreduce.map.failures.maxpercent` and
> 'mapreduce.reduce.failures.maxpercent' properties. Although this feature
> was introduced couple of years back, it was not documented. Had similar
> experience with 0.23 release also.

I do not know if we recommend using config strings directly when there's an=
 API in Job/JobConf supporting setting the same thing. Just saying - that t=
here was javadoc already available on this. But of course, it would be bett=
er if the tutorial covered this too. Doc-patches welcome!

> It would be really good for Hadoop adoption to automatically dig and
> document all the existing configurable properties in Hadoop and also to
> identify newly added properties in a particular release during the build
> processes. Documentation would also lead to fewer queries in the forums.
> Cloudera has done something similar [1], though it's not 100% accurate, i=
t
> would definitely help to some extent.

I'm +1 for this. We do request and consistently add entries to *-default.xm=
l files if we find them undocumented today. I think we should also enforce =
it at the review level, so that patches do not go in undocumented -- at min=
imum the configuration tweaks at least.


--_000_CB0274443003Aevansyahooinccom_--