Mailing-List: contact mapreduce-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mapreduce-issues@hadoop.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:from:to:date:subject:thread-topic:thread-index:
	message-id:in-reply-to:accept-language:content-language:
	x-ms-has-attach:x-ms-tnef-correlator:acceptlanguage:content-type:mime-version;
	b=NYCgtUKeRVae88WopQ6UKreW6bUkGiE4EV3aRF+uDWyMNYhho/l2OpdWJ4vh4KKx
From: Amar Kamat <amarrk@yahoo-inc.com>
To: "mapreduce-issues@hadoop.apache.org" <mapreduce-issues@hadoop.apache.org>
Date: Fri, 24 Sep 2010 13:23:10 +0530
Subject: Re: [jira] Updated: (MAPREDUCE-1978) [Rumen] TraceBuilder should
 provide recursive input folder scanning
Thread-Topic: [jira] Updated: (MAPREDUCE-1978) [Rumen] TraceBuilder should
 provide recursive input folder scanning
Thread-Index: Acta447Ly1w5eAK4RDmVgn1MCyASMQA2flGU
Message-ID: <C8C256BE.16042%amarrk@yahoo-inc.com>
In-Reply-To: <32958893.359631285221094347.JavaMail.jira@thor>
Accept-Language: en-US
Content-Language: en
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_C8C256BE16042amarrkyahooinccom_"
MIME-Version: 1.0

--_000_C8C256BE16042amarrkyahooinccom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Few comments
# For topology output, you can change 'output' to 'write'
# '-recursive' can be changed to
       <tr>
         <td><code>-recursive</code></td>
          <td>Recursively traverse input paths for job history logs.</td>
          <td>This options should be used to inform the TraceBuilder to rec=
ursively scan the input paths and process all the files under under it. Not=
e that, by default, only the history logs that are directly under the input=
 folder are considered for generating the trace.
          </td>
        </tr>

# Testcase
/**
   * Test {@link TraceBuilder#processInputArgument()}.
   * @throws IOException
   */
  @Test
  public void testTraceBuilderInputArgumentProcessing() throws IOException =
{
     List<String> inputPaths =3D createInput();

    // test with recursive turned OFF
    testTraceBuilderDeaultInputArgumentProcessing(inputPaths); // these can=
 be private unit test cases

    // test with recursive turned ON
    testTraceBuilderRecursiveInputArgumentProcessing(inputPaths);
  }

  Also "testRecursive()" should be more detailed.

   W.r.t to the discussion we had, the data/directory creation step can be

# TraceBuilder: "List<Path> inputPaths =3D new LinkedList<Path>();" can be =
done only after "if (inStatuses =3D=3D null || inStatuses.length =3D=3D 0) =
{" succeeds.

I will review the main code changes to TraceBuilder at your desk
Amar

On 9/23/10 11:21 AM, "Ravi Gummadi (JIRA)" <jira@apache.org> wrote:


     [ https://issues.apache.org/jira/browse/MAPREDUCE-1978?page=3Dcom.atla=
ssian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ravi Gummadi updated MAPREDUCE-1978:
------------------------------------

    Attachment: 1978.patch

Attaching patch that adds the option "-recursive" to TraceBuilder.

With -recursive option, TraceBuilder generates trace by scanning all the jo=
b history logs recursively under the given path.

Please review the patch and provide your comments.

> [Rumen] TraceBuilder should provide recursive input folder scanning
> -------------------------------------------------------------------
>
>                 Key: MAPREDUCE-1978
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1978
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: tools/rumen
>            Reporter: Amar Kamat
>            Assignee: Ravi Gummadi
>         Attachments: 1978.patch
>
>
> Currently, {{TraceBuilder}} assumes that the input is either jobhistory f=
iles or a folders containing jobhistory files directly underneath the speci=
fied folder. There could be a use cases where the input folder could contai=
n sub-folders containing jobhistory files. Rumen should support such input =
folders.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


--_000_C8C256BE16042amarrkyahooinccom_--