hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amar Kamat <ama...@yahoo-inc.com>
Subject Re: [jira] Updated: (MAPREDUCE-1978) [Rumen] TraceBuilder should provide recursive input folder scanning
Date Fri, 24 Sep 2010 07:53:10 GMT
Few comments
# For topology output, you can change 'output' to 'write'
# '-recursive' can be changed to
       <tr>
         <td><code>-recursive</code></td>
          <td>Recursively traverse input paths for job history logs.</td>
          <td>This options should be used to inform the TraceBuilder to recursively
scan the input paths and process all the files under under it. Note that, by default, only
the history logs that are directly under the input folder are considered for generating the
trace.
          </td>
        </tr>

# Testcase
/**
   * Test {@link TraceBuilder#processInputArgument()}.
   * @throws IOException
   */
  @Test
  public void testTraceBuilderInputArgumentProcessing() throws IOException {
     List<String> inputPaths = createInput();

    // test with recursive turned OFF
    testTraceBuilderDeaultInputArgumentProcessing(inputPaths); // these can be private unit
test cases

    // test with recursive turned ON
    testTraceBuilderRecursiveInputArgumentProcessing(inputPaths);
  }

  Also "testRecursive()" should be more detailed.

   W.r.t to the discussion we had, the data/directory creation step can be

# TraceBuilder: "List<Path> inputPaths = new LinkedList<Path>();" can be done
only after "if (inStatuses == null || inStatuses.length == 0) {" succeeds.

I will review the main code changes to TraceBuilder at your desk
Amar

On 9/23/10 11:21 AM, "Ravi Gummadi (JIRA)" <jira@apache.org> wrote:



     [ https://issues.apache.org/jira/browse/MAPREDUCE-1978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Ravi Gummadi updated MAPREDUCE-1978:
------------------------------------

    Attachment: 1978.patch

Attaching patch that adds the option "-recursive" to TraceBuilder.

With -recursive option, TraceBuilder generates trace by scanning all the job history logs
recursively under the given path.

Please review the patch and provide your comments.

> [Rumen] TraceBuilder should provide recursive input folder scanning
> -------------------------------------------------------------------
>
>                 Key: MAPREDUCE-1978
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1978
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: tools/rumen
>            Reporter: Amar Kamat
>            Assignee: Ravi Gummadi
>         Attachments: 1978.patch
>
>
> Currently, {{TraceBuilder}} assumes that the input is either jobhistory files or a folders
containing jobhistory files directly underneath the specified folder. There could be a use
cases where the input folder could contain sub-folders containing jobhistory files. Rumen
should support such input folders.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.



Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message