hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Nauroth (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MAPREDUCE-4885) streaming tests have multiple failures on Windows
Date Thu, 07 Mar 2013 21:34:12 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Chris Nauroth updated MAPREDUCE-4885:

    Attachment: MAPREDUCE-4885.1.patch

With this patch, all of the streaming tests pass consistently on Windows.  Note that to see
the tests pass, you'll also need the patch for MAPREDUCE-5006, which hasn't been committed

The problems were:

# The now-infamous problem of attempting to use paths rooted on test.build.data with HDFS,
which rejects paths containing ':', such as the Windows drive spec.  The patch implements
our standard work-around to allow overriding the test path to /tmp/<test name>.
# There was an assumption of Unix-style commands available for use as streaming mapper and
reducer functions.  To work around this, I introduced some cmd scripts that roughly approximate
Unix "cat" and "xargs cat".
# There was one actual bug in {{StreamJob}}.  It was attempting to pass a string file path
into the {{URI}} constructor.  On Windows, this would contain drive spec, and {{URI}} would
consider it invalid and throw an error.  The only reason we needed the {{URI}} was to pass
it in to the constructor of {{Path}}.  Fortunately, we already have the logic in the {{Path}}
constructor now to handle this case correctly cross-platform, so the simple fix is just to
call the {{Path}} constructor with the string file path directly.
# I've increased a few test timeouts.  The old timeout values were borderline in my environment,
sometimes causing the tests to fail sporadically on timeouts.  This was not a Windows-specific

I've tested this patch on Mac and Windows.

> streaming tests have multiple failures on Windows
> -------------------------------------------------
>                 Key: MAPREDUCE-4885
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4885
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: contrib/streaming, test
>    Affects Versions: 3.0.0, trunk-win
>            Reporter: Chris Nauroth
>            Assignee: Chris Nauroth
>         Attachments: MAPREDUCE-4885.1.patch
> There are multiple test failures due to "Queue configuration missing child queue names
for root".

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message