hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Nauroth (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MAPREDUCE-4885) streaming tests have multiple failures on Windows
Date Thu, 07 Mar 2013 21:34:12 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-4885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Chris Nauroth updated MAPREDUCE-4885:
-------------------------------------

    Attachment: MAPREDUCE-4885.1.patch

With this patch, all of the streaming tests pass consistently on Windows.  Note that to see
the tests pass, you'll also need the patch for MAPREDUCE-5006, which hasn't been committed
yet.

The problems were:

# The now-infamous problem of attempting to use paths rooted on test.build.data with HDFS,
which rejects paths containing ':', such as the Windows drive spec.  The patch implements
our standard work-around to allow overriding the test path to /tmp/<test name>.
# There was an assumption of Unix-style commands available for use as streaming mapper and
reducer functions.  To work around this, I introduced some cmd scripts that roughly approximate
Unix "cat" and "xargs cat".
# There was one actual bug in {{StreamJob}}.  It was attempting to pass a string file path
into the {{URI}} constructor.  On Windows, this would contain drive spec, and {{URI}} would
consider it invalid and throw an error.  The only reason we needed the {{URI}} was to pass
it in to the constructor of {{Path}}.  Fortunately, we already have the logic in the {{Path}}
constructor now to handle this case correctly cross-platform, so the simple fix is just to
call the {{Path}} constructor with the string file path directly.
# I've increased a few test timeouts.  The old timeout values were borderline in my environment,
sometimes causing the tests to fail sporadically on timeouts.  This was not a Windows-specific
problem.

I've tested this patch on Mac and Windows.

                
> streaming tests have multiple failures on Windows
> -------------------------------------------------
>
>                 Key: MAPREDUCE-4885
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4885
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: contrib/streaming, test
>    Affects Versions: 3.0.0, trunk-win
>            Reporter: Chris Nauroth
>            Assignee: Chris Nauroth
>         Attachments: MAPREDUCE-4885.1.patch
>
>
> There are multiple test failures due to "Queue configuration missing child queue names
for root".

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message