hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Nauroth (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-9387) TestDFVariations fails on Windows after the merge
Date Wed, 13 Mar 2013 23:38:15 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-9387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13601817#comment-13601817
] 

Chris Nauroth commented on HADOOP-9387:
---------------------------------------

Hi Ivan,

Thanks for working on cleaning it up.  This code has tripped me up in the past.

{code}
  public String getFilesystem() throws IOException {
    if (Shell.WINDOWS) {
      this.filesystem = dirFile.getCanonicalPath().substring(0, 2);
      return this.filesystem;
    } else {
      run();
      return filesystem;
    }
  }
{code}

It appears that there is a pitfall if running on non-Windows.  If the user of this class calls
{{getFilesystem}} before calling 
{{getMount}}, then {{filesystem}} will be uninitialized.  This is because on non-Windows,
we assign to the {{filesystem}} member variable inside {{parseOutput}}, which gets called
from {{getMount}}, but not {{getFilesystem}}.  Perhaps it's safer to move all of the parsing
logic back into {{parseExecResult}}.

{code}
    switch(getOSType()) {
      case OS_TYPE_AIX:
        Long.parseLong(tokens.nextToken()); // capacity
        Long.parseLong(tokens.nextToken()); // available
        Integer.parseInt(tokens.nextToken()); // pct used
        tokens.nextToken();
        tokens.nextToken();
        this.mount = tokens.nextToken();
        break;

      case OS_TYPE_WIN:
      case OS_TYPE_SOLARIS:
      case OS_TYPE_MAC:
      case OS_TYPE_UNIX:
      default:
        Long.parseLong(tokens.nextToken()); // capacity
        Long.parseLong(tokens.nextToken()); // used
        Long.parseLong(tokens.nextToken()); // available
        Integer.parseInt(tokens.nextToken()); // pct used
        this.mount = tokens.nextToken();
        break;
   }
{code}

The patch removes the special handling for AIX, so I think this would cause a regression if
running on that platform.  I've never used AIX, but what I infer from the old code is that
the output of df -k on Linux places mount in column 6, whereas on AIX it goes in column 7.
 Therefore, we need an extra call to {{StringTokenizer#nextToken}} if running on AIX.  Unfortunately,
I don't have access to an AIX machine to confirm.  Perhaps this code could be simplified to
always look at the last column without platform checks and special cases, i.e.:

{code}
String[] columns = line.split();
this.mount = columns[columns.length - 1];
{code}

However, that would be dependent on AIX printing mount in the last column, and without an
AIX machine, I can't confirm.

I did confirm that the patch works for Windows, so maybe the simplest path forward is to prepare
a smaller patch just for fixing Windows, and then file a follow-up jira for future refactoring
work.  That would give more time to track down someone with access to AIX to help with testing.

                
> TestDFVariations fails on Windows after the merge
> -------------------------------------------------
>
>                 Key: HADOOP-9387
>                 URL: https://issues.apache.org/jira/browse/HADOOP-9387
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 3.0.0
>            Reporter: Ivan Mitic
>            Assignee: Ivan Mitic
>         Attachments: HADOOP-9387.trunk.2.patch, HADOOP-9387.trunk.patch
>
>
> Test fails with the following errors:
> {code}
> Running org.apache.hadoop.fs.TestDFVariations
> Tests run: 4, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 0.186 sec <<<
FAILURE!
> testOSParsing(org.apache.hadoop.fs.TestDFVariations)  Time elapsed: 109 sec  <<<
ERROR!
> java.io.IOException: Fewer lines of output than expected
>         at org.apache.hadoop.fs.DF.parseOutput(DF.java:203)
>         at org.apache.hadoop.fs.DF.getMount(DF.java:150)
>         at org.apache.hadoop.fs.TestDFVariations.testOSParsing(TestDFVariations.java:59)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
>         at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
>         at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
>         at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
>         at org.junit.internal.runners.statements.FailOnTimeout$1.run(FailOnTimeout.java:28)
> testGetMountCurrentDirectory(org.apache.hadoop.fs.TestDFVariations)  Time elapsed: 1
sec  <<< ERROR!
> java.io.IOException: Fewer lines of output than expected
>         at org.apache.hadoop.fs.DF.parseOutput(DF.java:203)
>         at org.apache.hadoop.fs.DF.getMount(DF.java:150)
>         at org.apache.hadoop.fs.TestDFVariations.testGetMountCurrentDirectory(TestDFVariations.java:139)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
>         at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
>         at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
>         at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
>         at org.junit.internal.runners.statements.FailOnTimeout$1.run(FailOnTimeout.java:28)
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message