hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alan Burlison (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-12603) TestSymlinkLocalFSFileContext#testSetTimesSymlinkToDir occasionally fail
Date Tue, 01 Dec 2015 17:56:11 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-12603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15034198#comment-15034198

Alan Burlison commented on HADOOP-12603:

If I look on a Centos machine I see that / is mounted with the following options:

/dev/mapper/centos-root on / type xfs (rw,relatime,seclabel,attr2,inode64,noquota)

The mount(8) manpage says this about relatime:

              Update inode access times relative to  modify  or  change  time.
              Access time is only updated if the previous access time was earlier
              than the current modify or change time. (Similar  to  noatime,
              but  doesn't break mutt or other applications that need to
              know if a file has been read since the last time  it  was  modified.)

              Since Linux 2.6.30, the kernel defaults to the behavior provided
              by this option (unless noatime was  specified), and the strictatime
              option  is  required  to  obtain traditional semantics. In
              addition, since Linux 2.6.30, the file's  last  access  time  is
              always  updated  if  it  is more than 1 day old.

That's not POSIX-conformant behaviour, and as it says is the default from 2.6.30 onwards.
Solaris has a noatime mount option which is the same as Linux's noatime, but no direct equivalent
to relatime, although on UFS it has dfratime which is similar. ZFS, which is now the default
Solaris filesystem, only has noatime. BSD only appears to have noatime as well. Linux also
has nodiratime which isn't available on either Solaris or BSD. Windows varies depending on
filesystem type but NTFS may defer updates to atime for up to an hour and you can also disable
atime updates entirely. On FAT the resolution of atime is 1 day. 

Leaving aside the issue of POSIX correctness, unless you implement a probe that understands
the semantics of atime updates across all relevant systems and that probes for the mount options
of the filesystem you are testing on, any access time behaviour tests are almost certainly
going to be unreliable.

Rather than trying to bodge up these tests, I believe it's better to remove them entirely.

More here: https://en.wikibooks.org/wiki/C_Programming/POSIX_Reference/sys/stat.h/stat#Criticism_of_atime

> TestSymlinkLocalFSFileContext#testSetTimesSymlinkToDir occasionally fail
> ------------------------------------------------------------------------
>                 Key: HADOOP-12603
>                 URL: https://issues.apache.org/jira/browse/HADOOP-12603
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Wei-Chiu Chuang
>            Assignee: Wei-Chiu Chuang
>              Labels: solaris, windows
>         Attachments: HADOOP-12603.001.patch, HADOOP-12603.002.patch
> I have observed this test failure a few times in the past. When it fails, the expected
access time (of the file link) is always 1000 less than the actual access time.
> Error Message
> {noformat}
> expected:<1448478654000> but was:<1448478655000>
> {noformat}
> Stacktrace
> {noformat}
> java.lang.AssertionError: expected:<1448478654000> but was:<1448478655000>
> 	at org.junit.Assert.fail(Assert.java:88)
> 	at org.junit.Assert.failNotEquals(Assert.java:743)
> 	at org.junit.Assert.assertEquals(Assert.java:118)
> 	at org.junit.Assert.assertEquals(Assert.java:555)
> 	at org.junit.Assert.assertEquals(Assert.java:542)
> 	at org.apache.hadoop.fs.SymlinkBaseTest.testSetTimesSymlinkToDir(SymlinkBaseTest.java:1391)
> 	at org.apache.hadoop.fs.TestSymlinkLocalFS.testSetTimesSymlinkToDir(TestSymlinkLocalFS.java:233)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:606)
> 	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> 	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> 	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 	at org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
> Standard Output
> {noformat}
> 2015-11-25 19:10:55,231 WARN  fs.FileUtil (FileUtil.java:symLink(813)) - Command 'ln
-s /testptch/hadoop/hadoop-common-project/hadoop-common/target/test/data/4/vae1ng5t75/test1/file
failed 1 with: ln: failed to create symbolic link '/testptch/hadoop/hadoop-common-project/hadoop-common/target/test/data/4/vae1ng5t75/test2/linkToFile':
No such file or directory
> 2015-11-25 19:10:56,212 WARN  fs.FileUtil (FileUtil.java:symLink(813)) - Command 'ln
-s /testptch/hadoop/hadoop-common-project/hadoop-common/target/test/data/4/vae1ng5t75/test1/file
failed 1 with: ln: failed to create symbolic link '/testptch/hadoop/hadoop-common-project/hadoop-common/target/test/data/4/vae1ng5t75/test1/linkToFile':
File exists
> {noformat}

This message was sent by Atlassian JIRA

View raw message