hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eli Collins (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HDFS-780) Revive TestFuseDFS
Date Tue, 15 Mar 2011 07:29:29 GMT

     [ https://issues.apache.org/jira/browse/HDFS-780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Eli Collins updated HDFS-780:
-----------------------------

    Attachment: hdfs-780-4.patch

Thanks Todd! Comments follow.

bq. why use <exec executable="cp"...> instead of using the built-in ant copy task?

The copy task doesn't preserve the exec bit. See http://ant.apache.org/manual/Tasks/copy.html
Added a comment.

bq. not clear why you have to redefine test.classpath - is the one that comes from build-contrib.xml
not sufficient?

Not necessary, removed it. (not sure why I had it, did the first cut of this patch a while
back).

bq. why set fs.default.name property in build.xml? This becomes a Java system property which
afaik would be entirely ignored?

Ditto.

bq. I noticed AM_CPPFLAGS has -D_FILE_OFFSET_BITS=64, but seems to me this should be done
with autoconf AC_SYS_LARGEFILE. Probably out of scope for this review though.

For some reason adding AC_PROG_CC and AC_SYS_LARGEFILE to configure.ac was insufficient. Still
needed FILE_OFFSET_BITS, will punt on cleaning up the build further to a future change.

bq. spelling: "succesfully"

Fixed.

bq. javadoc for execAssertFalse is incorrect

Fixed.

bq. might be clearer to call those methods assertExecSucceeds and assertExecFails?

Agree, done.

bq. what's the purpose of the 1 second sleep in createFile? there's
some kind of delay between making a file and it being visible or
something? this seems like a suprising semantic and at least deserves
a comment

That was earlier debugging, however I just removed it and found that if there is not a sleep
after createFile in testCreate the call to checkFile fails with a premature EOF. I'll file
a bug to investigate further.

bq. in checkFile, maybe need a try...finally to avoid leaving file open on IOE?

Done.

bq. in checkFile, you assume that read won't return less than the
whole file. Maybe you could use IOUtil.copyBytes() into a
ByteArrayOutputStream?

Switched to IOUtil#readFully (ditto for creating the file).

bq. establishMount seems to assume libhdfs is installed in /usr/local/lib? Also uses the outdated
dfs:// URI type instead of hdfs://

Leftover from the previous version, fixed. fuse-dfs actually doesn't support "hdfs" in the
URI, will fix in a separate change.

bq. establishMount doesn't seem to check exit code of the mount command?

This is because the fuse-dfs process is run in debug mode (not daemonized) and it's stderr
and out are captured. On second thought it's better to run daemonized and just capture syslog.
Incidentally the hang in the last test was due to the test blocking on fuse-dfs and fuse-dfs
blocking on the test to read it's output. Fixed now.

To implement this last part I needed to make sure all useful debug info went to go both to
stderr and syslog. I've introduced macros that do this and modified all the fuse-dfs code
to use them. 

Btw I also removed bootstraph.sh and made the build file execute this (as part of the compile
target instead of the test target).

I also added a test that covers access via multiple threads.

I've run the tests on 64-bit Ubuntu Maverick and Centos 5 hosts.

> Revive TestFuseDFS
> ------------------
>
>                 Key: HDFS-780
>                 URL: https://issues.apache.org/jira/browse/HDFS-780
>             Project: Hadoop HDFS
>          Issue Type: Test
>          Components: contrib/fuse-dfs
>            Reporter: Eli Collins
>         Attachments: hdfs-780-1.patch, hdfs-780-2.patch, hdfs-780-3.patch, hdfs-780-4.patch
>
>
> Looks like TestFuseDFS has bit rot. Let's revive it.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message