hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-420) Fuse-dfs should cache fs handles
Date Mon, 20 Jun 2011 16:26:47 GMT

    [ https://issues.apache.org/jira/browse/HDFS-420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13052042#comment-13052042
] 

Hudson commented on HDFS-420:
-----------------------------

Integrated in Hadoop-Hdfs-trunk-Commit #751 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/751/])
    HDFS-420. Fuse-dfs should cache fs handles. Contributed by Brian Bockelman and Eli Collins

eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1137675
Files : 
* /hadoop/common/trunk/hdfs/src/contrib/build-contrib.xml
* /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_unlink.c
* /hadoop/common/trunk/hdfs/CHANGES.txt
* /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_getattr.c
* /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_release.c
* /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_utimens.c
* /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_options.c
* /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_stat_struct.c
* /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_dfs_wrapper.sh
* /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_rename.c
* /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_mkdir.c
* /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_statfs.c
* /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_rmdir.c
* /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/build.xml
* /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_users.c
* /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_init.c
* /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_access.c
* /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/configure.ac
* /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_truncate.c
* /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_connect.c
* /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_readdir.c
* /hadoop/common/trunk/hdfs/src/contrib/build.xml
* /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_open.c
* /hadoop/common/trunk/hdfs/src/c++/libhdfs/hdfs.c
* /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_connect.h
* /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_dfs.c
* /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_chmod.c
* /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_impls_chown.c
* /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_context_handle.h
* /hadoop/common/trunk/hdfs/src/contrib/fuse-dfs/src/fuse_dfs.h


> Fuse-dfs should cache fs handles
> --------------------------------
>
>                 Key: HDFS-420
>                 URL: https://issues.apache.org/jira/browse/HDFS-420
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: contrib/fuse-dfs
>    Affects Versions: 0.20.2
>         Environment: Fedora core 10, x86_64, 2.6.27.7-134.fc10.x86_64 #1 SMP (AMD 64),
gcc 4.3.2, java 1.6.0 (IcedTea6 1.4 (fedora-7.b12.fc10-x86_64) Runtime Environment (build
1.6.0_0-b12) OpenJDK 64-Bit Server VM (build 10.0-b19, mixed mode)
>            Reporter: Dima Brodsky
>            Assignee: Brian Bockelman
>             Fix For: 0.23.0
>
>         Attachments: fuse_dfs_020_memleaks.patch, fuse_dfs_020_memleaks_v3.patch, fuse_dfs_020_memleaks_v8.patch,
hdfs-420-1.patch, hdfs-420-2.patch, hdfs-420-3.patch
>
>
> Fuse-dfs should cache fs handles on a per-user basis. This significantly increases performance,
and has the side effect of fixing the current code which leaks fs handles.
> The original bug description follows:
> I run the following test:
> 1.  Run hadoop DFS in single node mode
> 2.  start up fuse_dfs
> 3.  copy my source tree, about 250 megs, into the DFS
>      cp -av * /mnt/hdfs/
> in /var/log/messages I keep seeing:
> Dec 22 09:02:08 bodum fuse_dfs: ERROR: hdfs trying to utime /bar/backend-trunk2/src/machinery/hadoop/output/2008/11/19
to 1229385138/1229963739
> and then eventually
> Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1333
> Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1333
> Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1037
> Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1333
> Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1037
> Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1333
> Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1209
> Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1037
> Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1037
> Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1037
> Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1037
> Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1037
> Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1037
> Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1037
> Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1037
> Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1037
> Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1209
> Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1037
> Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1037
> Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1037
> Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1333
> Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1209
> Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1037
> and the file system hangs.  hadoop is still running and I don't see any errors in it's
logs.  I have to unmount the dfs and restart fuse_dfs and then everything is fine again. 
At some point I see the following messages in the /var/log/messages:
> ERROR: dfs problem - could not close file_handle(139677114350528) for /bar/backend-trunk2/src/machinery/hadoop/input/2008/12/14/actionrecordlog-8339-93825052368848-1229278807.log
fuse_dfs.c:1464
> Dec 22 09:04:49 bodum fuse_dfs: ERROR: dfs problem - could not close file_handle(139676770220176)
for /bar/backend-trunk2/src/machinery/hadoop/input/2008/12/14/actionrecordlog-8140-93825025883216-1229278759.log
fuse_dfs.c:1464
> Dec 22 09:05:13 bodum fuse_dfs: ERROR: dfs problem - could not close file_handle(139677114812832)
for /bar/backend-trunk2/src/machinery/hadoop/input/2008/12/14/actionrecordlog-8138-93825070138960-1229251587.log
fuse_dfs.c:1464
> Is this a known issue?  Am I just flooding the system too much.  All of this is being
performed on a single, dual core, machine.
> Thanks!
> ttyl
> Dima

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message