hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rita M (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-4) tool to mount dfs on linux
Date Thu, 19 Jul 2012 11:14:34 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-4?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13418217#comment-13418217
] 

Rita M commented on HADOOP-4:
-----------------------------

By looking at 

hdfs            -fstype=fuse,rw,nodev,nonempty,noatime,allow_other  :/path/to/fuse_dfs_moutn/fuse_dfs.sh\#dfs\://namenode\:9000

Does this still work with a recent version of autofs? Can someone please confirm
                
> tool to mount dfs on linux
> --------------------------
>
>                 Key: HADOOP-4
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4
>             Project: Hadoop Common
>          Issue Type: Improvement
>         Environment: OSs that support FUSE. Includes Linux, MacOSx, OpenSolaris... http://fuse.sourceforge.net/wiki/index.php/OperatingSystems
>            Reporter: John Xing
>            Assignee: Pete Wyckoff
>             Fix For: 0.18.0
>
>         Attachments: HADOOP-4.patch, HADOOP-4.patch, HADOOP-4.patch, HADOOP-4.patch,
HADOOP-4.patch, HADOOP-4.patch, HADOOP-4.patch, Makefile, fuse-dfs.tar.gz, fuse-dfs.tar.gz,
fuse-dfs.tar.gz, fuse-dfs.tar.gz, fuse-dfs.tar.gz, fuse-hadoop-0.1.0_fuse-j.2.2.3_hadoop.0.5.0.tar.gz,
fuse-hadoop-0.1.0_fuse-j.2.4_hadoop.0.5.0.tar.gz, fuse-hadoop-0.1.1.tar.gz, fuse-j-hadoopfs-03.tar.gz,
fuse_dfs.c, fuse_dfs.c, fuse_dfs.c, fuse_dfs.c, fuse_dfs.c, fuse_dfs.sh, fuse_dfs.tar.gz,
patch.txt, patch.txt, patch2.txt, patch3.txt, patch4.txt, patch4.txt, patch4.txt, patch5.txt,
patch6.txt, patch6.txt
>
>
> This is a FUSE module for Hadoop's HDFS.
> It allows one to mount HDFS as a Unix filesystem and optionally export
> that mount point to other machines.
> rmdir, mv, mkdir, rm are all supported. just not cp, touch, ..., but actual writes require:
https://issues.apache.org/jira/browse/HADOOP-3485
> For the most up-to-date documentation, see: http://wiki.apache.org/hadoop/MountableHDFS
> BUILDING:
> Requirements:
>    1. a Linux kernel > 2.6.9 or a kernel module from FUSE - i.e., you
>    compile it yourself and then modprobe it. Better off with the
>    former option if possible.  (Note for now if you use the kernel
>    with fuse included, it doesn't allow you to export this through NFS
>    so be warned. See the FUSE email list for more about this.)
>    2. FUSE should be installed in /usr/local or FUSE_HOME ant
>    environment variable
> To build:
>    1. in HADOOP_HOME: ant compile-contrib -Dcompile.c++=1 -Dfusedfs=1 -Dlibhdfs=1
> NOTE: for amd64 architecture, libhdfs will not compile unless you edit
> the Makefile in src/c++/libhdfs/Makefile and set OS_ARCH=amd64
> (probably the same for others too).
> --------------------------------------------------------------------------------
> CONFIGURING:
> Look at all the paths in fuse_dfs_wrapper.sh and either correct them
> or set them in your environment before running. (note for automount
> and mount as root, you probably cannnot control the environment, so
> best to set them in the wrapper)
> INSTALLING:
> 1. mkdir /mnt/dfs (or wherever you want to mount it)
> 2. fuse_dfs_wrapper.sh dfs://hadoop_server1.foo.com:9000 /mnt/dfs -d
> ; and from another terminal, try ls /mnt/dfs
> If 2 works, try again dropping the debug mode, i.e., -d
> (note - common problems are that you don't have libhdfs.so or
> libjvm.so or libfuse.so on your LD_LIBRARY_PATH, and your CLASSPATH
> does not contain hadoop and other required jars.)
> --------------------------------------------------------------------------------
> DEPLOYING:
> in a root shell do the following:
> 1. add the following to /etc/fstab -
>   fuse_dfs#dfs://hadoop_server.foo.com:9000 /mnt/dfs fuse
>   allow_other,rw 0 0
> 2. mount /mnt/dfs Expect problems with not finding fuse_dfs. You will
>    need to probably add this to /sbin and then problems finding the
>    above 3 libraries. Add these using ldconfig.
> --------------------------------------------------------------------------------
> EXPORTING:
> Add the following to /etc/exports:
>   /mnt/hdfs *.foo.com(no_root_squash,rw,fsid=1,sync)
> NOTE - you cannot export this with a FUSE module built into the kernel
> - e.g., kernel 2.6.17. For info on this, refer to the FUSE wiki.
> --------------------------------------------------------------------------------
> ADVANCED:
> you may want to ensure certain directories cannot be deleted from the
> shell until the FS has permissions. You can set this in the build.xml
> file in src/contrib/fuse-dfs/build.xml

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message