Best reg= ards,
Zhang Bingjun (Eddy)

E-mail: eddymier@gmail.com, bingj= un@nus.edu.sg, bingjun@comp.= nus.edu.sg
Tel No: +65-96188110 (M)

On Tue, Oct 27, 2009 at 7:19 PM, Huy Pha= n <dachuy@gmail.co= m> wrote:

Hi Zhang,
I didn't play much with fuse-dfs, in my opinion, memory leak is somethi= ng solvable and I can see Apache had made some fixes for this issue on libh= dfs.
If you encounter these problems with older version of Hadoop, I think you s= hould give a try on the latest stable version.
Since I didn't have much fun so far with fuse-dfs, i cannot say it'= s the best or not, but it's definitely better than mixing davfs2 and we= bdav together.

Best,
Huy Phan

Zhang Bingjun (Eddy) wrote:

Dear Huy Phan,

Thanks for your quick reply.
I was using fuse-dfs before. But I found serious memory leak with fuse-dfs = about 10MB leakage per 10k file read/write. When the occupied memory size r= eached about 150MB, the read/write performance dropped dramatically. Did yo= u encounter these problems?

What I am trying to do is to mount HDFS as a local directory in Ubuntu. Do = you think fuse-dfs is the best option so far?

Thank you so much for your input!

Best regards,
Zhang Bingjun (Eddy)

E-mail: eddymier@gm= ail.com <mailto:eddymier@gmail.com>, bingjun@nus.edu.sg <mailto:bingjun@nus.edu.sg>, bingjun@comp.nus.edu.sg <m= ailto:bingjun@= comp.nus.edu.sg>
Tel No: +65-96188110 (M)

On Tue, Oct 27, 2009 at 6:55 PM, Huy Phan <dachuy@gmail.com <mailto:dachuy@gmail.com>> wrote:

=C2=A0 =C2=A0Hi Zhang,

=C2=A0 =C2=A0Here is the patch for davfs2 to solve "server does not s= upport
=C2=A0 =C2=A0WebDAV" issue:

=C2=A0 =C2=A0diff --git a/src/webdav.c b/src/webdav.c
=C2=A0 =C2=A0index 8ec7a2d..4bdaece 100644
=C2=A0 =C2=A0--- a/src/webdav.c
=C2=A0 =C2=A0+++ b/src/webdav.c
=C2=A0 =C2=A0@@ -472,7 +472,7 @@ dav_init_connection(const char *path)

=C2=A0 =C2=A0 =C2=A0 if (!ret) {
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 initialized =3D 1;
=C2=A0 =C2=A0- =C2=A0 =C2=A0 =C2=A0 =C2=A0if (!caps.dav_class1 && = !caps.dav_class2 &&
=C2=A0 =C2=A0 !ignore_dav_header) {
=C2=A0 =C2=A0+ =C2=A0 =C2=A0 =C2=A0 =C2=A0if (!caps.dav_class1 && = !ignore_dav_header) {
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 if (have_terminal) {
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 error(EXIT_= FAILURE, 0,
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 _("mounting failed; the server does not
=C2=A0 =C2=A0support WebDAV"));

=C2=A0 =C2=A0davfs2 and webdav is not a good mix actually, I had tried to = mix
=C2=A0 =C2=A0them together and the performance were really bad. With the l= oad
=C2=A0 =C2=A0test of 10 requests/s, load average on my namenode were alway= s >
=C2=A0 =C2=A015 and it took me about 5 mins for `ls` the root directory of= HDFS
=C2=A0 =C2=A0during the test.

=C2=A0 =C2=A0Since you're using Hadoop 0.20.1, it's better to use = fusedfs
=C2=A0 =C2=A0library provided in Hadoop package. You have to do some trick= s to
=C2=A0 =C2=A0compile fusedfs with Hadoop, otherwise it would take you a lo= t of
=C2=A0 =C2=A0time for compiling redundant things.

=C2=A0 =C2=A0Best,
=C2=A0 =C2=A0Huy Phan

=C2=A0 =C2=A0Zhang Bingjun (Eddy) wrote:

=C2=A0 =C2=A0 =C2=A0 =C2=A0Dear Huy Phan and others,

=C2=A0 =C2=A0 =C2=A0 =C2=A0Thanks a lot for your efforts in customizing th= e WebDav server
=C2=A0 =C2=A0 =C2=A0 =C2=A0<http://github.com/huyphan/HDFS-over-Webdav> and make it work
=C2=A0 =C2=A0 =C2=A0 =C2=A0for Hadoop-0.20.1.
=C2=A0 =C2=A0 =C2=A0 =C2=A0After setting up the WebDav server, I could acc= ess it using
=C2=A0 =C2=A0 =C2=A0 =C2=A0Cadaver client in Ubuntu without using any user= name password.
=C2=A0 =C2=A0 =C2=A0 =C2=A0Operations like deleting files, etc, were worki= ng. The command
=C2=A0 =C2=A0 =C2=A0 =C2=A0is: *cadaver http://server:9800*

=C2=A0 =C2=A0 =C2=A0 =C2=A0However, when I was trying to mount the WebDav = server using
=C2=A0 =C2=A0 =C2=A0 =C2=A0davfs2 in Ubuntu, I always get the following er= ror:
=C2=A0 =C2=A0 =C2=A0 =C2=A0"mount.davfs: mounting failed; the server = does not support
=C2=A0 =C2=A0 =C2=A0 =C2=A0WebDAV".

=C2=A0 =C2=A0 =C2=A0 =C2=A0I was promoted to input username and password l= ike below:
=C2=A0 =C2=A0 =C2=A0 =C2=A0hadoop@hdfs2:/mnt$ sudo mount.davfs
=C2=A0 =C2=A0 =C2=A0 =C2=A0 http://192.168.0.131:9800/test hdfs-webdav/
=C2=A0 =C2=A0 =C2=A0 =C2=A0Please enter the username to authenticate with = server
=C2=A0 =C2=A0 =C2=A0 =C2=A0http://192.168.0.131:9800/test or hit enter for none.
=C2=A0 =C2=A0 =C2=A0 =C2=A0Username: hadoop
=C2=A0 =C2=A0 =C2=A0 =C2=A0Please enter the password to authenticate user = hadoop with server
=C2=A0 =C2=A0 =C2=A0 =C2=A0http://192.168.0.131:9800/test or hit enter for none.
=C2=A0 =C2=A0 =C2=A0 =C2=A0Password:
=C2=A0 =C2=A0 =C2=A0 =C2=A0mount.davfs: mounting failed; the server does n= ot support WebDAV

=C2=A0 =C2=A0 =C2=A0 =C2=A0Even though I have tried all possible usernames= and passwords
=C2=A0 =C2=A0 =C2=A0 =C2=A0either from the WebDAV accounts.properties file= or from the
=C2=A0 =C2=A0 =C2=A0 =C2=A0Ubuntu system of the WebDAV server, I still got= this error
=C2=A0 =C2=A0 =C2=A0 =C2=A0message.
=C2=A0 =C2=A0 =C2=A0 =C2=A0Could you and anyone give me some hints on this= problem? How
=C2=A0 =C2=A0 =C2=A0 =C2=A0could I solve it? Very much appreciate your hel= p!

=C2=A0 =C2=A0 =C2=A0 =C2=A0Best regards,
=C2=A0 =C2=A0 =C2=A0 =C2=A0Zhang Bingjun (Eddy)

=C2=A0 =C2=A0 =C2=A0 =C2=A0E-mail: eddymier@gmail.com <mailto:eddymier@gmail.com>
=C2=A0 =C2=A0 =C2=A0 =C2=A0<mailto:eddymier@gmail.com <mailto:eddymier@gmail.com>>,

=C2=A0 =C2=A0 =C2=A0 =C2=A0bingjun@nus.edu.sg <mailto:bingjun@nus.edu.sg>
=C2=A0 =C2=A0 =C2=A0 =C2=A0<mailto:bingjun@nus.edu.sg <mailto:bingjun@nus.edu.sg>>,

=C2=A0 =C2=A0 =C2=A0 =C2=A0bingjun@comp.nus.edu.sg <mailto:bingjun@comp.nus.edu.sg>
<= /div> =C2=A0 =C2=A0 =C2=A0 =C2=A0<mailto:bingjun@comp.nus.edu.sg <mailto:bingjun@comp.nus.edu.sg>>

=C2=A0 =C2=A0 =C2=A0 =C2=A0Tel No: +65-96188110 (M)