hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eli Collins <...@cloudera.com>
Subject Re: Fuse-DFS
Date Thu, 06 May 2010 15:44:23 GMT
Hey Seraph,

The -d is just a debug option, to print output to the terminal,
shouldn't change the execution. I suspect something else is going on,
perhaps you have an old fuse process running?

Thanks,
Eli

On Thu, May 6, 2010 at 4:55 AM, Seraph Imalia <seraph@eisp.co.za> wrote:
> Hi,
>
> I am experiencing an annoying problem...
>
> When I run this "./fuse_dfs_wrapper.sh dfs://dynobuntu10:8020 /Volumes/hdfs
> -obig_writes -d" then everything works fine.
>
> When I run this "./fuse_dfs_wrapper.sh dfs://dynobuntu10:8020 /Volumes/hdfs
> -obig_writes" (i.e. remove the daemonize option) I get issues connecting to
> hadoop...
>
> Here is what is in /var/log/syslog after running "ls /Volumes/hdfs"
>
> May  6 12:57:16 dynobuntu17 fuse_dfs: ERROR: could not connect to
> dynobuntu10:8020 fuse_impls_getattr.c:37
> May  6 12:57:23 dynobuntu17 fuse_dfs: ERROR: could not connect to
> dynobuntu10:8020 fuse_impls_getattr.c:37
>
> No errors appear in the syslog when the -d option is specified.
>
> I also tried port 9000 with exactly the same results.  There is no firewall
> software installed on any of the servers.
>
> It almost appears as something is stopping the connection to a the
> nameserver when fuse_dfs runs as a daemon.  Please help.
> We are running OS:  Ubuntu 9.04 x64 (2.6.28-11-server #42-Ubuntu SMP Fri Apr
> 17 02:45:36 UTC 2009 GNU/Linux)
>
> Seraph
>
>
>
> On 06 May 2010, at 11:31 AM, Seraph Imalia wrote:
>
>> Hi Eli,
>>
>> Thank you very much - applying patch HDFS-961-2.patch and re-building
>> resolved the problem.
>>
>> Regards,
>> Seraph
>>
>> On 05 May 2010, at 11:20 PM, Eli Collins wrote:
>>
>>> Try using a port besides 8020 or applying the patch for HDFS-961.
>>>
>>> Thanks,
>>> Eli
>>>
>>> On Wed, May 5, 2010 at 9:02 AM, Seraph Imalia <seraph@eisp.co.za> wrote:
>>>>
>>>> I have successfully managed to mount a dfs - but here is what I am
>>>> experiencing...
>>>> I mounted to /Volumes/hdfs  when I run "ls /Volumes/hdfs" it works fine
>>>> and
>>>> returns no results.
>>>> so I ran "mkdir /Volumes/hdfs/test" and it was successful and I could
>>>> see
>>>> the new directory exists using the web interface to browse hadoop.
>>>> but now when I run "ls /Volumes/hdfs" it does this...
>>>> ls: cannot access /Volumes/hdfs/p-site.x0: No such file or directory
>>>> p-site.x0
>>>>
>>>> and on the terminal window where I mounted it, it dumps this...
>>>> LOOKUP /p-site.x0
>>>>  unique: 13, error: -2 (No such file or directory), outsize: 16
>>>> unique: 14, opcode: READDIR (28), nodeid: 1, insize: 80
>>>>  unique: 14, error: 0 (Success), outsize: 16
>>>> unique: 15, opcode: RELEASEDIR (29), nodeid: 1, insize: 64
>>>>  unique: 15, error: 0 (Success), outsize: 16
>>>> unique: 16, opcode: GETATTR (3), nodeid: 1, insize: 56
>>>>  unique: 16, error: 0 (Success), outsize: 112
>>>> unique: 17, opcode: OPENDIR (27), nodeid: 1, insize: 48
>>>>  unique: 17, error: 0 (Success), outsize: 32
>>>> unique: 18, opcode: READDIR (28), nodeid: 1, insize: 80
>>>>  unique: 18, error: 0 (Success), outsize: 112
>>>> unique: 19, opcode: GETATTR (3), nodeid: 1, insize: 56
>>>>  unique: 19, error: 0 (Success), outsize: 112
>>>> unique: 20, opcode: LOOKUP (1), nodeid: 1, insize: 42
>>>>
>>>> I am not sure what this debug info means?
>>>> Here is everything that happened on terminal 1 and 2...
>>>> terminal 1
>>>> root@dynobuntu17:/opt/hadoop-0.20.2#
>>>> contrib/fuse-dfs/fuse_dfs_wrapper.sh
>>>> dfs://dynobuntu10:8020 /Volumes/hdfs -d
>>>> port=8020,server=dynobuntu10
>>>> fuse-dfs didn't recognize /Volumes/hdfs,-2
>>>> fuse-dfs ignoring option -d
>>>> unique: 1, opcode: INIT (26), nodeid: 0, insize: 56
>>>> INIT: 7.10
>>>> flags=0x0000003b
>>>> max_readahead=0x00020000
>>>>  INIT: 7.8
>>>>  flags=0x00000001
>>>>  max_readahead=0x00020000
>>>>  max_write=0x00020000
>>>>  unique: 1, error: 0 (Success), outsize: 40
>>>> unique: 2, opcode: GETATTR (3), nodeid: 1, insize: 56
>>>>  unique: 2, error: 0 (Success), outsize: 112
>>>> unique: 3, opcode: OPENDIR (27), nodeid: 1, insize: 48
>>>>  unique: 3, error: 0 (Success), outsize: 32
>>>> unique: 4, opcode: READDIR (28), nodeid: 1, insize: 80
>>>>  unique: 4, error: -2 (No such file or directory), outsize: 16
>>>> unique: 5, opcode: RELEASEDIR (29), nodeid: 1, insize: 64
>>>>  unique: 5, error: 0 (Success), outsize: 16
>>>> unique: 6, opcode: GETATTR (3), nodeid: 1, insize: 56
>>>>  unique: 6, error: 0 (Success), outsize: 112
>>>> unique: 7, opcode: LOOKUP (1), nodeid: 1, insize: 45
>>>> LOOKUP /test
>>>>  unique: 7, error: -2 (No such file or directory), outsize: 16
>>>> unique: 8, opcode: MKDIR (9), nodeid: 1, insize: 53
>>>> MKDIR /test
>>>>  NODEID: 2
>>>>  unique: 8, error: 0 (Success), outsize: 136
>>>> unique: 9, opcode: GETATTR (3), nodeid: 1, insize: 56
>>>>  unique: 9, error: 0 (Success), outsize: 112
>>>> unique: 10, opcode: OPENDIR (27), nodeid: 1, insize: 48
>>>>  unique: 10, error: 0 (Success), outsize: 32
>>>> unique: 11, opcode: READDIR (28), nodeid: 1, insize: 80
>>>>  unique: 11, error: 0 (Success), outsize: 120
>>>> unique: 12, opcode: GETATTR (3), nodeid: 1, insize: 56
>>>>  unique: 12, error: 0 (Success), outsize: 112
>>>> unique: 13, opcode: LOOKUP (1), nodeid: 1, insize: 50
>>>> LOOKUP /p-site.x0
>>>>  unique: 13, error: -2 (No such file or directory), outsize: 16
>>>> unique: 14, opcode: READDIR (28), nodeid: 1, insize: 80
>>>>  unique: 14, error: 0 (Success), outsize: 16
>>>> unique: 15, opcode: RELEASEDIR (29), nodeid: 1, insize: 64
>>>>  unique: 15, error: 0 (Success), outsize: 16
>>>> unique: 16, opcode: GETATTR (3), nodeid: 1, insize: 56
>>>>  unique: 16, error: 0 (Success), outsize: 112
>>>> unique: 17, opcode: OPENDIR (27), nodeid: 1, insize: 48
>>>>  unique: 17, error: 0 (Success), outsize: 32
>>>> unique: 18, opcode: READDIR (28), nodeid: 1, insize: 80
>>>>  unique: 18, error: 0 (Success), outsize: 112
>>>> unique: 19, opcode: GETATTR (3), nodeid: 1, insize: 56
>>>>  unique: 19, error: 0 (Success), outsize: 112
>>>> unique: 20, opcode: LOOKUP (1), nodeid: 1, insize: 42
>>>> LOOKUP /%
>>>>  unique: 20, error: -2 (No such file or directory), outsize: 16
>>>> unique: 21, opcode: READDIR (28), nodeid: 1, insize: 80
>>>>  unique: 21, error: 0 (Success), outsize: 16
>>>> unique: 22, opcode: RELEASEDIR (29), nodeid: 1, insize: 64
>>>>  unique: 22, error: 0 (Success), outsize: 16
>>>> unique: 23, opcode: GETATTR (3), nodeid: 1, insize: 56
>>>>  unique: 23, error: 0 (Success), outsize: 112
>>>> unique: 24, opcode: OPENDIR (27), nodeid: 1, insize: 48
>>>>  unique: 24, error: 0 (Success), outsize: 32
>>>> unique: 25, opcode: READDIR (28), nodeid: 1, insize: 80
>>>>  unique: 25, error: 0 (Success), outsize: 112
>>>> unique: 26, opcode: GETATTR (3), nodeid: 1, insize: 56
>>>>  unique: 26, error: 0 (Success), outsize: 112
>>>> unique: 27, opcode: LOOKUP (1), nodeid: 1, insize: 42
>>>> LOOKUP /1
>>>>  unique: 27, error: -2 (No such file or directory), outsize: 16
>>>> unique: 28, opcode: READDIR (28), nodeid: 1, insize: 80
>>>>  unique: 28, error: 0 (Success), outsize: 16
>>>> unique: 29, opcode: RELEASEDIR (29), nodeid: 1, insize: 64
>>>>  unique: 29, error: 0 (Success), outsize: 16
>>>> unique: 30, opcode: GETATTR (3), nodeid: 1, insize: 56
>>>>  unique: 30, error: 0 (Success), outsize: 112
>>>> unique: 31, opcode: OPENDIR (27), nodeid: 2, insize: 48
>>>>  unique: 31, error: 0 (Success), outsize: 32
>>>> unique: 32, opcode: READDIR (28), nodeid: 2, insize: 80
>>>>  unique: 32, error: -2 (No such file or directory), outsize: 16
>>>> unique: 33, opcode: RELEASEDIR (29), nodeid: 2, insize: 64
>>>>  unique: 33, error: 0 (Success), outsize: 16
>>>> unique: 34, opcode: OPENDIR (27), nodeid: 1, insize: 48
>>>>  unique: 34, error: 0 (Success), outsize: 32
>>>> unique: 35, opcode: READDIR (28), nodeid: 1, insize: 80
>>>>  unique: 35, error: 0 (Success), outsize: 120
>>>> unique: 36, opcode: GETATTR (3), nodeid: 1, insize: 56
>>>>  unique: 36, error: 0 (Success), outsize: 112
>>>> unique: 37, opcode: LOOKUP (1), nodeid: 1, insize: 50
>>>> LOOKUP /d-site.x0
>>>>  unique: 37, error: -2 (No such file or directory), outsize: 16
>>>> unique: 38, opcode: READDIR (28), nodeid: 1, insize: 80
>>>>  unique: 38, error: 0 (Success), outsize: 16
>>>> unique: 39, opcode: RELEASEDIR (29), nodeid: 1, insize: 64
>>>>  unique: 39, error: 0 (Success), outsize: 16
>>>> unique: 40, opcode: GETATTR (3), nodeid: 1, insize: 56
>>>>  unique: 40, error: 0 (Success), outsize: 112
>>>> unique: 41, opcode: LOOKUP (1), nodeid: 1, insize: 45
>>>> LOOKUP /test
>>>>  NODEID: 2
>>>>  unique: 41, error: 0 (Success), outsize: 136
>>>> unique: 42, opcode: LOOKUP (1), nodeid: 2, insize: 47
>>>> LOOKUP /test/inside
>>>>  unique: 42, error: -2 (No such file or directory), outsize: 16
>>>> unique: 43, opcode: MKDIR (9), nodeid: 2, insize: 55
>>>> MKDIR /test/inside
>>>>  NODEID: 3
>>>>  unique: 43, error: 0 (Success), outsize: 136
>>>> unique: 44, opcode: GETATTR (3), nodeid: 2, insize: 56
>>>>  unique: 44, error: 0 (Success), outsize: 112
>>>> unique: 45, opcode: OPENDIR (27), nodeid: 2, insize: 48
>>>>  unique: 45, error: 0 (Success), outsize: 32
>>>> unique: 46, opcode: READDIR (28), nodeid: 2, insize: 80
>>>>  unique: 46, error: 0 (Success), outsize: 112
>>>> unique: 47, opcode: GETATTR (3), nodeid: 2, insize: 56
>>>>  unique: 47, error: 0 (Success), outsize: 112
>>>> unique: 48, opcode: LOOKUP (1), nodeid: 2, insize: 42
>>>> LOOKUP /test/e
>>>>  unique: 48, error: -2 (No such file or directory), outsize: 16
>>>> unique: 49, opcode: READDIR (28), nodeid: 2, insize: 80
>>>>  unique: 49, error: 0 (Success), outsize: 16
>>>> unique: 50, opcode: RELEASEDIR (29), nodeid: 2, insize: 64
>>>>  unique: 50, error: 0 (Success), outsize: 16
>>>> unique: 51, opcode: GETATTR (3), nodeid: 2, insize: 56
>>>>  unique: 51, error: 0 (Success), outsize: 112
>>>> unique: 52, opcode: OPENDIR (27), nodeid: 2, insize: 48
>>>>  unique: 52, error: 0 (Success), outsize: 32
>>>> unique: 53, opcode: READDIR (28), nodeid: 2, insize: 80
>>>>  unique: 53, error: 0 (Success), outsize: 112
>>>> unique: 54, opcode: GETATTR (3), nodeid: 2, insize: 56
>>>>  unique: 54, error: 0 (Success), outsize: 112
>>>> unique: 55, opcode: LOOKUP (1), nodeid: 2, insize: 42
>>>> LOOKUP /test/e
>>>>  unique: 55, error: -2 (No such file or directory), outsize: 16
>>>> unique: 56, opcode: READDIR (28), nodeid: 2, insize: 80
>>>>  unique: 56, error: 0 (Success), outsize: 16
>>>> unique: 57, opcode: RELEASEDIR (29), nodeid: 2, insize: 64
>>>>  unique: 57, error: 0 (Success), outsize: 16
>>>> unique: 58, opcode: GETATTR (3), nodeid: 2, insize: 56
>>>>  unique: 58, error: 0 (Success), outsize: 112
>>>> unique: 59, opcode: OPENDIR (27), nodeid: 2, insize: 48
>>>>  unique: 59, error: 0 (Success), outsize: 32
>>>> unique: 60, opcode: READDIR (28), nodeid: 2, insize: 80
>>>>  unique: 60, error: 0 (Success), outsize: 112
>>>> unique: 61, opcode: GETATTR (3), nodeid: 2, insize: 56
>>>>  unique: 61, error: 0 (Success), outsize: 112
>>>> unique: 62, opcode: LOOKUP (1), nodeid: 2, insize: 42
>>>> LOOKUP /test/e
>>>>  unique: 62, error: -2 (No such file or directory), outsize: 16
>>>> unique: 63, opcode: READDIR (28), nodeid: 2, insize: 80
>>>>  unique: 63, error: 0 (Success), outsize: 16
>>>> unique: 64, opcode: RELEASEDIR (29), nodeid: 2, insize: 64
>>>>  unique: 64, error: 0 (Success), outsize: 16
>>>> unique: 65, opcode: GETATTR (3), nodeid: 2, insize: 56
>>>>  unique: 65, error: 0 (Success), outsize: 112
>>>> unique: 66, opcode: OPENDIR (27), nodeid: 2, insize: 48
>>>>  unique: 66, error: 0 (Success), outsize: 32
>>>> unique: 67, opcode: READDIR (28), nodeid: 2, insize: 80
>>>>  unique: 67, error: 0 (Success), outsize: 112
>>>> unique: 68, opcode: GETATTR (3), nodeid: 2, insize: 56
>>>>  unique: 68, error: 0 (Success), outsize: 112
>>>> unique: 69, opcode: LOOKUP (1), nodeid: 2, insize: 42
>>>> LOOKUP /test/e
>>>>  unique: 69, error: -2 (No such file or directory), outsize: 16
>>>> unique: 70, opcode: READDIR (28), nodeid: 2, insize: 80
>>>>  unique: 70, error: 0 (Success), outsize: 16
>>>> unique: 71, opcode: RELEASEDIR (29), nodeid: 2, insize: 64
>>>>  unique: 71, error: 0 (Success), outsize: 16
>>>> unique: 72, opcode: GETATTR (3), nodeid: 2, insize: 56
>>>>  unique: 72, error: 0 (Success), outsize: 112
>>>> unique: 73, opcode: OPENDIR (27), nodeid: 2, insize: 48
>>>>  unique: 73, error: 0 (Success), outsize: 32
>>>> unique: 74, opcode: READDIR (28), nodeid: 2, insize: 80
>>>>  unique: 74, error: 0 (Success), outsize: 112
>>>> unique: 75, opcode: GETATTR (3), nodeid: 2, insize: 56
>>>>  unique: 75, error: 0 (Success), outsize: 112
>>>> unique: 76, opcode: LOOKUP (1), nodeid: 2, insize: 42
>>>> LOOKUP /test/e
>>>>  unique: 76, error: -2 (No such file or directory), outsize: 16
>>>> unique: 77, opcode: READDIR (28), nodeid: 2, insize: 80
>>>>  unique: 77, error: 0 (Success), outsize: 16
>>>> unique: 78, opcode: RELEASEDIR (29), nodeid: 2, insize: 64
>>>>  unique: 78, error: 0 (Success), outsize: 16
>>>> terminal 2
>>>> root@dynobuntu17:~# ls /Volumes/hdfs/
>>>> root@dynobuntu17:~# mkdir /Volumes/hdfs/test
>>>> root@dynobuntu17:~# ls /Volumes/hdfs/
>>>> ls: cannot access /Volumes/hdfs/p-site.x0: No such file or directory
>>>> p-site.x0
>>>> root@dynobuntu17:~# ls /Volumes/hdfs/
>>>> ls: cannot access /Volumes/hdfs/%: No such file or directory
>>>> %
>>>> root@dynobuntu17:~# ls /Volumes/hdfs
>>>> ls: cannot access /Volumes/hdfs/1: No such file or directory
>>>> 1
>>>> root@dynobuntu17:~# ls /Volumes/hdfs/test
>>>> root@dynobuntu17:~# ls /Volumes/hdfs/
>>>> ls: cannot access /Volumes/hdfs/d-site.x0: No such file or directory
>>>> d-site.x0
>>>> root@dynobuntu17:~# mkdir /Volumes/hdfs/test/inside
>>>> root@dynobuntu17:~# ls /Volumes/hdfs/test
>>>> ls: cannot access /Volumes/hdfs/test/e: No such file or directory
>>>> e
>>>> root@dynobuntu17:~# ls /Volumes/hdfs/test
>>>> ls: cannot access /Volumes/hdfs/test/e: No such file or directory
>>>> e
>>>> root@dynobuntu17:~# ls /Volumes/hdfs/\test
>>>> ls: cannot access /Volumes/hdfs/test/e: No such file or directory
>>>> e
>>>> root@dynobuntu17:~# ls /Volumes/hdfs/\te\s\t
>>>> ls: cannot access /Volumes/hdfs/test/e: No such file or directory
>>>> e
>>>> root@dynobuntu17:~# ls /Volumes/hdfs/\t\e\s\t
>>>> ls: cannot access /Volumes/hdfs/test/e: No such file or directory
>>>> e
>>>>
>>>> On 05 May 2010, at 4:57 PM, Seraph Imalia wrote:
>>>>
>>>> Awesome! - thank you  after running apt-get install libfuse-dev I got a
>>>> successful build.
>>>>
>>>> Thanks for your help :)
>>>>
>>>>
>>>> On 05 May 2010, at 4:19 PM, Jason Venner wrote:
>>>>
>>>> It does not look like you have the fuse-devel package installed on your
>>>> system.
>>>>
>>>> On Wed, May 5, 2010 at 7:14 AM, Seraph Imalia <seraph@eisp.co.za> wrote:
>>>>
>>>> Hi Jason,
>>>>
>>>> Thank you,  I had autoconf installed but not automake - after installing
>>>>
>>>> automake, the build went further, but this time failed with...
>>>>
>>>> BUILD FAILED
>>>>
>>>> /opt/hadoop-0.20.2/build.xml:497: The following error occurred while
>>>>
>>>> executing this line:
>>>>
>>>> /opt/hadoop-0.20.2/src/contrib/build.xml:30: The following error
>>>> occurred
>>>>
>>>> while executing this line:
>>>>
>>>> /opt/hadoop-0.20.2/src/contrib/fuse-dfs/build.xml:57: exec returned: 2
>>>>
>>>> The output from the build is attached.
>>>>
>>>>
>>>>
>>>> We are running OS:  Ubuntu 9.04 x64 (2.6.28-11-server #42-Ubuntu SMP Fri
>>>> Apr
>>>>
>>>> 17 02:45:36 UTC 2009 GNU/Linux)
>>>>
>>>> Regards,
>>>>
>>>> Seraph
>>>>
>>>>
>>>> On 05 May 2010, at 3:47 PM, Jason Venner wrote:
>>>>
>>>> You will need to install the gnu development tool chain for your
>>>>
>>>> platform, a quick check on an older redhat system I have suggests the
>>>>
>>>> automake and autoconf rpm's will provide aclocal, automake and
>>>>
>>>> autoconf.
>>>>
>>>> The ./configure: not found error is an artifact of the earlier failures.
>>>>
>>>>
>>>> On Wed, May 5, 2010 at 5:37 AM, Seraph Imalia <seraph@eisp.co.za> wrote:
>>>>
>>>> Hi,
>>>>
>>>> I have been following the following URL to mount an HDFS using
>>>>
>>>> fuse-dfs: http://wiki.apache.org/hadoop/MountableHDFS
>>>>
>>>> I have had many problems trying to build it but have managed to get
>>>>
>>>> through
>>>>
>>>> the first two build commands without build errors.  Running the last
>>>>
>>>> build
>>>>
>>>> command "ant compile-contrib -Dlibhdfs=1 -Dfusedfs=1" results in the
>>>>
>>>> following error:
>>>>
>>>> compile:
>>>>
>>>>  [echo] contrib: fuse-dfs
>>>>
>>>>  [exec] /opt/hadoop-0.20.1/src/contrib/fuse-dfs/bootstrap.sh: 18:
>>>>
>>>> aclocal: not found
>>>>
>>>>  [exec] /opt/hadoop-0.20.1/src/contrib/fuse-dfs/bootstrap.sh: 19:
>>>>
>>>> automake: not found
>>>>
>>>>  [exec] /opt/hadoop-0.20.1/src/contrib/fuse-dfs/bootstrap.sh: 20:
>>>>
>>>> autoconf: not found
>>>>
>>>>  [exec] /opt/hadoop-0.20.1/src/contrib/fuse-dfs/bootstrap.sh: 21:
>>>>
>>>> ./configure: not found
>>>>
>>>> BUILD FAILED
>>>>
>>>> /opt/hadoop-0.20.1/build.xml:497: The following error occurred while
>>>>
>>>> executing this line:
>>>>
>>>> /opt/hadoop-0.20.1/src/contrib/build.xml:30: The following error
>>>> occurred
>>>>
>>>> while executing this line:
>>>>
>>>> /opt/hadoop-0.20.1/src/contrib/fuse-dfs/build.xml:54: exec returned: 127
>>>>
>>>> I found this link on
>>>>
>>>> google: http://issues.apache.org/jira/browse/HADOOP-4 which does not
>>>> seem
>>>>
>>>> to
>>>>
>>>> help me.
>>>>
>>>> Please can you assist?
>>>>
>>>> Regards,
>>>>
>>>> Seraph
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Pro Hadoop, a book to guide you from beginner to hadoop mastery,
>>>>
>>>> http://www.amazon.com/dp/1430219424?tag=jewlerymall
>>>>
>>>> www.prohadoopbook.com a community for Hadoop Professionals
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Pro Hadoop, a book to guide you from beginner to hadoop mastery,
>>>>
>>>> http://www.amazon.com/dp/1430219424?tag=jewlerymall
>>>>
>>>> www.prohadoopbook.com a community for Hadoop Professionals
>>>>
>>>>
>>>>
>>>>
>>
>>
>
>
>

Mime
View raw message