hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eli Collins <...@cloudera.com>
Subject Re: Try to mount HDFS
Date Mon, 26 Apr 2010 21:00:14 GMT
The issue that required you changing ports is HDFS-961.

Thanks,
Eli


On Fri, Apr 23, 2010 at 6:30 AM, Christian Baun <cray@unix-ag.uni-kl.de> wrote:
> Brian,
>
> You got it!!! :-)
> It works (partly)!
>
> i switched to Port 9000. core-site.xml includes now:
>
>        <property>
>                <name>fs.default.name</name>
>                <value>hdfs://ec2-75-101-210-65.compute-1.amazonaws.com:9000</value>
>                <final>true</final>
>        </property>
>
>
> $ hadoop fs -ls /
> Found 1 items
> drwxr-xr-x   - hadoop supergroup          0 2010-04-23 05:18 /mnt
>
> $ hadoop fs -ls /mnt/
> Found 1 items
> drwxr-xr-x   - hadoop supergroup          0 2010-04-23 13:00 /mnt/mapred
>
> # ./fuse_dfs_wrapper.sh dfs://ec2-75-101-210-65.compute-1.amazonaws.com:9000 /mnt/hdfs/
> port=9000,server=ec2-75-101-210-65.compute-1.amazonaws.com
> fuse-dfs didn't recognize /mnt/hdfs/,-2
>
> This tiny error message remains.
>
> # mount | grep fuse
> fuse_dfs on /hdfs type fuse.fuse_dfs (rw,nosuid,nodev,allow_other,default_permissions)
>
> # ls /mnt/hdfs/
> mnt
> # mkdir /mnt/hdfs/testverzeichnis
> # touch /mnt/hdfs/testdatei
> # ls -l /mnt/hdfs/
> total 8
> drwxr-xr-x 3 hadoop 99 4096 2010-04-23 05:18 mnt
> -rw-r--r-- 1 root   99    0 2010-04-23 13:07 testdatei
> drwxr-xr-x 2 root   99 4096 2010-04-23 13:05 testverzeichnis
>
> In /var/log/messages there was no information about hdfs/fuse.
>
> Only in /var/log/user.log were these lines:
> Apr 23 13:04:34 ip-10-242-231-63 fuse_dfs: mounting dfs://ec2-75-101-210-65.compute-1.amazonaws.com:9000/
>
> mkdir and touch works. But I cannot write data into files(?!). They are all read only.
> When I try to copy files "from outside" into the HDFS, only an empty file is created
and in user.log appear these error messages:
>
> Apr 23 13:18:46 ip-10-242-231-63 fuse_dfs: ERROR: fuse problem - could not write all
the bytes for /testordner/testfile -1!=4096fuse_impls_write.c:60
> Apr 23 13:18:46 ip-10-242-231-63 fuse_dfs: WARN: fuse problem - could not write all the
bytes for /testordner/testfile -1!=4096fuse_impls_write.c:64
> Apr 23 13:18:46 ip-10-242-231-63 fuse_dfs: ERROR: fuse problem - could not write all
the bytes for /testordner/testfile -1!=4096fuse_impls_write.c:60
> Apr 23 13:18:46 ip-10-242-231-63 fuse_dfs: WARN: fuse problem - could not write all the
bytes for /testordner/testfile -1!=4096fuse_impls_write.c:64
> Apr 23 13:18:46 ip-10-242-231-63 fuse_dfs: ERROR: dfs problem - could not close file_handle(23486496)
for /testordner/testfile fuse_impls_release.c:58
>
> Weird...
>
> But this is a big step forward.
>
> Thanks a lot!!!
>
> Best Regards
>    Christian
>
>
> Am Freitag, 23. April 2010 schrieb Brian Bockelman:
>> Hm, ok, now you have me stumped.
>>
>> One last hunch - can you include the port information, but also switch to port 9000?
>>
>> Additionally, can you do the following:
>>
>> 1) In /var/log/messages and copy out the hdfs/fuse-related messages and post them
>> 2) Using the hadoop clients do,
>> hadoop fs -ls /
>>
>> Brian
>>
>> On Apr 23, 2010, at 12:33 AM, Christian Baun wrote:
>>
>> > Hi,
>> >
>> > When adding the port information inside core-site.xml, the problem remains:
>> >
>> >     <property>
>> >             <name>fs.default.name</name>
>> >             <value>hdfs://ec2-75-101-210-65.compute-1.amazonaws.com:8020</value>
>> >             <final>true</final>
>> >     </property>
>> >
>> > # ./fuse_dfs_wrapper.sh dfs://ec2-75-101-210-65.compute-1.amazonaws.com:8020
/mnt/hdfs/
>> > port=8020,server=ec2-75-101-210-65.compute-1.amazonaws.com
>> > fuse-dfs didn't recognize /mnt/hdfs/,-2
>> >
>> > # ls /mnt/hdfs
>> > ls: cannot access /mnt/hdfs/®1  : No such file or directory
>> >
>> > Best Regards,
>> >   Christian
>> >
>> >
>> > Am Freitag, 23. April 2010 schrieb Christian Baun:
>> >> Hi Brian,
>> >>
>> >> this is inside my core-site.xml
>> >>
>> >> <configuration>
>> >>    <property>
>> >>            <name>fs.default.name</name>
>> >>            <value>hdfs://ec2-75-101-210-65.compute-1.amazonaws.com/</value>
>> >>            <final>true</final>
>> >>    </property>
>> >>    <property>
>> >>            <name>hadoop.tmp.dir</name>
>> >>            <value>/mnt</value>
>> >>            <description>A base for other temporary directories.</description>
>> >>    </property>
>> >> </configuration>
>> >>
>> >> Do I need to give the port here?
>> >>
>> >> this is inside my hdfs-site.xml
>> >>
>> >> <configuration>
>> >>    <property>
>> >>            <name>dfs.name.dir</name>
>> >>            <value>${hadoop.tmp.dir}/dfs/name</value>
>> >>            <final>true</final>
>> >>    </property>
>> >>    <property>
>> >>            <name>dfs.data.dir</name>
>> >>            <value>${hadoop.tmp.dir}/dfs/data</value>
>> >>
>> >> </property>
>> >>    <property>
>> >>            <name>fs.checkpoint.dir</name>
>> >>            <value>${hadoop.tmp.dir}/dfs/namesecondary</value>
>> >>            <final>true</final>
>> >>            <final>true</final>
>> >>    </property>
>> >> </configuration>
>> >>
>> >> These directories do all exist
>> >>
>> >> # ls -l /mnt/dfs/
>> >> total 12
>> >> drwxr-xr-x 2 hadoop hadoop 4096 2010-04-23 05:08 data
>> >> drwxr-xr-x 4 hadoop hadoop 4096 2010-04-23 05:17 name
>> >> drwxr-xr-x 2 hadoop hadoop 4096 2010-04-23 05:08 namesecondary
>> >>
>> >> I don't have the config file hadoop-site.xml in /etc/...
>> >> In the source directory of hadoop I have a hadoop-site.xml but with this
information
>> >>
>> >> <!-- DO NOT PUT ANY PROPERTY IN THIS FILE. INSTEAD USE -->
>> >> <!-- core-site.xml, mapred-site.xml OR hdfs-site.xml -->
>> >> <!-- This empty script is to avoid picking properties from  -->
>> >> <!-- conf/hadoop-site.xml This would be removed once support  -->
>> >> <!-- for hadoop-site.xml is removed.  -->
>> >>
>> >> Best Regards,
>> >>   Christian
>> >>
>> >>
>> >>
>> >> Am Freitag, 23. April 2010 schrieb Brian Bockelman:
>> >>> Hey Christian,
>> >>>
>> >>> I've run into this before.
>> >>>
>> >>> Make sure that the hostname/port you give to fuse is EXACTLY the same
as listed in hadoop-site.xml.
>> >>>
>> >>> If these aren't the same text string (including the ":8020"), then you
get those sort of issues.
>> >>>
>> >>> Brian
>> >>>
>> >>> On Apr 22, 2010, at 5:00 AM, Christian Baun wrote:
>> >>>
>> >>>> Dear All,
>> >>>>
>> >>>> I want to test HDFS inside Amazon EC2.
>> >>>>
>> >>>> Two Ubuntu instances are running inside EC2.
>> >>>> One server is namenode and jobtracker. The other server is the datanode.
>> >>>> Cloudera (hadoop-0.20) is installed and running.
>> >>>>
>> >>>> Now, I want to mount HDFS.
>> >>>> I tried to install contrib/fuse-dfs as described here:
>> >>>> http://wiki.apache.org/hadoop/MountableHDFS
>> >>>>
>> >>>> The compilation worked via:
>> >>>>
>> >>>> # ant compile-c++-libhdfs -Dlibhdfs=1
>> >>>> # ant package -Djava5.home=/usr/lib/jvm/java-1.5.0-sun-1.5.0.06/
-Dforrest.home=/home/ubuntu/apache-forrest-0.8/
>> >>>> # ant compile-contrib -Dlibhdfs=1 -Dfusedfs=1
>> >>>>
>> >>>> But now, when I try to mount the filesystem:
>> >>>>
>> >>>> # ./fuse_dfs_wrapper.sh dfs://ec2-75-101-210-65.compute-1.amazonaws.com:8020
/mnt/hdfs/ -d
>> >>>> port=8020,server=ec2-75-101-210-65.compute-1.amazonaws.com
>> >>>> fuse-dfs didn't recognize /mnt/hdfs/,-2
>> >>>> fuse-dfs ignoring option -d
>> >>>> FUSE library version: 2.8.1
>> >>>> nullpath_ok: 0
>> >>>> unique: 1, opcode: INIT (26), nodeid: 0, insize: 56
>> >>>> INIT: 7.13
>> >>>> flags=0x0000007b
>> >>>> max_readahead=0x00020000
>> >>>>  INIT: 7.12
>> >>>>  flags=0x00000011
>> >>>>  max_readahead=0x00020000
>> >>>>  max_write=0x00020000
>> >>>>  unique: 1, success, outsize: 40
>> >>>>
>> >>>>
>> >>>> # ./fuse_dfs_wrapper.sh dfs://ec2-75-101-210-65.compute-1.amazonaws.com:8020
/mnt/hdfs/
>> >>>> port=8020,server=ec2-75-101-210-65.compute-1.amazonaws.com
>> >>>> fuse-dfs didn't recognize /mnt/hdfs/,-2
>> >>>>
>> >>>> # ls /mnt/hdfs/
>> >>>> ls: reading directory /mnt/hdfs/: Input/output error
>> >>>> # ls /mnt/hdfs/
>> >>>> ls: cannot access /mnt/hdfs/o¢  : No such file or directory
>> >>>> o???
>> >>>> # ls /mnt/hdfs/
>> >>>> ls: reading directory /mnt/hdfs/: Input/output error
>> >>>> # ls /mnt/hdfs/
>> >>>> ls: cannot access /mnt/hdfs/`á Óÿ : No such file or directory
>> >>>> `?????
>> >>>> # ls /mnt/hdfs/
>> >>>> ls: reading directory /mnt/hdfs/: Input/output error
>> >>>> ...
>> >>>>
>> >>>>
>> >>>> What can I do at this point?
>> >>>>
>> >>>> Thanks in advance
>> >>>>    Christian
>> >>>
>> >>>
>> >>
>> >>
>> >
>>
>>
>
>
>

Mime
View raw message