hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thomas Anderson <t.dt.aander...@gmail.com>
Subject Re: could only be replicated to 0 nodes, instead of 1
Date Fri, 15 Jul 2011 11:10:40 GMT
When doing partition, I remember only / and swap was specified for all
nodes during creation. So I think /tmp is also mounted under /, which
should have size around 9G. The total size of hardisk specified is
10G.

The df -kh shows

server01:
/dev/sda1             9.4G  2.3G  6.7G  25% /
tmpfs                 5.0M  4.0K  5.0M   1% /lib/init/rw
tmpfs                 5.0M     0  5.0M   0% /var/run/lock
tmpfs                 101M  132K  101M   1% /tmp
udev                  247M     0  247M   0% /dev
tmpfs                 101M     0  101M   0% /var/run/shm
tmpfs                  51M  176K   51M   1% /var/run

server02:
/dev/sda1             9.4G  2.2G  6.8G  25% /
tmpfs                 5.0M  4.0K  5.0M   1% /lib/init/rw
tmpfs                 5.0M     0  5.0M   0% /var/run/lock
tmpfs                 101M   44K  101M   1% /tmp
udev                  247M     0  247M   0% /dev
tmpfs                 101M     0  101M   0% /var/run/shm
tmpfs                  51M  176K   51M   1% /var/run

server03:
/dev/sda1             9.4G  2.2G  6.8G  25% /
tmpfs                 5.0M  4.0K  5.0M   1% /lib/init/rw
tmpfs                 5.0M     0  5.0M   0% /var/run/lock
tmpfs                 101M   44K  101M   1% /tmp
udev                  247M     0  247M   0% /dev
tmpfs                 101M     0  101M   0% /var/run/shm
tmpfs                  51M  176K   51M   1% /var/run

server04:
/dev/sda1             9.4G  2.2G  6.8G  25% /
tmpfs                 5.0M  4.0K  5.0M   1% /lib/init/rw
tmpfs                 5.0M     0  5.0M   0% /var/run/lock
tmpfs                 101M   44K  101M   1% /tmp
udev                  247M     0  247M   0% /dev
tmpfs                 101M     0  101M   0% /var/run/shm
tmpfs                  51M  176K   51M   1% /var/run

server05:
/dev/sda1             9.4G  2.2G  6.8G  25% /
tmpfs                 5.0M  4.0K  5.0M   1% /lib/init/rw
tmpfs                 5.0M     0  5.0M   0% /var/run/lock
tmpfs                 101M   44K  101M   1% /tmp
udev                  247M     0  247M   0% /dev
tmpfs                 101M     0  101M   0% /var/run/shm
tmpfs                  51M  176K   51M   1% /var/run

In addition, the output of dfs (du -sk /tmp/hadoop-user/dfs) is

server02:
8	/tmp/hadoop-user/dfs/

server03:
8	/tmp/hadoop-user/dfs/

server04:
8	/tmp/hadoop-user/dfs/

server05:
8	/tmp/hadoop-user/dfs/

On Fri, Jul 15, 2011 at 7:01 PM, Harsh J <harsh@cloudera.com> wrote:
> (P.s. I asked that cause if you look at your NN's live nodes tables,
> the reported space is all 0)
>
> What's the output of:
>
> du -sk /tmp/hadoop-user/dfs on all your DNs?
>
> On Fri, Jul 15, 2011 at 4:01 PM, Harsh J <harsh@cloudera.com> wrote:
>> Thomas,
>>
>> Is your /tmp/ mount point also under the / or is it separate? Your
>> dfs.data.dir are /tmp/hadoop-user/dfs/data in all DNs, and if they are
>> separately mounted then what's the available space on that?
>>
>> (bad idea in production to keep things default on /tmp though, like
>> dfs.name.dir, dfs.data.dir -- reconfigure+restart as necessary)
>>
>> On Fri, Jul 15, 2011 at 3:47 PM, Thomas Anderson
>> <t.dt.aanderson@gmail.com> wrote:
>>> 1.) The disk usage (with df -kh) on namenode (server01)
>>>
>>> Filesystem            Size  Used Avail Use% Mounted on
>>> /dev/sda1             9.4G  2.3G  6.7G  25% /
>>>
>>> and datanodes (server02 ~ server05)
>>> /dev/sda1             9.4G  2.2G  6.8G  25% /
>>> /dev/sda1             9.4G  2.2G  6.8G  25% /
>>> /dev/sda1             9.4G  2.2G  6.8G  25% /
>>> /dev/sda1             9.4G  2.2G  6.8G  25% /
>>>
>>> 2.) How can I make sure that datanode is busy? The environment is only
>>> for testing so there is no other user processes are running at that
>>> moment. Also it is a fresh installation, so only hadoop required
>>> packages are installed such as hadoop and jdk.
>>>
>>> 3.) fs.block.size is not set in hdfs-site.xml, including datanodes and
>>> namenode, because its purpose is for testing. I thought it would use
>>> the default value, which should be 512?
>>>
>>> 4.) What might be a good way for fast check if network is not stable?
>>> I check the healthy page e.g. server01:50070/dfshealth.jsp where
>>> livenodes are up and  last contact varies when checking the page.
>>>
>>> Node     Last Contact    Admin State     Configured  Capacity (GB)  
    Used
>>> (GB)     Non DFS  Used (GB)      Remaining  (GB)         Used  (%)
      Used  (%)
>>> Remaining  (%)   Blocks
>>> server02         2      In Service      0.1     0       0    
  0.1     0.01     99.96  0
>>> server03         0      In Service      0.1     0       0    
  0.1     0.01     99.96  0
>>> server04         1      In Service      0.1     0       0    
  0.1     0.01     99.96  0
>>> server05         2      In Service      0.1     0       0    
  0.1     0.01     99.96  0
>>>
>>> 5.) Only command `hadoop fs -put /tmp/testfile test` is issued as it
>>> is just to test if the installation is working. So the file e.g.
>>> testfile will be removed first (hadoop fs -rm test/testfile), then
>>> upload again with hadoop put command.
>>>
>>> The logs are listed as below:
>>>
>>> namenode:
>>> server01: http://pastebin.com/TLpDmmPx
>>>
>>> datanodes:
>>> server02: http://pastebin.com/pdE5XKfi
>>> server03: http://pastebin.com/4aV7ECCV
>>> server04: http://pastebin.com/tF7HiRZj
>>> server05: http://pastebin.com/5qwSPrvU
>>>
>>> Please let me know if more information needs to be provided.
>>>
>>> I really appreciate your suggestion.
>>>
>>> Thank you.
>>>
>>>
>>> On Fri, Jul 15, 2011 at 4:54 PM, Brahma Reddy <brahmareddyb@huawei.com>
wrote:
>>>> Hi,
>>>>
>>>> By seeing this exception(could only be replicated to 0 nodes, instead of
1)
>>>> ,datanode is not available to Name Node..
>>>>
>>>> This are the following cases Data Node may not available to Name Node
>>>>
>>>> 1)Data Node disk is Full
>>>>
>>>> 2)Data Node is Busy with block report and block scanning
>>>>
>>>> 3)If Block Size is Negative value(dfs.block.size in hdfs-site.xml)
>>>>
>>>> 4)while write in progress primary datanode goes down(Any n/w fluctations
b/w
>>>> Name Node and Data Node Machines)
>>>>
>>>> 5)when Ever we append any partial chunk and call sync for subsequent partial
>>>> chunk appends client should store the previous data in buffer.
>>>>
>>>> For example after appending "a" I have called sync and when I am trying the
>>>> to append the buffer should have "ab"
>>>>
>>>> And Server side when the chunk is not multiple of 512 then it will try to
do
>>>> Crc comparison for the data present in block file as well as crc present
in
>>>> metafile. But while constructing crc for the data present in block it is
>>>> always comparing till the initial Offeset
>>>>
>>>> Or For more analysis Please the data node logs
>>>>
>>>> Warm Regards
>>>>
>>>> Brahma Reddy
>>>>
>>>> ****************************************************************************
>>>> ***********
>>>> This e-mail and attachments contain confidential information from HUAWEI,
>>>> which is intended only for the person or entity whose address is listed
>>>> above. Any use of the information contained herein in any way (including,
>>>> but not limited to, total or partial disclosure, reproduction, or
>>>> dissemination) by persons other than the intended recipient's) is
>>>> prohibited. If you receive this e-mail in error, please notify the sender
by
>>>> phone or email immediately and delete it!
>>>> -----Original Message-----
>>>> From: Thomas Anderson [mailto:t.dt.aanderson@gmail.com]
>>>> Sent: Friday, July 15, 2011 9:09 AM
>>>> To: hdfs-user@hadoop.apache.org
>>>> Subject: could only be replicated to 0 nodes, instead of 1
>>>>
>>>> I have fresh  hadoop 0.20.2 installed on virtualbox 4.0.8 with jdk
>>>> 1.6.0_26. The problem is when trying to put a file to hdfs, it throws
>>>> error `org.apache.hadoop.ipc.RemoteException: java.io.IOException:
>>>> File /path/to/file could only be replicated to 0 nodes, instead of 1';
>>>> however, there is no problem to create a folder, as the command ls
>>>> print the result
>>>>
>>>> Found 1 items
>>>> drwxr-xr-x   - user supergroup          0 2011-07-15 11:09 /user/user/test
>>>>
>>>> I also try with flushing firewall (remove all iptables restriction),
>>>> but the error message is still thrown out when uploading (hadoop fs
>>>> -put /tmp/x test) a file from local fs.
>>>>
>>>> The name node log shows
>>>>
>>>> 2011-07-15 10:42:43,491 INFO org.apache.hadoop.hdfs.StateChange:
>>>> BLOCK* NameSystem.registerDatanode: node registration from
>>>> aaa.bbb.ccc.ddd.22:50010 storage DS-929017105-aaa.bbb.ccc.22-50010-13
>>>> 10697763488
>>>> 2011-07-15 10:42:43,495 INFO org.apache.hadoop.net.NetworkTopology:
>>>> Adding a new node: /default-rack/aaa.bbb.ccc.22:50010
>>>> 2011-07-15 10:42:44,169 INFO org.apache.hadoop.hdfs.StateChange:
>>>> BLOCK* NameSystem.registerDatanode: node registration from
>>>> aaa.bbb.ccc.35:50010 storage DS-884574392-aaa.bbb.ccc.35-50010-13
>>>> 10697764164
>>>> 2011-07-15 10:42:44,170 INFO org.apache.hadoop.net.NetworkTopology:
>>>> Adding a new node: /default-rack/aaa.bbb.ccc.35:50010
>>>> 2011-07-15 10:42:44,507 INFO org.apache.hadoop.hdfs.StateChange:
>>>> BLOCK* NameSystem.registerDatanode: node registration from
>>>> aaa.bbb.ccc.ddd.11:50010 storage DS-1537583073-aaa.bbb.ccc.11-50010-1
>>>> 310697764488
>>>> 2011-07-15 10:42:44,507 INFO org.apache.hadoop.net.NetworkTopology:
>>>> Adding a new node: /default-rack/aaa.bbb.ccc.11:50010
>>>> 2011-07-15 10:42:45,796 INFO org.apache.hadoop.hdfs.StateChange:
>>>> BLOCK* NameSystem.registerDatanode: node registration from
>>>> 140.127.220.25:50010 storage DS-1500589162-aaa.bbb.ccc.25-50010-1
>>>> 310697765386
>>>> 2011-07-15 10:42:45,797 INFO org.apache.hadoop.net.NetworkTopology:
>>>> Adding a new node: /default-rack/aaa.bbb.ccc.25:50010
>>>>
>>>> And all datanodes have similar message as below:
>>>>
>>>> 2011-07-15 10:42:46,562 INFO
>>>> org.apache.hadoop.hdfs.server.datanode.DataNode: using
>>>> BLOCKREPORT_INTERVAL of 3600000msec Initial delay: 0msec
>>>> 2011-07-15 10:42:47,163 INFO
>>>> org.apache.hadoop.hdfs.server.datanode.DataNode: BlockReport of 0
>>>> blocks got processed in 3 msecs
>>>> 2011-07-15 10:42:47,187 INFO
>>>> org.apache.hadoop.hdfs.server.datanode.DataNode: Starting Periodic
>>>> block scanner.
>>>> 2011-07-15 11:19:42,931 INFO
>>>> org.apache.hadoop.hdfs.server.datanode.DataNode: BlockReport of 0
>>>> blocks got processed in 1 msecs
>>>>
>>>> Command `hadoop fsck /`  displays
>>>>
>>>> Status: HEALTHY
>>>>  Total size:    0 B
>>>>  Total dirs:    3
>>>>  Total files:   0 (Files currently being written: 1)
>>>>  Total blocks (validated):      0
>>>>  Minimally replicated blocks:   0
>>>>  Over-replicated blocks:        0
>>>>  Under-replicated blocks:       0
>>>>  Mis-replicated blocks:         0
>>>>  Default replication factor:    3
>>>>  Average block replication:     0.0
>>>>  Corrupt blocks:                0
>>>>  Missing replicas:              0
>>>>  Number of data-nodes:          4
>>>>
>>>> The setting in conf include:
>>>>
>>>> - Master node:
>>>> core-site.xml
>>>>  <property>
>>>>    <name>fs.default.name</name>
>>>>    <value>hdfs://lab01:9000/</value>
>>>>  </property>
>>>>
>>>> hdfs-site.xml
>>>>  <property>
>>>>    <name>dfs.replication</name>
>>>>    <value>3</value>
>>>>  </property>
>>>>
>>>> -Slave nodes:
>>>> core-site.xml
>>>>  <property>
>>>>    <name>fs.default.name</name>
>>>>    <value>hdfs://lab01:9000/</value>
>>>>  </property>
>>>>
>>>> hdfs-site.xml
>>>>  <property>
>>>>    <name>dfs.replication</name>
>>>>    <value>3</value>
>>>>  </property>
>>>>
>>>> Do I missing any configuration? Or any place that I can check?
>>>>
>>>> Thanks.
>>>>
>>>>
>>>
>>
>>
>>
>> --
>> Harsh J
>>
>
>
>
> --
> Harsh J
>

Mime
View raw message