hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: could only be replicated to 0 nodes, instead of 1
Date Fri, 15 Jul 2011 11:01:51 GMT
(P.s. I asked that cause if you look at your NN's live nodes tables,
the reported space is all 0)

What's the output of:

du -sk /tmp/hadoop-user/dfs on all your DNs?

On Fri, Jul 15, 2011 at 4:01 PM, Harsh J <harsh@cloudera.com> wrote:
> Thomas,
>
> Is your /tmp/ mount point also under the / or is it separate? Your
> dfs.data.dir are /tmp/hadoop-user/dfs/data in all DNs, and if they are
> separately mounted then what's the available space on that?
>
> (bad idea in production to keep things default on /tmp though, like
> dfs.name.dir, dfs.data.dir -- reconfigure+restart as necessary)
>
> On Fri, Jul 15, 2011 at 3:47 PM, Thomas Anderson
> <t.dt.aanderson@gmail.com> wrote:
>> 1.) The disk usage (with df -kh) on namenode (server01)
>>
>> Filesystem            Size  Used Avail Use% Mounted on
>> /dev/sda1             9.4G  2.3G  6.7G  25% /
>>
>> and datanodes (server02 ~ server05)
>> /dev/sda1             9.4G  2.2G  6.8G  25% /
>> /dev/sda1             9.4G  2.2G  6.8G  25% /
>> /dev/sda1             9.4G  2.2G  6.8G  25% /
>> /dev/sda1             9.4G  2.2G  6.8G  25% /
>>
>> 2.) How can I make sure that datanode is busy? The environment is only
>> for testing so there is no other user processes are running at that
>> moment. Also it is a fresh installation, so only hadoop required
>> packages are installed such as hadoop and jdk.
>>
>> 3.) fs.block.size is not set in hdfs-site.xml, including datanodes and
>> namenode, because its purpose is for testing. I thought it would use
>> the default value, which should be 512?
>>
>> 4.) What might be a good way for fast check if network is not stable?
>> I check the healthy page e.g. server01:50070/dfshealth.jsp where
>> livenodes are up and  last contact varies when checking the page.
>>
>> Node     Last Contact    Admin State     Configured  Capacity (GB)      
Used
>> (GB)     Non DFS  Used (GB)      Remaining  (GB)         Used  (%)  
    Used  (%)
>> Remaining  (%)   Blocks
>> server02         2      In Service      0.1     0       0      
0.1     0.01     99.96  0
>> server03         0      In Service      0.1     0       0      
0.1     0.01     99.96  0
>> server04         1      In Service      0.1     0       0      
0.1     0.01     99.96  0
>> server05         2      In Service      0.1     0       0      
0.1     0.01     99.96  0
>>
>> 5.) Only command `hadoop fs -put /tmp/testfile test` is issued as it
>> is just to test if the installation is working. So the file e.g.
>> testfile will be removed first (hadoop fs -rm test/testfile), then
>> upload again with hadoop put command.
>>
>> The logs are listed as below:
>>
>> namenode:
>> server01: http://pastebin.com/TLpDmmPx
>>
>> datanodes:
>> server02: http://pastebin.com/pdE5XKfi
>> server03: http://pastebin.com/4aV7ECCV
>> server04: http://pastebin.com/tF7HiRZj
>> server05: http://pastebin.com/5qwSPrvU
>>
>> Please let me know if more information needs to be provided.
>>
>> I really appreciate your suggestion.
>>
>> Thank you.
>>
>>
>> On Fri, Jul 15, 2011 at 4:54 PM, Brahma Reddy <brahmareddyb@huawei.com> wrote:
>>> Hi,
>>>
>>> By seeing this exception(could only be replicated to 0 nodes, instead of 1)
>>> ,datanode is not available to Name Node..
>>>
>>> This are the following cases Data Node may not available to Name Node
>>>
>>> 1)Data Node disk is Full
>>>
>>> 2)Data Node is Busy with block report and block scanning
>>>
>>> 3)If Block Size is Negative value(dfs.block.size in hdfs-site.xml)
>>>
>>> 4)while write in progress primary datanode goes down(Any n/w fluctations b/w
>>> Name Node and Data Node Machines)
>>>
>>> 5)when Ever we append any partial chunk and call sync for subsequent partial
>>> chunk appends client should store the previous data in buffer.
>>>
>>> For example after appending "a" I have called sync and when I am trying the
>>> to append the buffer should have "ab"
>>>
>>> And Server side when the chunk is not multiple of 512 then it will try to do
>>> Crc comparison for the data present in block file as well as crc present in
>>> metafile. But while constructing crc for the data present in block it is
>>> always comparing till the initial Offeset
>>>
>>> Or For more analysis Please the data node logs
>>>
>>> Warm Regards
>>>
>>> Brahma Reddy
>>>
>>> ****************************************************************************
>>> ***********
>>> This e-mail and attachments contain confidential information from HUAWEI,
>>> which is intended only for the person or entity whose address is listed
>>> above. Any use of the information contained herein in any way (including,
>>> but not limited to, total or partial disclosure, reproduction, or
>>> dissemination) by persons other than the intended recipient's) is
>>> prohibited. If you receive this e-mail in error, please notify the sender by
>>> phone or email immediately and delete it!
>>> -----Original Message-----
>>> From: Thomas Anderson [mailto:t.dt.aanderson@gmail.com]
>>> Sent: Friday, July 15, 2011 9:09 AM
>>> To: hdfs-user@hadoop.apache.org
>>> Subject: could only be replicated to 0 nodes, instead of 1
>>>
>>> I have fresh  hadoop 0.20.2 installed on virtualbox 4.0.8 with jdk
>>> 1.6.0_26. The problem is when trying to put a file to hdfs, it throws
>>> error `org.apache.hadoop.ipc.RemoteException: java.io.IOException:
>>> File /path/to/file could only be replicated to 0 nodes, instead of 1';
>>> however, there is no problem to create a folder, as the command ls
>>> print the result
>>>
>>> Found 1 items
>>> drwxr-xr-x   - user supergroup          0 2011-07-15 11:09 /user/user/test
>>>
>>> I also try with flushing firewall (remove all iptables restriction),
>>> but the error message is still thrown out when uploading (hadoop fs
>>> -put /tmp/x test) a file from local fs.
>>>
>>> The name node log shows
>>>
>>> 2011-07-15 10:42:43,491 INFO org.apache.hadoop.hdfs.StateChange:
>>> BLOCK* NameSystem.registerDatanode: node registration from
>>> aaa.bbb.ccc.ddd.22:50010 storage DS-929017105-aaa.bbb.ccc.22-50010-13
>>> 10697763488
>>> 2011-07-15 10:42:43,495 INFO org.apache.hadoop.net.NetworkTopology:
>>> Adding a new node: /default-rack/aaa.bbb.ccc.22:50010
>>> 2011-07-15 10:42:44,169 INFO org.apache.hadoop.hdfs.StateChange:
>>> BLOCK* NameSystem.registerDatanode: node registration from
>>> aaa.bbb.ccc.35:50010 storage DS-884574392-aaa.bbb.ccc.35-50010-13
>>> 10697764164
>>> 2011-07-15 10:42:44,170 INFO org.apache.hadoop.net.NetworkTopology:
>>> Adding a new node: /default-rack/aaa.bbb.ccc.35:50010
>>> 2011-07-15 10:42:44,507 INFO org.apache.hadoop.hdfs.StateChange:
>>> BLOCK* NameSystem.registerDatanode: node registration from
>>> aaa.bbb.ccc.ddd.11:50010 storage DS-1537583073-aaa.bbb.ccc.11-50010-1
>>> 310697764488
>>> 2011-07-15 10:42:44,507 INFO org.apache.hadoop.net.NetworkTopology:
>>> Adding a new node: /default-rack/aaa.bbb.ccc.11:50010
>>> 2011-07-15 10:42:45,796 INFO org.apache.hadoop.hdfs.StateChange:
>>> BLOCK* NameSystem.registerDatanode: node registration from
>>> 140.127.220.25:50010 storage DS-1500589162-aaa.bbb.ccc.25-50010-1
>>> 310697765386
>>> 2011-07-15 10:42:45,797 INFO org.apache.hadoop.net.NetworkTopology:
>>> Adding a new node: /default-rack/aaa.bbb.ccc.25:50010
>>>
>>> And all datanodes have similar message as below:
>>>
>>> 2011-07-15 10:42:46,562 INFO
>>> org.apache.hadoop.hdfs.server.datanode.DataNode: using
>>> BLOCKREPORT_INTERVAL of 3600000msec Initial delay: 0msec
>>> 2011-07-15 10:42:47,163 INFO
>>> org.apache.hadoop.hdfs.server.datanode.DataNode: BlockReport of 0
>>> blocks got processed in 3 msecs
>>> 2011-07-15 10:42:47,187 INFO
>>> org.apache.hadoop.hdfs.server.datanode.DataNode: Starting Periodic
>>> block scanner.
>>> 2011-07-15 11:19:42,931 INFO
>>> org.apache.hadoop.hdfs.server.datanode.DataNode: BlockReport of 0
>>> blocks got processed in 1 msecs
>>>
>>> Command `hadoop fsck /`  displays
>>>
>>> Status: HEALTHY
>>>  Total size:    0 B
>>>  Total dirs:    3
>>>  Total files:   0 (Files currently being written: 1)
>>>  Total blocks (validated):      0
>>>  Minimally replicated blocks:   0
>>>  Over-replicated blocks:        0
>>>  Under-replicated blocks:       0
>>>  Mis-replicated blocks:         0
>>>  Default replication factor:    3
>>>  Average block replication:     0.0
>>>  Corrupt blocks:                0
>>>  Missing replicas:              0
>>>  Number of data-nodes:          4
>>>
>>> The setting in conf include:
>>>
>>> - Master node:
>>> core-site.xml
>>>  <property>
>>>    <name>fs.default.name</name>
>>>    <value>hdfs://lab01:9000/</value>
>>>  </property>
>>>
>>> hdfs-site.xml
>>>  <property>
>>>    <name>dfs.replication</name>
>>>    <value>3</value>
>>>  </property>
>>>
>>> -Slave nodes:
>>> core-site.xml
>>>  <property>
>>>    <name>fs.default.name</name>
>>>    <value>hdfs://lab01:9000/</value>
>>>  </property>
>>>
>>> hdfs-site.xml
>>>  <property>
>>>    <name>dfs.replication</name>
>>>    <value>3</value>
>>>  </property>
>>>
>>> Do I missing any configuration? Or any place that I can check?
>>>
>>> Thanks.
>>>
>>>
>>
>
>
>
> --
> Harsh J
>



-- 
Harsh J

Mime
View raw message