hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "WenJin Ma (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-4515) ReplicaMap thread-safe synchronization lead to a large number of threads blocking
Date Tue, 19 Feb 2013 03:17:14 GMT
WenJin Ma created HDFS-4515:
-------------------------------

             Summary: ReplicaMap thread-safe synchronization lead to a large number of threads
blocking
                 Key: HDFS-4515
                 URL: https://issues.apache.org/jira/browse/HDFS-4515
             Project: Hadoop HDFS
          Issue Type: Bug
          Components: datanode
    Affects Versions: 2.0.3-alpha
         Environment: CentOS release 6.3
            Reporter: WenJin Ma


1.I use programs simulate 3000 users to write 100K small files in HDFS。(3 datanodes)

2.Found that after a period of time the client and datanodes a lot of timeout errors or errors
caused by the socket timeout
example:log1.txt

3.dump datanode java stack log(view file:java.log)
cat java.log |grep BLOCKED|wc -l
2635

A large number of threads blocking in ReplicaMap, because there used synchronize guarantee
thread safety
{code}
"DataXceiver for client DFSClient_NONMAPREDUCE_-1528766469_27 at /10.28.171.254:59064 [Receiving
block BP-560172827-10.28.171.226-1360119691522:blk_4662142724658079555_330234]" daemon prio=10
tid=0x00000000428d2800 nid=0x9e32 waiting for monitor entry [0x00007f20ea56d000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:670)
        - waiting to lock <0x00000000f81c05d0> (a org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl)
        at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:89)
        at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:159)
        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:393)
        at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:98)
        at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:66)
        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:219)
        at java.lang.Thread.run(Thread.java:662)
{code}
Linux users resources may be exhausted after running for some time.

HDFS this realization in our test environment can only guarantee stable operation in 800 users.

4. I will be synchronized mechanism is changed to read-write locks, and try to run in the
4500 user does not appear a large number of thread blocks.

5. dump java stack log(view file java3.log).

cat java3.log |grep BLOCKED|wc -l
0

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message