hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-4515) ReplicaMap thread-safe synchronization lead to a large number of threads blocking
Date Tue, 19 Feb 2013 14:17:12 GMT

    [ https://issues.apache.org/jira/browse/HDFS-4515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13581301#comment-13581301
] 

Steve Loughran commented on HDFS-4515:
--------------------------------------

Do have any plans to publish the tests? I've been thinking about having more FS scale tests
-load simulation should be part of this
                
> ReplicaMap thread-safe synchronization lead to a large number of threads blocking
> ---------------------------------------------------------------------------------
>
>                 Key: HDFS-4515
>                 URL: https://issues.apache.org/jira/browse/HDFS-4515
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: datanode
>    Affects Versions: 2.0.3-alpha
>         Environment: CentOS release 6.3
>            Reporter: WenJin Ma
>         Attachments: java3.log, java.log, logs1.txt
>
>   Original Estimate: 612h
>  Remaining Estimate: 612h
>
> 1.I use programs simulate 3000 users to write 100K small files in HDFS。(3 datanodes)
> 2.Found that after a period of time the client and datanodes a lot of timeout errors
or errors caused by the socket timeout
> example:log1.txt
> 3.dump datanode java stack log(view file:java.log)
> cat java.log |grep BLOCKED|wc -l
> 2635
> A large number of threads blocking in ReplicaMap, because there used synchronize guarantee
thread safety
> {code}
> "DataXceiver for client DFSClient_NONMAPREDUCE_-1528766469_27 at /10.28.171.254:59064
[Receiving block BP-560172827-10.28.171.226-1360119691522:blk_4662142724658079555_330234]"
daemon prio=10 tid=0x00000000428d2800 nid=0x9e32 waiting for monitor entry [0x00007f20ea56d000]
>    java.lang.Thread.State: BLOCKED (on object monitor)
>         at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:670)
>         - waiting to lock <0x00000000f81c05d0> (a org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl)
>         at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.createRbw(FsDatasetImpl.java:89)
>         at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:159)
>         at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:393)
>         at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:98)
>         at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:66)
>         at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:219)
>         at java.lang.Thread.run(Thread.java:662)
> {code}
> Linux users resources may be exhausted after running for some time.
> HDFS this realization in our test environment can only guarantee stable operation in
800 users.
> 4. I will be synchronized mechanism is changed to read-write locks, and try to run in
the 4500 user does not appear a large number of thread blocks.
> 5. dump java stack log(view file java3.log).
> cat java3.log |grep BLOCKED|wc -l
> 0

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message