hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-797) TestHDFSCLI much slower after HDFS-265 merge
Date Tue, 01 Dec 2009 00:16:20 GMT

    [ https://issues.apache.org/jira/browse/HDFS-797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12783957#action_12783957
] 

Todd Lipcon commented on HDFS-797:
----------------------------------

Strange. It reproducibly takes 12-15 minutes on my laptop (Linux, Sun JDK 1.6.0_16) with a
clean checkout. On one of our build boxes (Linux, JDK 1.6.0_14) it takes around 2 minutes
like it's supposed to. "Poor man's profiling" of jstacking every few seconds shows most of
the traces like this:

"main" prio=10 tid=0x0000000040d83800 nid=0x147b in Object.wait() [0x00007f738eeea000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00007f7381623b30> (a org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer)
        at java.lang.Thread.join(Thread.java:1143)
        - locked <0x00007f7381623b30> (a org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer)
        at java.lang.Thread.join(Thread.java:1196)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeThreads(DFSClient.java:3639)
        at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.close(DFSClient.java:3683)
        - locked <0x00007f7381622e58> (a org.apache.hadoop.hdfs.DFSClient$DFSOutputStream)
        at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:61)
        at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:86)
        at org.apache.hadoop.fs.FsShell.touchz(FsShell.java:818)

Can anyone else reproduce?

> TestHDFSCLI much slower after HDFS-265 merge
> --------------------------------------------
>
>                 Key: HDFS-797
>                 URL: https://issues.apache.org/jira/browse/HDFS-797
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: hdfs client
>    Affects Versions: 0.21.0, 0.22.0
>            Reporter: Todd Lipcon
>
> After the merge of HDFS-265 into trunk, TestHDFSCLI takes much longer to run (>10min,
got tired of waiting). Changing dataQueue.wait(1000) to dataQueue.wait(10) on line 2708 speeds
up the test back to its original performance. This isn't a fix, but displays the issue - somehow
the thread isn't getting properly interrupted. In general this probably represents a performance
regression for the exit of all HDFS writers (most noticeable in CLI tools)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message