hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7358) Clients may get stuck waiting when using ByteArrayManager
Date Tue, 11 Nov 2014 06:57:35 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14206049#comment-14206049
] 

stack commented on HDFS-7358:
-----------------------------

bq. So the synchronization is required.

What of Packet#writeTo did the buffer release?

bq. We already has close(), which is the public user API for closing the stream.

Ok.

Here's a few numbers:

With the feature turned off:

||threads||seconds||ops/second||
|10|133.662|7481.558|
|10|133.599|7485.086|
|10|134.046|7460.125|
|20|140.972|14187.215|
|20|141.949|14089.566|
|20|140.861|14198.395|
|100|153.941|64959.953|
|100|153.751|65040.223|
|100|153.372|65200.953|

With version of patch that does NOT synchronize Packet:

||threads||seconds||ops/second||
|10|126.219|7922.737|
|10|127.792|7825.216|
|10|124.829|8010.959|
|20|138.132|14478.904|
|20|137.051|14593.108|
|20|139.604|14326.236|
|100|149.311|66974.297|
|100|149.849|66733.844|
|100|149.537|66873.078|

Here is latest patch numbers:

||threads||seconds||ops/second||
|10|127.079|7869.121|
|10|128.357|7790.771|
|10|129.122|7744.614|
|20|135.525|14757.426|
|20|139.531|14333.731|
|20|135.595|14749.807|
|100|149.802|66754.781|
|100|149.262|66996.289|
|100|149.925|66700.016|

Threads in above are client threads. Actual number of writing and syncing threads stays constant
at 1 and 5.  More threads just means more writing per second.

Comparing the last run of 100 threads with the feature off vs the last run of the latest patch
I see more stalls, about same instructions per cycle but less cycles so it comes out a bit
better.

Perf summary on unpatched run:
{code}
 Performance counter stats for '/home/stack/hbase/bin/hbase --config /home/stack/conf_hbase
org.apache.hadoop.hbase.regionserver.wal.HLogPerformanceEvaluation -threads 100 -iterations
100000 -keySize 50 -valueSize 100':

     587172.254075 task-clock                #    3.666 CPUs utilized
        18,700,961 context-switches          #    0.032 M/sec
         4,596,456 CPU-migrations            #    0.008 M/sec
           650,547 page-faults               #    0.001 M/sec
   891,035,644,874 cycles                    #    1.518 GHz                     [83.31%]
   674,789,502,548 stalled-cycles-frontend   #   75.73% frontend cycles idle    [83.32%]
   400,621,650,589 stalled-cycles-backend    #   44.96% backend  cycles idle    [66.74%]
   422,912,592,386 instructions              #    0.47  insns per cycle
                                             #    1.60  stalled cycles per insn [83.41%]
    78,498,471,337 branches                  #  133.689 M/sec                   [83.37%]
     2,768,724,048 branch-misses             #    3.53% of all branches         [83.26%]

     160.168742742 seconds time elapsed
{code}

Here is patched version perf output.
{code}
 Performance counter stats for '/home/stack/hbase/bin/hbase --config /home/stack/conf_hbase
org.apache.hadoop.hbase.regionserver.wal.HLogPerformanceEvaluation -threads 100 -iterations
100000 -keySize 50 -valueSize 100':

     556038.390042 task-clock                #    3.550 CPUs utilized
        18,699,748 context-switches          #    0.034 M/sec
         4,534,830 CPU-migrations            #    0.008 M/sec
           636,724 page-faults               #    0.001 M/sec
   843,860,285,154 cycles                    #    1.518 GHz                     [83.29%]
   642,851,753,015 stalled-cycles-frontend   #   76.18% frontend cycles idle    [83.34%]
   384,260,620,446 stalled-cycles-backend    #   45.54% backend  cycles idle    [66.66%]
   392,462,867,299 instructions              #    0.47  insns per cycle
                                             #    1.64  stalled cycles per insn [83.36%]
    71,358,339,182 branches                  #  128.333 M/sec                   [83.43%]
     2,712,426,902 branch-misses             #    3.80% of all branches         [83.29%]

     156.646653202 seconds time elapsed
{code}



> Clients may get stuck waiting when using ByteArrayManager
> ---------------------------------------------------------
>
>                 Key: HDFS-7358
>                 URL: https://issues.apache.org/jira/browse/HDFS-7358
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: hdfs-client
>            Reporter: Tsz Wo Nicholas Sze
>            Assignee: Tsz Wo Nicholas Sze
>         Attachments: h7358_20141104.patch, h7358_20141104_wait_timeout.patch, h7358_20141105.patch,
h7358_20141106.patch, h7358_20141107.patch, h7358_20141108.patch
>
>
> [~stack] reported that clients might get stuck waiting when using ByteArrayManager; see
[his comments|https://issues.apache.org/jira/browse/HDFS-7276?focusedCommentId=14197036&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14197036].



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message