cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "graham sanderson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-7546) AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory
Date Mon, 21 Jul 2014 19:06:39 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-7546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14069059#comment-14069059
] 

graham sanderson commented on CASSANDRA-7546:
---------------------------------------------

FYI here are the same synthetic test results for 7546.20_2.txt

{code}
    [junit] --------------------------------------------------
    [junit] 1 THREAD; ELEMENT SIZE 64
    [junit] 
    [junit] Threads = 1 elements = 100000 (of size 64) partitions = 1
    [junit]  original code:
    [junit]   Duration = 993ms maxConcurrency = 1
    [junit]   GC for PS Scavenge: 34 ms for 3 collections
    [junit]   Approx allocation = 553MB vs 8MB; ratio to raw data size = 69.13799428571429
    [junit]   loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0 sync 0/0 up
0 down 0
    [junit] 
    [junit]  modified code: 
    [junit]   Duration = 761ms maxConcurrency = 1
    [junit]   GC for PS Scavenge: 34 ms for 3 collections
    [junit]   Approx allocation = 579MB vs 8MB; ratio to raw data size = 72.31675047619048
    [junit]   loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0 sync 0/0 up
0 down 0
    [junit] 
    [junit] 
    [junit] Threads = 1 elements = 100000 (of size 64) partitions = 16
    [junit]  original code:
    [junit]   Duration = 780ms maxConcurrency = 1
    [junit]   GC for PS Scavenge: 25 ms for 2 collections
    [junit]   Approx allocation = 436MB vs 8MB; ratio to raw data size = 54.48992095238095
    [junit]   loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0 sync 0/0 up
0 down 0
    [junit] 
    [junit]  modified code: 
    [junit]   Duration = 671ms maxConcurrency = 1
    [junit]   GC for PS Scavenge: 24 ms for 2 collections
    [junit]   Approx allocation = 477MB vs 8MB; ratio to raw data size = 59.545997142857146
    [junit]   loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0 sync 0/0 up
0 down 0
    [junit] 
    [junit] 
    [junit] Threads = 1 elements = 100000 (of size 64) partitions = 256
    [junit]  original code:
    [junit]   Duration = 452ms maxConcurrency = 1
    [junit]   GC for PS Scavenge: 11 ms for 1 collections
    [junit]   Approx allocation = 321MB vs 8MB; ratio to raw data size = 40.14510761904762
    [junit]   loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0 sync 0/0 up
0 down 0
    [junit] 
    [junit]  modified code: 
    [junit]   Duration = 460ms maxConcurrency = 1
    [junit]   GC for PS Scavenge: 10 ms for 1 collections
    [junit]   Approx allocation = 341MB vs 8MB; ratio to raw data size = 42.63770857142857
    [junit]   loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0 sync 0/0 up
0 down 0
    [junit] 
    [junit] 
    [junit] Threads = 1 elements = 100000 (of size 64) partitions = 1024
    [junit]  original code:
    [junit]   Duration = 462ms maxConcurrency = 1
    [junit]   GC for PS Scavenge: 14 ms for 1 collections
    [junit]   Approx allocation = 264MB vs 8MB; ratio to raw data size = 32.99879142857143
    [junit]   loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0 sync 0/0 up
0 down 0
    [junit] 
    [junit]  modified code: 
    [junit]   Duration = 543ms maxConcurrency = 1
    [junit]   GC for PS Scavenge: 14 ms for 1 collections
    [junit]   Approx allocation = 272MB vs 8MB; ratio to raw data size = 34.047360952380956
    [junit]   loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0 sync 0/0 up
0 down 0
    [junit] 
    [junit] 
    [junit] --------------------------------------------------
    [junit] 100 THREADS; ELEMENT SIZE 64
    [junit] 
    [junit] Threads = 100 elements = 100000 (of size 64) partitions = 1
    [junit]  original code:
    [junit]   Duration = 2318ms maxConcurrency = 100
    [junit]   GC for PS Scavenge: 119 ms for 32 collections
    [junit]   Approx allocation = 10547MB vs 8MB; ratio to raw data size = 1316.62704
    [junit]   loopRatio (closest to 1 best) 18.35448 raw 100000/1835448 counted 0/0 sync 0/0
up 0 down 0
    [junit] 
    [junit]  modified code: 
    [junit]   Duration = 1315ms maxConcurrency = 100
    [junit]   GC for PS Scavenge: 14 ms for 1 collections
    [junit]   Approx allocation = 629MB vs 8MB; ratio to raw data size = 78.62949142857143
    [junit]   loopRatio (closest to 1 best) 1.11563 raw 13653/13653 counted 0/0 sync 88223/97910
up 0 down 0
    [junit] 
    [junit] 
    [junit] Threads = 100 elements = 100000 (of size 64) partitions = 16
    [junit]  original code:
    [junit]   Duration = 215ms maxConcurrency = 100
    [junit]   GC for PS Scavenge: 23 ms for 2 collections
    [junit]   Approx allocation = 776MB vs 8MB; ratio to raw data size = 96.92138285714286
    [junit]   loopRatio (closest to 1 best) 1.95927 raw 100000/195927 counted 0/0 sync 0/0
up 0 down 0
    [junit] 
    [junit]  modified code: 
    [junit]   Duration = 201ms maxConcurrency = 99
    [junit]   GC for PS Scavenge: 9 ms for 1 collections
    [junit]   Approx allocation = 530MB vs 8MB; ratio to raw data size = 66.21505238095239
    [junit]   loopRatio (closest to 1 best) 1.26719 raw 72820/72820 counted 0/0 sync 49837/53899
up 0 down 0
    [junit] 
    [junit] 
    [junit] Threads = 100 elements = 100000 (of size 64) partitions = 256
    [junit]  original code:
    [junit]   Duration = 187ms maxConcurrency = 96
    [junit]   GC for PS Scavenge: 15 ms for 1 collections
    [junit]   Approx allocation = 319MB vs 8MB; ratio to raw data size = 39.93250857142857
    [junit]   loopRatio (closest to 1 best) 1.01989 raw 100000/101989 counted 0/0 sync 0/0
up 0 down 0
    [junit] 
    [junit]  modified code: 
    [junit]   Duration = 188ms maxConcurrency = 99
    [junit]   GC for PS Scavenge: 12 ms for 1 collections
    [junit]   Approx allocation = 327MB vs 8MB; ratio to raw data size = 40.83917619047619
    [junit]   loopRatio (closest to 1 best) 1.01869 raw 99963/99963 counted 0/0 sync 1877/1906
up 0 down 0
    [junit] 
    [junit] 
    [junit] Threads = 100 elements = 100000 (of size 64) partitions = 1024
    [junit]  original code:
    [junit]   Duration = 169ms maxConcurrency = 98
    [junit]   Approx allocation = 274MB vs 8MB; ratio to raw data size = 34.27162761904762
    [junit]   loopRatio (closest to 1 best) 1.00376 raw 100000/100376 counted 0/0 sync 0/0
up 0 down 0
    [junit] 
    [junit]  modified code: 
    [junit]   Duration = 180ms maxConcurrency = 98
    [junit]   GC for PS Scavenge: 13 ms for 1 collections
    [junit]   Approx allocation = 278MB vs 8MB; ratio to raw data size = 34.80582571428572
    [junit]   loopRatio (closest to 1 best) 1.00348 raw 99999/99999 counted 0/0 sync 349/349
up 0 down 0
    [junit] 
    [junit] 
    [junit] --------------------------------------------------
    [junit] 1 THREAD; ELEMENT SIZE 256
    [junit] 
    [junit] Threads = 1 elements = 100000 (of size 256) partitions = 1
    [junit]  original code:
    [junit]   Duration = 735ms maxConcurrency = 1
    [junit]   GC for PS Scavenge: 26 ms for 2 collections
    [junit]   Approx allocation = 553MB vs 26MB; ratio to raw data size = 21.025497101449275
    [junit]   loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0 sync 0/0 up
0 down 0
    [junit] 
    [junit]  modified code: 
    [junit]   Duration = 384ms maxConcurrency = 1
    [junit]   GC for PS Scavenge: 24 ms for 2 collections
    [junit]   Approx allocation = 556MB vs 26MB; ratio to raw data size = 21.123911594202898
    [junit]   loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0 sync 0/0 up
0 down 0
    [junit] 
    [junit] 
    [junit] Threads = 1 elements = 100000 (of size 256) partitions = 16
    [junit]  original code:
    [junit]   Duration = 329ms maxConcurrency = 1
    [junit]   GC for PS Scavenge: 10 ms for 1 collections
    [junit]   Approx allocation = 448MB vs 26MB; ratio to raw data size = 17.03236608695652
    [junit]   loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0 sync 0/0 up
0 down 0
    [junit] 
    [junit]  modified code: 
    [junit]   Duration = 324ms maxConcurrency = 1
    [junit]   GC for PS Scavenge: 9 ms for 1 collections
    [junit]   Approx allocation = 449MB vs 26MB; ratio to raw data size = 17.06260608695652
    [junit]   loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0 sync 0/0 up
0 down 0
    [junit] 
    [junit] 
    [junit] Threads = 1 elements = 100000 (of size 256) partitions = 256
    [junit]  original code:
    [junit]   Duration = 692ms maxConcurrency = 1
    [junit]   GC for PS Scavenge: 14 ms for 1 collections
    [junit]   Approx allocation = 339MB vs 26MB; ratio to raw data size = 12.898537971014493
    [junit]   loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0 sync 0/0 up
0 down 0
    [junit] 
    [junit]  modified code: 
    [junit]   Duration = 681ms maxConcurrency = 1
    [junit]   GC for PS Scavenge: 13 ms for 1 collections
    [junit]   Approx allocation = 338MB vs 26MB; ratio to raw data size = 12.876152753623188
    [junit]   loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0 sync 0/0 up
0 down 0
    [junit] 
    [junit] 
    [junit] Threads = 1 elements = 100000 (of size 256) partitions = 1024
    [junit]  original code:
    [junit]   Duration = 484ms maxConcurrency = 1
    [junit]   GC for PS Scavenge: 14 ms for 1 collections
    [junit]   Approx allocation = 282MB vs 26MB; ratio to raw data size = 10.73973768115942
    [junit]   loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0 sync 0/0 up
0 down 0
    [junit] 
    [junit]  modified code: 
    [junit]   Duration = 463ms maxConcurrency = 1
    [junit]   GC for PS Scavenge: 14 ms for 1 collections
    [junit]   Approx allocation = 282MB vs 26MB; ratio to raw data size = 10.74182811594203
    [junit]   loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0 sync 0/0 up
0 down 0
    [junit] 
    [junit] 
    [junit] --------------------------------------------------
    [junit] 100 THREADS; ELEMENT SIZE 256
    [junit] 
    [junit] Threads = 100 elements = 100000 (of size 256) partitions = 1
    [junit]  original code:
    [junit]   Duration = 2231ms maxConcurrency = 100
    [junit]   GC for PS Scavenge: 121 ms for 33 collections
    [junit]   Approx allocation = 10752MB vs 26MB; ratio to raw data size = 408.504135942029
    [junit]   loopRatio (closest to 1 best) 18.49143 raw 100000/1849143 counted 0/0 sync 0/0
up 0 down 0
    [junit] 
    [junit]  modified code: 
    [junit]   Duration = 1330ms maxConcurrency = 100
    [junit]   GC for PS Scavenge: 32 ms for 2 collections
    [junit]   Approx allocation = 587MB vs 26MB; ratio to raw data size = 22.319155362318842
    [junit]   loopRatio (closest to 1 best) 1.12889 raw 16479/16479 counted 0/0 sync 85804/96410
up 0 down 0
    [junit] 
    [junit] 
    [junit] Threads = 100 elements = 100000 (of size 256) partitions = 16
    [junit]  original code:
    [junit]   Duration = 227ms maxConcurrency = 99
    [junit]   GC for PS Scavenge: 25 ms for 2 collections
    [junit]   Approx allocation = 752MB vs 26MB; ratio to raw data size = 28.593344057971013
    [junit]   loopRatio (closest to 1 best) 1.82137 raw 100000/182137 counted 0/0 sync 0/0
up 0 down 0
    [junit] 
    [junit]  modified code: 
    [junit]   Duration = 204ms maxConcurrency = 98
    [junit]   GC for PS Scavenge: 10 ms for 1 collections
    [junit]   Approx allocation = 552MB vs 26MB; ratio to raw data size = 21.00069768115942
    [junit]   loopRatio (closest to 1 best) 1.26846 raw 72715/72715 counted 0/0 sync 49948/54131
up 0 down 0
    [junit] 
    [junit] 
    [junit] Threads = 100 elements = 100000 (of size 256) partitions = 256
    [junit]  original code:
    [junit]   Duration = 180ms maxConcurrency = 94
    [junit]   GC for PS Scavenge: 13 ms for 1 collections
    [junit]   Approx allocation = 351MB vs 26MB; ratio to raw data size = 13.347453333333334
    [junit]   loopRatio (closest to 1 best) 1.01903 raw 100000/101903 counted 0/0 sync 0/0
up 0 down 0
    [junit] 
    [junit]  modified code: 
    [junit]   Duration = 178ms maxConcurrency = 97
    [junit]   GC for PS Scavenge: 13 ms for 1 collections
    [junit]   Approx allocation = 350MB vs 26MB; ratio to raw data size = 13.320123768115941
    [junit]   loopRatio (closest to 1 best) 1.0193 raw 99954/99954 counted 0/0 sync 1957/1976
up 0 down 0
    [junit] 
    [junit] 
    [junit] Threads = 100 elements = 100000 (of size 256) partitions = 1024
    [junit]  original code:
    [junit]   Duration = 179ms maxConcurrency = 99
    [junit]   GC for PS Scavenge: 13 ms for 1 collections
    [junit]   Approx allocation = 292MB vs 26MB; ratio to raw data size = 11.113111884057972
    [junit]   loopRatio (closest to 1 best) 1.00393 raw 100000/100393 counted 0/0 sync 0/0
up 0 down 0
    [junit] 
    [junit]  modified code: 
    [junit]   Duration = 178ms maxConcurrency = 93
    [junit]   GC for PS Scavenge: 13 ms for 1 collections
    [junit]   Approx allocation = 293MB vs 26MB; ratio to raw data size = 11.15906231884058
    [junit]   loopRatio (closest to 1 best) 1.00396 raw 99997/99997 counted 0/0 sync 398/399
up 0 down 0
    [junit] 
    [junit] 
    [junit] --------------------------------------------------
    [junit] 1 THREAD; ELEMENT SIZE 1024
    [junit] 
    [junit] Threads = 1 elements = 100000 (of size 1024) partitions = 1
    [junit]  original code:
    [junit]   Duration = 997ms maxConcurrency = 1
    [junit]   GC for PS Scavenge: 30 ms for 2 collections
    [junit]   Approx allocation = 586MB vs 99MB; ratio to raw data size = 5.885971954022988
    [junit]   loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0 sync 0/0 up
0 down 0
    [junit] 
    [junit]  modified code: 
    [junit]   Duration = 836ms maxConcurrency = 1
    [junit]   GC for PS Scavenge: 42 ms for 3 collections
    [junit]   Approx allocation = 623MB vs 99MB; ratio to raw data size = 6.264844827586207
    [junit]   loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0 sync 0/0 up
0 down 0
    [junit] 
    [junit] 
    [junit] Threads = 1 elements = 100000 (of size 1024) partitions = 16
    [junit]  original code:
    [junit]   Duration = 600ms maxConcurrency = 1
    [junit]   GC for PS Scavenge: 28 ms for 2 collections
    [junit]   Approx allocation = 508MB vs 99MB; ratio to raw data size = 5.104469731800767
    [junit]   loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0 sync 0/0 up
0 down 0
    [junit] 
    [junit]  modified code: 
    [junit]   Duration = 572ms maxConcurrency = 1
    [junit]   GC for PS Scavenge: 26 ms for 2 collections
    [junit]   Approx allocation = 513MB vs 99MB; ratio to raw data size = 5.159989042145594
    [junit]   loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0 sync 0/0 up
0 down 0
    [junit] 
    [junit] 
    [junit] Threads = 1 elements = 100000 (of size 1024) partitions = 256
    [junit]  original code:
    [junit]   Duration = 413ms maxConcurrency = 1
    [junit]   GC for PS Scavenge: 25 ms for 2 collections
    [junit]   Approx allocation = 440MB vs 99MB; ratio to raw data size = 4.425642222222222
    [junit]   loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0 sync 0/0 up
0 down 0
    [junit] 
    [junit]  modified code: 
    [junit]   Duration = 368ms maxConcurrency = 1
    [junit]   GC for PS Scavenge: 27 ms for 2 collections
    [junit]   Approx allocation = 393MB vs 99MB; ratio to raw data size = 3.948129195402299
    [junit]   loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0 sync 0/0 up
0 down 0
    [junit] 
    [junit] 
    [junit] Threads = 1 elements = 100000 (of size 1024) partitions = 1024
    [junit]  original code:
    [junit]   Duration = 296ms maxConcurrency = 1
    [junit]   GC for PS Scavenge: 27 ms for 2 collections
    [junit]   Approx allocation = 343MB vs 99MB; ratio to raw data size = 3.4517041379310345
    [junit]   loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0 sync 0/0 up
0 down 0
    [junit] 
    [junit]  modified code: 
    [junit]   Duration = 550ms maxConcurrency = 1
    [junit]   GC for PS Scavenge: 35 ms for 2 collections
    [junit]   Approx allocation = 343MB vs 99MB; ratio to raw data size = 3.4461845977011496
    [junit]   loopRatio (closest to 1 best) 1.0 raw 100000/100000 counted 0/0 sync 0/0 up
0 down 0
    [junit] 
    [junit] 
    [junit] --------------------------------------------------
    [junit] 100 THREADS; ELEMENT SIZE 1024
    [junit] 
    [junit] Threads = 100 elements = 100000 (of size 1024) partitions = 1
    [junit]  original code:
    [junit]   Duration = 2343ms maxConcurrency = 100
    [junit]   GC for PS Scavenge: 143 ms for 38 collections
    [junit]   Approx allocation = 12065MB vs 99MB; ratio to raw data size = 121.18560452107279
    [junit]   loopRatio (closest to 1 best) 18.22144 raw 100000/1822144 counted 0/0 sync 0/0
up 0 down 0
    [junit] 
    [junit]  modified code: 
    [junit]   Duration = 1238ms maxConcurrency = 100
    [junit]   GC for PS Scavenge: 31 ms for 2 collections
    [junit]   Approx allocation = 569MB vs 99MB; ratio to raw data size = 5.7199810727969345
    [junit]   loopRatio (closest to 1 best) 1.11326 raw 13522/13522 counted 0/0 sync 88452/97804
up 0 down 0
    [junit] 
    [junit] 
    [junit] Threads = 100 elements = 100000 (of size 1024) partitions = 16
    [junit]  original code:
    [junit]   Duration = 235ms maxConcurrency = 100
    [junit]   GC for PS Scavenge: 35 ms for 3 collections
    [junit]   Approx allocation = 792MB vs 99MB; ratio to raw data size = 7.96215540229885
    [junit]   loopRatio (closest to 1 best) 1.79828 raw 100000/179828 counted 0/0 sync 0/0
up 0 down 0
    [junit] 
    [junit]  modified code: 
    [junit]   Duration = 228ms maxConcurrency = 100
    [junit]   GC for PS Scavenge: 30 ms for 2 collections
    [junit]   Approx allocation = 618MB vs 99MB; ratio to raw data size = 6.211963754789272
    [junit]   loopRatio (closest to 1 best) 1.26855 raw 75427/75427 counted 0/0 sync 47424/51428
up 0 down 0
    [junit] 
    [junit] 
    [junit] Threads = 100 elements = 100000 (of size 1024) partitions = 256
    [junit]  original code:
    [junit]   Duration = 189ms maxConcurrency = 98
    [junit]   GC for PS Scavenge: 12 ms for 1 collections
    [junit]   Approx allocation = 421MB vs 99MB; ratio to raw data size = 4.237997701149426
    [junit]   loopRatio (closest to 1 best) 1.01838 raw 100000/101838 counted 0/0 sync 0/0
up 0 down 0
    [junit] 
    [junit]  modified code: 
    [junit]   Duration = 199ms maxConcurrency = 98
    [junit]   GC for PS Scavenge: 30 ms for 2 collections
    [junit]   Approx allocation = 405MB vs 99MB; ratio to raw data size = 4.073938620689655
    [junit]   loopRatio (closest to 1 best) 1.01976 raw 99960/99960 counted 0/0 sync 2000/2016
up 0 down 0
    [junit] 
    [junit] 
    [junit] Threads = 100 elements = 100000 (of size 1024) partitions = 1024
    [junit]  original code:
    [junit]   Duration = 195ms maxConcurrency = 100
    [junit]   GC for PS Scavenge: 32 ms for 2 collections
    [junit]   Approx allocation = 356MB vs 99MB; ratio to raw data size = 3.577380153256705
    [junit]   loopRatio (closest to 1 best) 1.00401 raw 100000/100401 counted 0/0 sync 0/0
up 0 down 0
    [junit] 
    [junit]  modified code: 
    [junit]   Duration = 197ms maxConcurrency = 96
    [junit]   GC for PS Scavenge: 31 ms for 2 collections
    [junit]   Approx allocation = 352MB vs 99MB; ratio to raw data size = 3.538006283524904
    [junit]   loopRatio (closest to 1 best) 1.00377 raw 99999/99999 counted 0/0 sync 376/378
up 0 down 0
{code}

> AtomicSortedColumns.addAllWithSizeDelta has a spin loop that allocates memory
> -----------------------------------------------------------------------------
>
>                 Key: CASSANDRA-7546
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-7546
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>            Reporter: graham sanderson
>            Assignee: graham sanderson
>         Attachments: 7546.20.txt, 7546.20_2.txt, 7546.20_alt.txt, suggestion1.txt, suggestion1_21.txt
>
>
> In order to preserve atomicity, this code attempts to read, clone/update, then CAS the
state of the partition.
> Under heavy contention for updating a single partition this can cause some fairly staggering
memory growth (the more cores on your machine the worst it gets).
> Whilst many usage patterns don't do highly concurrent updates to the same partition,
hinting today, does, and in this case wild (order(s) of magnitude more than expected) memory
allocation rates can be seen (especially when the updates being hinted are small updates to
different partitions which can happen very fast on their own) - see CASSANDRA-7545
> It would be best to eliminate/reduce/limit the spinning memory allocation whilst not
slowing down the very common un-contended case.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message