cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jacek Furmankiewicz (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-7361) Cassandra locks up in full GC, unable to free memory
Date Fri, 06 Jun 2014 16:14:03 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-7361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14019976#comment-14019976 ] 

Jacek Furmankiewicz commented on CASSANDRA-7361:
------------------------------------------------

I have a tar.gz of the heapdump (2.3 GB). I am not able to attach it to JIRA. Let me know if there is anywhere we could upload it for you.

> Cassandra locks up in full GC, unable to free memory
> ----------------------------------------------------
>
>                 Key: CASSANDRA-7361
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-7361
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: Ubuntu, RedHat, JDK 1.7
>            Reporter: Jacek Furmankiewicz
>            Priority: Blocker
>
> We have a long running batch load process, which runs for many hours.
> Massive amount of writes, in large mutation batches (we increase the thrift frame size to 45 MB).
> Everything goes well, but after about 3 hrs of processing everything locks up. We start getting NoHostsAvailable exceptions on the Java application side (with Astyanax as our driver), eventually socket timeouts.
> Looking at Cassandra, we can see that it is using nearly the full 8GB of heap and unable to free it. It spends most of its time in full GC, but the amount of memory does not go down.
> Here is a long sample from jstat to show this over an extended time period
> e.g.
> {quote}
> -bash-4.1$ $JAVA_HOME/bin/jstat -gc 3728 5s
>  S0C    S1C    S0U    S1U      EC       EU        OC         OU       PC     PU    YGC     YGCT    FGC    FGCT     GCT   
> 81920.0 81920.0  0.0   4097.0 655360.0 150419.0 7569408.0  7202990.3  43280.0 25855.8   5271  268.378 4940  9877.726 10146.104
> 81920.0 81920.0  0.0   4097.0 655360.0 150657.2 7569408.0  7202990.3  43280.0 25855.8   5271  268.378 4940  9877.726 10146.104
> 81920.0 81920.0  0.0   4097.0 655360.0 150708.5 7569408.0  7202990.3  43280.0 25855.8   5271  268.378 4941  9877.950 10146.328
> 81920.0 81920.0  0.0   4097.0 655360.0 150920.4 7569408.0  7202990.2  43280.0 25855.8   5271  268.378 4942  9878.085 10146.463
> 81920.0 81920.0  0.0   4097.0 655360.0 150930.7 7569408.0  7202990.2  43280.0 25855.8   5271  268.378 4942  9878.085 10146.463
> 81920.0 81920.0  0.0   4097.0 655360.0 150998.2 7569408.0  7202990.2  43280.0 25855.8   5271  268.378 4942  9878.085 10146.463
> 81920.0 81920.0  0.0   4097.0 655360.0 151335.6 7569408.0  7202990.2  43280.0 25855.8   5271  268.378 4943  9878.253 10146.630
> 81920.0 81920.0  0.0   4097.0 655360.0 151483.3 7569408.0  7202990.2  43280.0 25855.8   5271  268.378 4944  9878.389 10146.766
> 81920.0 81920.0  0.0   4097.0 655360.0 151750.4 7569408.0  7202990.2  43280.0 25855.8   5271  268.378 4944  9878.389 10146.766
> 81920.0 81920.0  0.0   4097.0 655360.0 151752.5 7569408.0  7202990.2  43280.0 25855.8   5271  268.378 4944  9878.389 10146.766
> 81920.0 81920.0  0.0   4097.0 655360.0 151997.4 7569408.0  7202990.2  43280.0 25855.8   5271  268.378 4945  9878.593 10146.970
> 81920.0 81920.0  0.0   4097.0 655360.0 152063.0 7569408.0  7202988.6  43280.0 25855.8   5271  268.378 4946  9878.732 10147.110
> 81920.0 81920.0  0.0   4097.0 655360.0 152256.2 7569408.0  7202988.6  43280.0 25855.8   5271  268.378 4946  9878.732 10147.110
> 81920.0 81920.0  0.0   4097.0 655360.0 152276.2 7569408.0  7202988.6  43280.0 25855.8   5271  268.378 4946  9878.732 10147.110
> 81920.0 81920.0  0.0   4097.0 655360.0 152276.2 7569408.0  7202988.6  43280.0 25855.8   5271  268.378 4947  9878.968 10147.345
> 81920.0 81920.0  0.0   4097.0 655360.0 152525.3 7569408.0  7202857.5  43280.0 25855.8   5271  268.378 4948  9879.107 10147.484
> 81920.0 81920.0  0.0   4097.0 655360.0 152673.0 7569408.0  7202857.5  43280.0 25855.8   5271  268.378 4948  9879.107 10147.484
> 81920.0 81920.0  0.0   4097.0 655360.0 152855.5 7569408.0  7202857.5  43280.0 25855.8   5271  268.378 4948  9879.107 10147.484
> 81920.0 81920.0  0.0   4097.0 655360.0 152871.3 7569408.0  7202857.5  43280.0 25855.8   5271  268.378 4949  9879.275 10147.653
> 81920.0 81920.0  0.0   4097.0 655360.0 153116.2 7569408.0  7202857.0  43280.0 25855.8   5271  268.378 4950  9879.415 10147.793
> 81920.0 81920.0  0.0   4097.0 655360.0 153191.6 7569408.0  7202857.0  43280.0 25855.8   5271  268.378 4950  9879.415 10147.793
> 81920.0 81920.0  0.0   4097.0 655360.0 153454.3 7569408.0  7202857.0  43280.0 25855.8   5271  268.378 4950  9879.415 10147.793
> 81920.0 81920.0  0.0   4097.0 655360.0 153528.0 7569408.0  7202857.0  43280.0 25855.8   5271  268.378 4951  9879.583 10147.961
> 81920.0 81920.0  0.0   4097.0 655360.0 153766.1 7569408.0  7202857.0  43280.0 25855.8   5271  268.378 4952  9879.721 10148.099
> 81920.0 81920.0  0.0   4097.0 655360.0 153791.9 7569408.0  7202857.0  43280.0 25855.8   5271  268.378 4952  9879.721 10148.099
> 81920.0 81920.0  0.0   4097.0 655360.0 153855.1 7569408.0  7202857.0  43280.0 25855.8   5271  268.378 4952  9879.721 10148.099
> 81920.0 81920.0  0.0   4097.0 655360.0 154114.8 7569408.0  7202857.0  43280.0 25855.8   5271  268.378 4953  9879.953 10148.331
> 81920.0 81920.0  0.0   4097.0 655360.0 154118.8 7569408.0  7202856.8  43280.0 25855.8   5271  268.378 4954  9880.097 10148.474
> 81920.0 81920.0  0.0   4097.0 655360.0 154382.7 7569408.0  7202856.8  43280.0 25855.8   5271  268.378 4954  9880.097 10148.474
> 81920.0 81920.0  0.0   4097.0 655360.0 154511.8 7569408.0  7202856.8  43280.0 25855.8   5271  268.378 4954  9880.097 10148.474
> 81920.0 81920.0  0.0   4097.0 655360.0 154699.2 7569408.0  7202856.8  43280.0 25855.8   5271  268.378 4955  9880.273 10148.650
> 81920.0 81920.0  0.0   4097.0 655360.0 154707.4 7569408.0  7202856.7  43280.0 25855.8   5271  268.378 4956  9880.273 10148.650
> 81920.0 81920.0  0.0   4097.0 655360.0 154985.2 7569408.0  7202856.7  43280.0 25855.8   5271  268.378 4956  9880.414 10148.792
> 81920.0 81920.0  0.0   4097.0 655360.0 155006.1 7569408.0  7202856.7  43280.0 25855.8   5271  268.378 4956  9880.414 10148.792
> 81920.0 81920.0  0.0   4097.0 655360.0 155050.8 7569408.0  7202856.7  43280.0 25855.8   5271  268.378 4957  9880.585 10148.963
> 81920.0 81920.0  0.0   4097.0 655360.0 155233.1 7569408.0  7202856.7  43280.0 25855.8   5271  268.378 4957  9880.585 10148.963
> 81920.0 81920.0  0.0   4097.0 655360.0 155536.7 7569408.0  7202856.7  43280.0 25855.8   5271  268.378 4958  9880.730 10149.108
> 81920.0 81920.0  0.0   4097.0 655360.0 156764.1 7569408.0  7202856.7  43280.0 25855.8   5271  268.378 4958  9880.730 10149.108
> 81920.0 81920.0  0.0   4097.0 655360.0 156829.3 7569408.0  7202856.7  43280.0 25855.8   5271  268.378 4959  9880.963 10149.341
> 81920.0 81920.0  0.0   4097.0 655360.0 157015.2 7569408.0  7202856.6  43280.0 25855.8   5271  268.378 4959  9880.963 10149.341
> 81920.0 81920.0  0.0   4097.0 655360.0 157089.1 7569408.0  7202856.6  43280.0 25855.8   5271  268.378 4960  9881.104 10149.481
> 81920.0 81920.0  0.0   4097.0 655360.0 157353.1 7569408.0  7202856.6  43280.0 25855.8   5271  268.378 4960  9881.104 10149.481
> 81920.0 81920.0  0.0   4097.0 655360.0 157368.8 7569408.0  7202856.6  43280.0 25855.8   5271  268.378 4961  9881.279 10149.656
> 81920.0 81920.0  0.0   4097.0 655360.0 157608.5 7569408.0  7202856.6  43280.0 25855.8   5271  268.378 4961  9881.279 10149.656
> 81920.0 81920.0  0.0   4097.0 655360.0 157684.2 7569408.0  7202856.6  43280.0 25855.8   5271  268.378 4962  9881.422 10149.800
> 81920.0 81920.0  0.0   4097.0 655360.0 157922.4 7569408.0  7202856.6  43280.0 25855.8   5271  268.378 4962  9881.422 10149.800
> 81920.0 81920.0  0.0   4097.0 655360.0 157959.9 7569408.0  7202856.6  43280.0 25855.8   5271  268.378 4963  9881.600 10149.977
> 81920.0 81920.0  0.0   4097.0 655360.0 158004.8 7569408.0  7202856.5  43280.0 25855.8   5271  268.378 4963  9881.600 10149.977
> 81920.0 81920.0  0.0   4097.0 655360.0 158197.6 7569408.0  7202856.5  43280.0 25855.8   5271  268.378 4964  9881.742 10150.120
> 81920.0 81920.0  0.0   4097.0 655360.0 158209.8 7569408.0  7202856.5  43280.0 25855.8   5271  268.378 4964  9881.742 10150.120
> 81920.0 81920.0  0.0   4097.0 655360.0 158473.8 7569408.0  7202856.5  43280.0 25855.8   5271  268.378 4965  9881.920 10150.297
> 81920.0 81920.0  0.0   4097.0 655360.0 158473.8 7569408.0  7202856.5  43280.0 25855.8   5271  268.378 4965  9881.920 10150.297
> 81920.0 81920.0  0.0   4097.0 655360.0 158769.7 7569408.0  7202856.5  43280.0 25855.8   5271  268.378 4966  9882.063 10150.441
> 81920.0 81920.0  0.0   4097.0 655360.0 158796.6 7569408.0  7202856.5  43280.0 25855.8   5271  268.378 4966  9882.063 10150.441
> 81920.0 81920.0  0.0   4097.0 655360.0 159076.4 7569408.0  7202856.5  43280.0 25855.8   5271  268.378 4966  9882.063 10150.441
> 81920.0 81920.0  0.0   4097.0 655360.0 159080.7 7569408.0  7202856.4  43280.0 25855.8   5271  268.378 4967  9882.250 10150.628
> 81920.0 81920.0  0.0   4097.0 655360.0 159190.8 7569408.0  7202856.4  43280.0 25855.8   5271  268.378 4968  9882.394 10150.772
> 81920.0 81920.0  0.0   4097.0 655360.0 159391.8 7569408.0  7202856.4  43280.0 25855.8   5271  268.378 4968  9882.394 10150.772
> 81920.0 81920.0  0.0   4097.0 655360.0 159697.5 7569408.0  7202856.4  43280.0 25855.8   5271  268.378 4968  9882.394 10150.772
> 81920.0 81920.0  0.0   4097.0 655360.0 159737.1 7569408.0  7202856.3  43280.0 25855.8   5271  268.378 4969  9882.636 10151.014
> 81920.0 81920.0  0.0   4097.0 655360.0 159737.1 7569408.0  7202856.3  43280.0 25855.8   5271  268.378 4970  9882.782 10151.159
> 81920.0 81920.0  0.0   4097.0 655360.0 159958.2 7569408.0  7202856.3  43280.0 25855.8   5271  268.378 4970  9882.782 10151.159
> 81920.0 81920.0  0.0   4097.0 655360.0 159985.0 7569408.0  7202856.3  43280.0 25855.8   5271  268.378 4970  9882.782 10151.159
> 81920.0 81920.0  0.0   4097.0 655360.0 160249.0 7569408.0  7202856.3  43280.0 25855.8   5271  268.378 4971  9882.974 10151.352
> 81920.0 81920.0  0.0   4097.0 655360.0 160249.0 7569408.0  7202856.3  43280.0 25855.8   5271  268.378 4972  9883.121 10151.499
> 81920.0 81920.0  0.0   4097.0 655360.0 160361.3 7569408.0  7202856.3  43280.0 25855.8   5271  268.378 4972  9883.121 10151.499
> 81920.0 81920.0  0.0   4097.0 655360.0 160638.2 7569408.0  7202856.3  43280.0 25855.8   5271  268.378 4972  9883.121 10151.499
> 81920.0 81920.0  0.0   4097.0 655360.0 160894.9 7569408.0  7202856.3  43280.0 25855.8   5271  268.378 4973  9883.368 10151.746
> 81920.0 81920.0  0.0   4097.0 655360.0 160920.7 7569408.0  7202856.3  43280.0 25855.8   5271  268.378 4974  9883.519 10151.897
> 81920.0 81920.0  0.0   4097.0 655360.0 160920.7 7569408.0  7202856.3  43280.0 25855.8   5271  268.378 4974  9883.519 10151.897
> 81920.0 81920.0  0.0   4097.0 655360.0 161139.5 7569408.0  7202856.3  43280.0 25855.8   5271  268.378 4975  9883.519 10151.897
> 81920.0 81920.0  0.0   4097.0 655360.0 161168.4 7569408.0  7202856.1  43280.0 25855.8   5271  268.378 4975  9883.735 10152.113
> 81920.0 81920.0  0.0   4097.0 655360.0 161427.2 7569408.0  7202856.1  43280.0 25855.8   5271  268.378 4976  9883.884 10152.262
> 81920.0 81920.0  0.0   4097.0 655360.0 161446.2 7569408.0  7202856.1  43280.0 25855.8   5271  268.378 4976  9883.884 10152.262
> 81920.0 81920.0  0.0   4097.0 655360.0 161490.9 7569408.0  7202856.1  43280.0 25855.8   5271  268.378 4976  9883.884 10152.262
> 81920.0 81920.0  0.0   4097.0 655360.0 161673.2 7569408.0  7202856.1  43280.0 25855.8   5271  268.378 4977  9884.066 10152.444
> 81920.0 81920.0  0.0   4097.0 655360.0 162002.3 7569408.0  7202856.1  43280.0 25855.8   5271  268.378 4978  9884.217 10152.595
> 81920.0 81920.0  0.0   4097.0 655360.0 162023.6 7569408.0  7202856.1  43280.0 25855.8   5271  268.378 4978  9884.217 10152.595
> 81920.0 81920.0  0.0   4097.0 655360.0 162086.0 7569408.0  7202856.1  43280.0 25855.8   5271  268.378 4978  9884.217 10152.595
> 81920.0 81920.0  0.0   4097.0 655360.0 162271.9 7569408.0  7202856.1  43280.0 25855.8   5271  268.378 4979  9884.405 10152.782
> 81920.0 81920.0  0.0   4097.0 655360.0 162537.0 7569408.0  7202856.1  43280.0 25855.8   5271  268.378 4980  9884.557 10152.935
> 81920.0 81920.0  0.0   4097.0 655360.0 162696.2 7569408.0  7202856.1  43280.0 25855.8   5271  268.378 4980  9884.557 10152.935
> 81920.0 81920.0  0.0   4097.0 655360.0 162696.2 7569408.0  7202856.1  43280.0 25855.8   5271  268.378 4980  9884.557 10152.935
> 81920.0 81920.0  0.0   4097.0 655360.0 162740.8 7569408.0  7202856.1  43280.0 25855.8   5271  268.378 4981  9884.748 10153.125
> 81920.0 81920.0  0.0   4097.0 655360.0 163009.5 7569408.0  7202856.1  43280.0 25855.8   5271  268.378 4982  9884.901 10153.279
> 81920.0 81920.0  0.0   4097.0 655360.0 163285.1 7569408.0  7202856.1  43280.0 25855.8   5271  268.378 4982  9884.901 10153.279
> 81920.0 81920.0  0.0   4097.0 655360.0 163285.1 7569408.0  7202856.1  43280.0 25855.8   5271  268.378 4982  9884.901 10153.279
> 81920.0 81920.0  0.0   4097.0 655360.0 163397.4 7569408.0  7202856.1  43280.0 25855.8   5271  268.378 4983  9885.093 10153.471
> 81920.0 81920.0  0.0   4097.0 655360.0 163590.2 7569408.0  7202855.9  43280.0 25855.8   5271  268.378 4984  9885.251 10153.628
> 81920.0 81920.0  0.0   4097.0 655360.0 165038.0 7569408.0  7202855.9  43280.0 25855.8   5271  268.378 4984  9885.251 10153.628
> 81920.0 81920.0  0.0   4097.0 655360.0 165142.5 7569408.0  7202855.9  43280.0 25855.8   5271  268.378 4984  9885.251 10153.628
> 81920.0 81920.0  0.0   4097.0 655360.0 165142.5 7569408.0  7202855.9  43280.0 25855.8   5271  268.378 4985  9885.435 10153.813
> 81920.0 81920.0  0.0   4097.0 655360.0 165391.7 7569408.0  7202855.8  43280.0 25855.8   5271  268.378 4986  9885.588 10153.966
> 81920.0 81920.0  0.0   4097.0 655360.0 165401.9 7569408.0  7202855.8  43280.0 25855.8   5271  268.378 4986  9885.588 10153.966
> 81920.0 81920.0  0.0   4097.0 655360.0 165665.9 7569408.0  7202855.8  43280.0 25855.8   5271  268.378 4986  9885.588 10153.966
> 81920.0 81920.0  0.0   4097.0 655360.0 165667.9 7569408.0  7202855.8  43280.0 25855.8   5271  268.378 4987  9885.848 10154.226
> 81920.0 81920.0  0.0   4097.0 655360.0 165712.6 7569408.0  7202855.8  43280.0 25855.8   5271  268.378 4988  9886.002 10154.379
> 81920.0 81920.0  0.0   4097.0 655360.0 165971.0 7569408.0  7202855.8  43280.0 25855.8   5271  268.378 4988  9886.002 10154.379
> 81920.0 81920.0  0.0   4097.0 655360.0 166254.9 7569408.0  7202855.8  43280.0 25855.8   5271  268.378 4988  9886.002 10154.379
> 81920.0 81920.0  0.0   4097.0 655360.0 166257.1 7569408.0  7202855.8  43280.0 25855.8   5271  268.378 4989  9886.256 10154.633
> 81920.0 81920.0  0.0   4097.0 655360.0 166257.1 7569408.0  7202855.7  43280.0 25855.8   5271  268.378 4989  9886.256 10154.633
> 81920.0 81920.0  0.0   4097.0 655360.0 166496.5 7569408.0  7202855.7  43280.0 25855.8   5271  268.378 4990  9886.412 10154.789
> 81920.0 81920.0  0.0   4097.0 655360.0 166843.1 7569408.0  7202855.7  43280.0 25855.8   5271  268.378 4990  9886.412 10154.789
> 81920.0 81920.0  0.0   4097.0 655360.0 166852.0 7569408.0  7202855.7  43280.0 25855.8   5271  268.378 4991  9886.605 10154.983
> 81920.0 81920.0  0.0   4097.0 655360.0 166852.0 7569408.0  7202855.6  43280.0 25855.8   5271  268.378 4991  9886.605 10154.983
> 81920.0 81920.0  0.0   4097.0 655360.0 166908.3 7569408.0  7202855.6  43280.0 25855.8   5271  268.378 4992  9886.762 10155.140
> 81920.0 81920.0  0.0   4097.0 655360.0 167122.1 7569408.0  7202855.6  43280.0 25855.8   5271  268.378 4992  9886.762 10155.140
> 81920.0 81920.0  0.0   4097.0 655360.0 167375.4 7569408.0  7202855.6  43280.0 25855.8   5271  268.378 4993  9887.023 10155.401
> 81920.0 81920.0  0.0   4097.0 655360.0 167375.4 7569408.0  7202855.6  43280.0 25855.8   5271  268.378 4993  9887.023 10155.401
> 81920.0 81920.0  0.0   4097.0 655360.0 167377.6 7569408.0  7202855.6  43280.0 25855.8   5271  268.378 4994  9887.179 10155.557
> 81920.0 81920.0  0.0   4097.0 655360.0 167683.0 7569408.0  7202855.6  43280.0 25855.8   5271  268.378 4994  9887.179 10155.557
> 81920.0 81920.0  0.0   4097.0 655360.0 167948.1 7569408.0  7202855.6  43280.0 25855.8   5271  268.378 4994  9887.179 10155.557
> 81920.0 81920.0  0.0   4097.0 655360.0 167968.7 7569408.0  7202855.6  43280.0 25855.8   5271  268.378 4995  9887.376 10155.754
> 81920.0 81920.0  0.0   4097.0 655360.0 168034.2 7569408.0  7202855.6  43280.0 25855.8   5271  268.378 4996  9887.534 10155.911
> 81920.0 81920.0  0.0   4097.0 655360.0 168094.6 7569408.0  7202855.6  43280.0 25855.8   5271  268.378 4996  9887.534 10155.911
> 81920.0 81920.0  0.0   4097.0 655360.0 168270.8 7569408.0  7202855.6  43280.0 25855.8   5271  268.378 4996  9887.534 10155.911
> 81920.0 81920.0  0.0   4097.0 655360.0 168561.5 7569408.0  7202855.6  43280.0 25855.8   5271  268.378 4997  9887.808 10156.185
> 81920.0 81920.0  0.0   4097.0 655360.0 168627.3 7569408.0  7202855.5  43280.0 25855.8   5271  268.378 4998  9887.968 10156.345
> 81920.0 81920.0  0.0   4097.0 655360.0 168641.1 7569408.0  7202855.5  43280.0 25855.8   5271  268.378 4998  9887.968 10156.345
> 81920.0 81920.0  0.0   4097.0 655360.0 168859.9 7569408.0  7202855.5  43280.0 25855.8   5271  268.378 4998  9887.968 10156.345
> 81920.0 81920.0  0.0   4097.0 655360.0 169143.9 7569408.0  7202855.5  43280.0 25855.8   5271  268.378 4999  9888.170 10156.547
> 81920.0 81920.0  0.0   4097.0 655360.0 169152.9 7569408.0  7202855.4  43280.0 25855.8   5271  268.378 5000  9888.330 10156.708
> 81920.0 81920.0  0.0   4097.0 655360.0 169152.9 7569408.0  7202855.4  43280.0 25855.8   5271  268.378 5000  9888.330 10156.708
> 81920.0 81920.0  0.0   4097.0 655360.0 169373.9 7569408.0  7202855.4  43280.0 25855.8   5271  268.378 5000  9888.330 10156.708
> 81920.0 81920.0  0.0   4097.0 655360.0 169639.0 7569408.0  7202855.4  43280.0 25855.8   5271  268.378 5001  9888.591 10156.968
> 81920.0 81920.0  0.0   4097.0 655360.0 169744.2 7569408.0  7202855.4  43280.0 25855.8   5271  268.378 5002  9888.754 10157.131
> 81920.0 81920.0  0.0   4097.0 655360.0 169822.8 7569408.0  7202855.4  43280.0 25855.8   5271  268.378 5002  9888.754 10157.131
> 81920.0 81920.0  0.0   4097.0 655360.0 169888.3 7569408.0  7202855.4  43280.0 25855.8   5271  268.378 5002  9888.754 10157.131
> 81920.0 81920.0  0.0   4097.0 655360.0 170125.7 7569408.0  7202855.4  43280.0 25855.8   5271  268.378 5003  9888.946 10157.323
> 81920.0 81920.0  0.0   4097.0 655360.0 170399.8 7569408.0  7202855.3  43280.0 25855.8   5271  268.378 5004  9889.106 10157.483
> 81920.0 81920.0  0.0   4097.0 655360.0 170399.8 7569408.0  7202855.3  43280.0 25855.8   5271  268.378 5004  9889.106 10157.483
> 81920.0 81920.0  0.0   4097.0 655360.0 170420.7 7569408.0  7202855.3  43280.0 25855.8   5271  268.378 5004  9889.106 10157.483
> 81920.0 81920.0  0.0   4097.0 655360.0 170651.4 7569408.0  7202855.3  43280.0 25855.8   5271  268.378 5005  9889.301 10157.679
> 81920.0 81920.0  0.0   4097.0 655360.0 170916.8 7569408.0  7202855.2  43280.0 25855.8   5271  268.378 5006  9889.464 10157.842
> 81920.0 81920.0  0.0   4097.0 655360.0 170923.6 7569408.0  7202855.2  43280.0 25855.8   5271  268.378 5006  9889.464 10157.842
> 81920.0 81920.0  0.0   4097.0 655360.0 170925.6 7569408.0  7202855.2  43280.0 25855.8   5271  268.378 5006  9889.464 10157.842
> 81920.0 81920.0  0.0   4097.0 655360.0 171009.8 7569408.0  7202855.2  43280.0 25855.8   5271  268.378 5007  9889.660 10158.037
> 81920.0 81920.0  0.0   4097.0 655360.0 171249.8 7569408.0  7202855.2  43280.0 25855.8   5271  268.378 5008  9889.823 10158.201
> 81920.0 81920.0  0.0   4097.0 655360.0 171518.8 7569408.0  7202855.2  43280.0 25855.8   5271  268.378 5008  9889.823 10158.201
> 81920.0 81920.0  0.0   4097.0 655360.0 171530.6 7569408.0  7202855.2  43280.0 25855.8   5271  268.378 5008  9889.823 10158.201
> 81920.0 81920.0  0.0   4097.0 655360.0 171549.2 7569408.0  7202855.2  43280.0 25855.8   5271  268.378 5009  9890.024 10158.402
> 81920.0 81920.0  0.0   4097.0 655360.0 173035.0 7569408.0  7202850.4  43280.0 25855.8   5271  268.378 5010  9890.190 10158.568
> 81920.0 81920.0  0.0   4097.0 655360.0 173367.8 7569408.0  7202850.4  43280.0 25855.8   5271  268.378 5010  9890.190 10158.568
> 81920.0 81920.0  0.0   4097.0 655360.0 173376.8 7569408.0  7202850.4  43280.0 25855.8   5271  268.378 5010  9890.190 10158.568
> 81920.0 81920.0  0.0   4097.0 655360.0 173395.5 7569408.0  7202850.4  43280.0 25855.8   5271  268.378 5011  9890.456 10158.834
> 81920.0 81920.0  0.0   4097.0 655360.0 173451.8 7569408.0  7202850.3  43280.0 25855.8   5271  268.378 5011  9890.456 10158.834
> 81920.0 81920.0  0.0   4097.0 655360.0 173933.5 7569408.0  7202850.3  43280.0 25855.8   5271  268.378 5012  9890.622 10158.999
> 81920.0 81920.0  0.0   4097.0 655360.0 173969.7 7569408.0  7202850.3  43280.0 25855.8   5271  268.378 5012  9890.622 10158.999
> 81920.0 81920.0  0.0   4097.0 655360.0 173988.3 7569408.0  7202850.3  43280.0 25855.8   5271  268.378 5013  9890.820 10159.197
> 81920.0 81920.0  0.0   4097.0 655360.0 174053.8 7569408.0  7202850.2  43280.0 25855.8   5271  268.378 5013  9890.820 10159.197
> 81920.0 81920.0  0.0   4097.0 655360.0 174286.1 7569408.0  7202850.2  43280.0 25855.8   5271  268.378 5014  9890.986 10159.363
> 81920.0 81920.0  0.0   4097.0 655360.0 174551.0 7569408.0  7202850.2  43280.0 25855.8   5271  268.378 5014  9890.986 10159.363
> 81920.0 81920.0  0.0   4097.0 655360.0 174572.0 7569408.0  7202850.2  43280.0 25855.8   5271  268.378 5015  9890.986 10159.363
> 81920.0 81920.0  0.0   4097.0 655360.0 174592.9 7569408.0  7202850.1  43280.0 25855.8   5271  268.378 5015  9891.185 10159.562
> 81920.0 81920.0  0.0   4097.0 655360.0 174637.4 7569408.0  7202850.1  43280.0 25855.8   5271  268.378 5016  9891.350 10159.728
> 81920.0 81920.0  0.0   4097.0 655360.0 175049.8 7569408.0  7202850.1  43280.0 25855.8   5271  268.378 5016  9891.350 10159.728
> 81920.0 81920.0  0.0   4097.0 655360.0 175130.7 7569408.0  7202850.1  43280.0 25855.8   5271  268.378 5016  9891.350 10159.728
> 81920.0 81920.0  0.0   4097.0 655360.0 175168.4 7569408.0  7202850.1  43280.0 25855.8   5271  268.378 5017  9891.554 10159.932
> 81920.0 81920.0  0.0   4097.0 655360.0 175182.2 7569408.0  7202850.1  43280.0 25855.8   5271  268.378 5018  9891.721 10160.098
> 81920.0 81920.0  0.0   4097.0 655360.0 175402.9 7569408.0  7202850.1  43280.0 25855.8   5271  268.378 5018  9891.721 10160.098
> 81920.0 81920.0  0.0   4097.0 655360.0 175723.5 7569408.0  7202850.1  43280.0 25855.8   5271  268.378 5018  9891.721 10160.098
> 81920.0 81920.0  0.0   4097.0 655360.0 175742.2 7569408.0  7202850.1  43280.0 25855.8   5271  268.378 5019  9891.921 10160.298
> 81920.0 81920.0  0.0   4097.0 655360.0 175778.9 7569408.0  7202850.0  43280.0 25855.8   5271  268.378 5020  9892.089 10160.466
> 81920.0 81920.0  0.0   4097.0 655360.0 175823.6 7569408.0  7202850.0  43280.0 25855.8   5271  268.378 5020  9892.089 10160.466
> 81920.0 81920.0  0.0   4097.0 655360.0 176315.6 7569408.0  7202850.0  43280.0 25855.8   5271  268.378 5020  9892.089 10160.466
> 81920.0 81920.0  0.0   4097.0 655360.0 176351.6 7569408.0  7202850.0  43280.0 25855.8   5271  268.378 5021  9892.291 10160.669
> 81920.0 81920.0  0.0   4097.0 655360.0 176351.6 7569408.0  7202849.9  43280.0 25855.8   5271  268.378 5022  9892.460 10160.838
> 81920.0 81920.0  0.0   4097.0 655360.0 176370.6 7569408.0  7202849.9  43280.0 25855.8   5271  268.378 5022  9892.460 10160.838
> 81920.0 81920.0  0.0   4097.0 655360.0 176846.1 7569408.0  7202849.9  43280.0 25855.8   5271  268.378 5022  9892.460 10160.838
> 81920.0 81920.0  0.0   4097.0 655360.0 176865.2 7569408.0  7202849.9  43280.0 25855.8   5271  268.378 5023  9892.665 10161.043
> 81920.0 81920.0  0.0   4097.0 655360.0 176865.2 7569408.0  7202849.9  43280.0 25855.8   5271  268.378 5024  9892.834 10161.212
> 81920.0 81920.0  0.0   4097.0 655360.0 176942.8 7569408.0  7202849.9  43280.0 25855.8   5271  268.378 5024  9892.834 10161.212
> 81920.0 81920.0  0.0   4097.0 655360.0 177025.1 7569408.0  7202849.9  43280.0 25855.8   5271  268.378 5024  9892.834 10161.212
> 81920.0 81920.0  0.0   4097.0 655360.0 177437.4 7569408.0  7202849.9  43280.0 25855.8   5271  268.378 5025  9893.039 10161.416
> 81920.0 81920.0  0.0   4097.0 655360.0 177468.4 7569408.0  7202849.9  43280.0 25855.8   5271  268.378 5026  9893.213 10161.590
> 81920.0 81920.0  0.0   4097.0 655360.0 177472.6 7569408.0  7202849.9  43280.0 25855.8   5271  268.378 5026  9893.213 10161.590
> 81920.0 81920.0  0.0   4097.0 655360.0 177556.8 7569408.0  7202849.9  43280.0 25855.8   5271  268.378 5026  9893.213 10161.590
> 81920.0 81920.0  0.0   4097.0 655360.0 178032.6 7569408.0  7202849.9  43280.0 25855.8   5271  268.378 5027  9893.454 10161.832
> 81920.0 81920.0  0.0   4097.0 655360.0 178059.4 7569408.0  7202849.8  43280.0 25855.8   5271  268.378 5028  9893.624 10162.001
> 81920.0 81920.0  0.0   4097.0 655360.0 178059.4 7569408.0  7202849.8  43280.0 25855.8   5271  268.378 5028  9893.624 10162.001
> 81920.0 81920.0  0.0   4097.0 655360.0 178059.4 7569408.0  7202849.8  43280.0 25855.8   5271  268.378 5028  9893.624 10162.001
> 81920.0 81920.0  0.0   4097.0 655360.0 178300.8 7569408.0  7202849.8  43280.0 25855.8   5271  268.378 5029  9893.901 10162.279
> 81920.0 81920.0  0.0   4097.0 655360.0 178630.4 7569408.0  7202849.7  43280.0 25855.8   5271  268.378 5030  9894.075 10162.453
> 81920.0 81920.0  0.0   4097.0 655360.0 178638.6 7569408.0  7202849.7  43280.0 25855.8   5271  268.378 5030  9894.075 10162.453
> 81920.0 81920.0  0.0   4097.0 655360.0 178638.6 7569408.0  7202849.7  43280.0 25855.8   5271  268.378 5030  9894.075 10162.453
> 81920.0 81920.0  0.0   4097.0 655360.0 178734.5 7569408.0  7202849.7  43280.0 25855.8   5271  268.378 5031  9894.294 10162.672
> 81920.0 81920.0  0.0   4097.0 655360.0 179272.2 7569408.0  7202849.7  43280.0 25855.8   5271  268.378 5032  9894.468 10162.845
> 81920.0 81920.0  0.0   4097.0 655360.0 179308.5 7569408.0  7202849.7  43280.0 25855.8   5271  268.378 5032  9894.468 10162.845
> 81920.0 81920.0  0.0   4097.0 655360.0 179308.5 7569408.0  7202849.7  43280.0 25855.8   5271  268.378 5032  9894.468 10162.845
> 81920.0 81920.0  0.0   4097.0 655360.0 179308.5 7569408.0  7202849.7  43280.0 25855.8   5271  268.378 5033  9894.681 10163.058
> 81920.0 81920.0  0.0   4097.0 655360.0 179784.0 7569408.0  7202849.6  43280.0 25855.8   5271  268.378 5033  9894.681 10163.058
> 81920.0 81920.0  0.0   4097.0 655360.0 179804.6 7569408.0  7202849.6  43280.0 25855.8   5271  268.378 5034  9894.854 10163.231
> 81920.0 81920.0  0.0   4097.0 655360.0 181033.4 7569408.0  7202849.6  43280.0 25855.8   5271  268.378 5034  9894.854 10163.231
> 81920.0 81920.0  0.0   4097.0 655360.0 181033.4 7569408.0  7202849.6  43280.0 25855.8   5271  268.378 5035  9895.135 10163.513
> 81920.0 81920.0  0.0   4097.0 655360.0 181052.0 7569408.0  7202849.5  43280.0 25855.8   5271  268.378 5035  9895.135 10163.513
> 81920.0 81920.0  0.0   4097.0 655360.0 181509.0 7569408.0  7202849.5  43280.0 25855.8   5271  268.378 5036  9895.311 10163.689
> 81920.0 81920.0  0.0   4097.0 655360.0 181608.5 7569408.0  7202849.5  43280.0 25855.8   5271  268.378 5036  9895.311 10163.689
> 81920.0 81920.0  0.0   4097.0 655360.0 181610.7 7569408.0  7202849.5  43280.0 25855.8   5271  268.378 5037  9895.311 10163.689
> 81920.0 81920.0  0.0   4097.0 655360.0 181640.3 7569408.0  7202849.5  43280.0 25855.8   5271  268.378 5037  9895.539 10163.917
> 81920.0 81920.0  0.0   4097.0 655360.0 182116.0 7569408.0  7202849.5  43280.0 25855.8   5271  268.378 5038  9895.716 10164.093
> 81920.0 81920.0  0.0   4097.0 655360.0 182207.3 7569408.0  7202849.5  43280.0 25855.8   5271  268.378 5038  9895.716 10164.093
> 81920.0 81920.0  0.0   4097.0 655360.0 182281.0 7569408.0  7202849.5  43280.0 25855.8   5271  268.378 5038  9895.716 10164.093
> 81920.0 81920.0  0.0   4097.0 655360.0 182281.0 7569408.0  7202849.5  43280.0 25855.8   5271  268.378 5039  9896.000 10164.378
> 81920.0 81920.0  0.0   4097.0 655360.0 182285.1 7569408.0  7202849.5  43280.0 25855.8   5271  268.378 5040  9896.180 10164.558
> 81920.0 81920.0  0.0   4097.0 655360.0 182844.7 7569408.0  7202849.5  43280.0 25855.8   5271  268.378 5040  9896.180 10164.558
> 81920.0 81920.0  0.0   4097.0 655360.0 182872.0 7569408.0  7202849.5  43280.0 25855.8   5271  268.378 5040  9896.180 10164.558
> 81920.0 81920.0  0.0   4097.0 655360.0 182872.0 7569408.0  7202849.5  43280.0 25855.8   5271  268.378 5041  9896.463 10164.841
> 81920.0 81920.0  0.0   4097.0 655360.0 182872.0 7569408.0  7202849.3  43280.0 25855.8   5271  268.378 5042  9896.641 10165.019
> 81920.0 81920.0  0.0   4097.0 655360.0 183413.3 7569408.0  7202849.3  43280.0 25855.8   5271  268.378 5042  9896.641 10165.019
> 81920.0 81920.0  0.0   4097.0 655360.0 183443.4 7569408.0  7202849.3  43280.0 25855.8   5271  268.378 5042  9896.641 10165.019
> 81920.0 81920.0  0.0   4097.0 655360.0 183464.8 7569408.0  7202849.3  43280.0 25855.8   5271  268.378 5043  9896.857 10165.235
> 81920.0 81920.0  0.0   4097.0 655360.0 183476.6 7569408.0  7202849.2  43280.0 25855.8   5271  268.378 5044  9897.033 10165.411
> 81920.0 81920.0  0.0   4097.0 655360.0 183733.3 7569408.0  7202849.2  43280.0 25855.8   5271  268.378 5044  9897.033 10165.411
> 81920.0 81920.0  0.0   4097.0 655360.0 184045.6 7569408.0  7202849.2  43280.0 25855.8   5271  268.378 5044  9897.033 10165.411
> 81920.0 81920.0  0.0   4097.0 655360.0 184055.9 7569408.0  7202849.2  43280.0 25855.8   5271  268.378 5045  9897.314 10165.691
> 81920.0 81920.0  0.0   4097.0 655360.0 184057.9 7569408.0  7202849.2  43280.0 25855.8   5271  268.378 5046  9897.492 10165.869
> 81920.0 81920.0  0.0   4097.0 655360.0 184057.9 7569408.0  7202849.2  43280.0 25855.8   5271  268.378 5046  9897.492 10165.869
> 81920.0 81920.0  0.0   4097.0 655360.0 184618.0 7569408.0  7202849.2  43280.0 25855.8   5271  268.378 5046  9897.492 10165.869
> 81920.0 81920.0  0.0   4097.0 655360.0 184644.8 7569408.0  7202849.2  43280.0 25855.8   5271  268.378 5047  9897.706 10166.084
> 81920.0 81920.0  0.0   4097.0 655360.0 184647.0 7569408.0  7202849.1  43280.0 25855.8   5271  268.378 5048  9897.886 10166.264
> 81920.0 81920.0  0.0   4097.0 655360.0 184647.0 7569408.0  7202849.1  43280.0 25855.8   5271  268.378 5048  9897.886 10166.264
> 81920.0 81920.0  0.0   4097.0 655360.0 185105.9 7569408.0  7202849.1  43280.0 25855.8   5271  268.378 5048  9897.886 10166.264
> 81920.0 81920.0  0.0   4097.0 655360.0 185231.6 7569408.0  7202849.1  43280.0 25855.8   5271  268.378 5049  9898.095 10166.472
> 81920.0 81920.0  0.0   4097.0 655360.0 185239.8 7569408.0  7202849.1  43280.0 25855.8   5271  268.378 5050  9898.276 10166.654
> 81920.0 81920.0  0.0   4097.0 655360.0 185251.5 7569408.0  7202849.1  43280.0 25855.8   5271  268.378 5050  9898.276 10166.654
> 81920.0 81920.0  0.0   4097.0 655360.0 194817.6 7569408.0  7202849.1  43280.0 25855.8   5271  268.378 5050  9898.276 10166.654
> 81920.0 81920.0  0.0   4097.0 655360.0 195326.2 7569408.0  7202849.1  43280.0 25855.8   5271  268.378 5051  9898.570 10166.948
> 81920.0 81920.0  0.0   4097.0 655360.0 195333.0 7569408.0  7202849.0  43280.0 25855.8   5271  268.378 5052  9898.752 10167.130
> 81920.0 81920.0  0.0   4097.0 655360.0 195341.3 7569408.0  7202849.0  43280.0 25855.8   5271  268.378 5052  9898.752 10167.130
> 81920.0 81920.0  0.0   4097.0 655360.0 195341.3 7569408.0  7202849.0  43280.0 25855.8   5271  268.378 5052  9898.752 10167.130
> 81920.0 81920.0  0.0   4097.0 655360.0 196751.1 7569408.0  7202849.0  43280.0 25855.8   5271  268.378 5053  9899.045 10167.423
> 81920.0 81920.0  0.0   4097.0 655360.0 196934.4 7569408.0  7202848.8  43280.0 25855.8   5271  268.378 5053  9899.045 10167.423
> 81920.0 81920.0  0.0   4097.0 655360.0 196944.7 7569408.0  7202848.8  43280.0 25855.8   5271  268.378 5054  9899.230 10167.608
> 81920.0 81920.0  0.0   4097.0 655360.0 197021.9 7569408.0  7202848.8  43280.0 25855.8   5271  268.378 5054  9899.230 10167.608
> 81920.0 81920.0  0.0   4097.0 655360.0 197037.8 7569408.0  7202848.8  43280.0 25855.8   5271  268.378 5055  9899.451 10167.829
> 81920.0 81920.0  0.0   4097.0 655360.0 197541.2 7569408.0  7202848.7  43280.0 25855.8   5271  268.378 5055  9899.451 10167.829
> 81920.0 81920.0  0.0   4097.0 655360.0 197541.2 7569408.0  7202848.7  43280.0 25855.8   5271  268.378 5056  9899.638 10168.015
> 81920.0 81920.0  0.0   4097.0 655360.0 197614.9 7569408.0  7202848.7  43280.0 25855.8   5271  268.378 5056  9899.638 10168.015
> 81920.0 81920.0  0.0   4097.0 655360.0 197617.0 7569408.0  7202848.7  43280.0 25855.8   5271  268.378 5057  9899.858 10168.235
> 81920.0 81920.0  0.0   4097.0 655360.0 198123.4 7569408.0  7202848.7  43280.0 25855.8   5271  268.378 5057  9899.858 10168.235
> 81920.0 81920.0  0.0   4097.0 655360.0 198130.2 7569408.0  7202848.7  43280.0 25855.8   5271  268.378 5058  9900.043 10168.420
> 81920.0 81920.0  0.0   4097.0 655360.0 198140.7 7569408.0  7202848.7  43280.0 25855.8   5271  268.378 5058  9900.043 10168.420
> 81920.0 81920.0  0.0   4097.0 655360.0 199340.2 7569408.0  7202848.7  43280.0 25855.8   5271  268.378 5059  9900.340 10168.718
> 81920.0 81920.0  0.0   4097.0 655360.0 199580.3 7569408.0  7202848.6  43280.0 25855.8   5271  268.378 5059  9900.340 10168.718
> 81920.0 81920.0  0.0   4097.0 655360.0 199843.5 7569408.0  7202848.6  43280.0 25855.8   5271  268.378 5060  9900.528 10168.905
> 81920.0 81920.0  0.0   4097.0 655360.0 199917.3 7569408.0  7202848.6  43280.0 25855.8   5271  268.378 5060  9900.528 10168.905
> 81920.0 81920.0  0.0   4097.0 655360.0 200007.7 7569408.0  7202848.6  43280.0 25855.8   5271  268.378 5061  9900.528 10168.905
> 81920.0 81920.0  0.0   4097.0 655360.0 200009.8 7569408.0  7202848.6  43280.0 25855.8   5271  268.378 5061  9900.818 10169.196
> 81920.0 81920.0  0.0   4097.0 655360.0 200466.7 7569408.0  7202848.6  43280.0 25855.8   5271  268.378 5062  9901.005 10169.383
> 81920.0 81920.0  0.0   4097.0 655360.0 200513.2 7569408.0  7202848.6  43280.0 25855.8   5271  268.378 5062  9901.005 10169.383
> 81920.0 81920.0  0.0   4097.0 655360.0 200521.4 7569408.0  7202848.6  43280.0 25855.8   5271  268.378 5062  9901.005 10169.383
> 81920.0 81920.0  0.0   4097.0 655360.0 200521.4 7569408.0  7202848.5  43280.0 25855.8   5271  268.378 5063  9901.302 10169.680
> 81920.0 81920.0  0.0   4097.0 655360.0 200759.4 7569408.0  7202848.5  43280.0 25855.8   5271  268.378 5064  9901.490 10169.868
> 81920.0 81920.0  0.0   4097.0 655360.0 201018.1 7569408.0  7202848.5  43280.0 25855.8   5271  268.378 5064  9901.490 10169.868
> 81920.0 81920.0  0.0   4097.0 655360.0 201045.0 7569408.0  7202848.5  43280.0 25855.8   5271  268.378 5064  9901.490 10169.868
> 81920.0 81920.0  0.0   4097.0 655360.0 201112.5 7569408.0  7202848.5  43280.0 25855.8   5271  268.378 5065  9901.710 10170.087
> 81920.0 81920.0  0.0   4097.0 655360.0 201112.5 7569408.0  7202848.5  43280.0 25855.8   5271  268.378 5066  9901.900 10170.277
> 81920.0 81920.0  0.0   4097.0 655360.0 201590.6 7569408.0  7202848.5  43280.0 25855.8   5271  268.378 5066  9901.900 10170.277
> 81920.0 81920.0  0.0   4097.0 655360.0 201613.2 7569408.0  7202848.5  43280.0 25855.8   5271  268.378 5066  9901.900 10170.277
> 81920.0 81920.0  0.0   4097.0 655360.0 201640.1 7569408.0  7202848.5  43280.0 25855.8   5271  268.378 5067  9902.200 10170.578
> 81920.0 81920.0  0.0   4097.0 655360.0 201717.4 7569408.0  7202848.4  43280.0 25855.8   5271  268.378 5068  9902.390 10170.768
> 81920.0 81920.0  0.0   4097.0 655360.0 202023.1 7569408.0  7202848.4  43280.0 25855.8   5271  268.378 5068  9902.390 10170.768
> 81920.0 81920.0  0.0   4097.0 655360.0 202267.7 7569408.0  7202848.4  43280.0 25855.8   5271  268.378 5068  9902.390 10170.768
> 81920.0 81920.0  0.0   4097.0 655360.0 202278.1 7569408.0  7202848.4  43280.0 25855.8   5271  268.378 5069  9902.693 10171.071
> 81920.0 81920.0  0.0   4097.0 655360.0 202362.5 7569408.0  7202848.4  43280.0 25855.8   5271  268.378 5070  9902.883 10171.261
> 81920.0 81920.0  0.0   4097.0 655360.0 202600.7 7569408.0  7202848.4  43280.0 25855.8   5271  268.378 5070  9902.883 10171.261
> 81920.0 81920.0  0.0   4097.0 655360.0 202854.2 7569408.0  7202848.4  43280.0 25855.8   5271  268.378 5070  9902.883 10171.261
> 81920.0 81920.0  0.0   4097.0 655360.0 202926.5 7569408.0  7202848.3  43280.0 25855.8   5271  268.378 5071  9903.113 10171.491
> 81920.0 81920.0  0.0   4097.0 655360.0 202934.8 7569408.0  7202848.2  43280.0 25855.8   5271  268.378 5072  9903.303 10171.681
> 81920.0 81920.0  0.0   4097.0 655360.0 202955.5 7569408.0  7202848.2  43280.0 25855.8   5271  268.378 5072  9903.303 10171.681
> 81920.0 81920.0  0.0   4097.0 655360.0 203291.3 7569408.0  7202848.2  43280.0 25855.8   5271  268.378 5072  9903.303 10171.681
> 81920.0 81920.0  0.0   4097.0 655360.0 203517.0 7569408.0  7202848.2  43280.0 25855.8   5271  268.378 5073  9903.607 10171.985
> 81920.0 81920.0  0.0   4097.0 655360.0 203539.0 7569408.0  7202848.0  43280.0 25855.8   5271  268.378 5074  9903.799 10172.177
> 81920.0 81920.0  0.0   4097.0 655360.0 203557.6 7569408.0  7202848.0  43280.0 25855.8   5271  268.378 5074  9903.799 10172.177
> 81920.0 81920.0  0.0   4097.0 655360.0 203795.7 7569408.0  7202848.0  43280.0 25855.8   5271  268.378 5074  9903.799 10172.177
> 81920.0 81920.0  0.0   4097.0 655360.0 204042.6 7569408.0  7202848.0  43280.0 25855.8   5271  268.378 5075  9904.025 10172.403
> 81920.0 81920.0  0.0   4097.0 655360.0 204042.6 7569408.0  7202848.0  43280.0 25855.8   5271  268.378 5076  9904.220 10172.598
> 81920.0 81920.0  0.0   4097.0 655360.0 204135.0 7569408.0  7202848.0  43280.0 25855.8   5271  268.378 5076  9904.220 10172.598
> 81920.0 81920.0  0.0   4097.0 655360.0 204137.0 7569408.0  7202848.0  43280.0 25855.8   5271  268.378 5076  9904.220 10172.598
> 81920.0 81920.0  0.0   4097.0 655360.0 204582.2 7569408.0  7202848.0  43280.0 25855.8   5271  268.378 5077  9904.451 10172.829
> 81920.0 81920.0  0.0   4097.0 655360.0 204633.6 7569408.0  7202847.9  43280.0 25855.8   5271  268.378 5078  9904.644 10173.022
> 81920.0 81920.0  0.0   4097.0 655360.0 204660.4 7569408.0  7202847.9  43280.0 25855.8   5271  268.378 5078  9904.644 10173.022
> 81920.0 81920.0  0.0   4097.0 655360.0 204741.6 7569408.0  7202847.9  43280.0 25855.8   5271  268.378 5078  9904.644 10173.022
> 81920.0 81920.0  0.0   4097.0 655360.0 205000.7 7569408.0  7202847.9  43280.0 25855.8   5271  268.378 5079  9904.952 10173.329
> 81920.0 81920.0  0.0   4097.0 655360.0 205226.3 7569408.0  7202847.8  43280.0 25855.8   5271  268.378 5080  9904.952 10173.329
> 81920.0 81920.0  0.0   4097.0 655360.0 205228.5 7569408.0  7202847.8  43280.0 25855.8   5271  268.378 5080  9905.144 10173.521
> 81920.0 81920.0  0.0   4097.0 655360.0 205267.2 7569408.0  7202847.8  43280.0 25855.8   5271  268.378 5080  9905.144 10173.521
> 81920.0 81920.0  0.0   4097.0 655360.0 205267.2 7569408.0  7202847.8  43280.0 25855.8   5271  268.378 5081  9905.369 10173.746
> 81920.0 81920.0  0.0   4097.0 655360.0 205698.6 7569408.0  7202847.8  43280.0 25855.8   5271  268.378 5081  9905.369 10173.746
> 81920.0 81920.0  0.0   4097.0 655360.0 205752.3 7569408.0  7202847.8  43280.0 25855.8   5271  268.378 5082  9905.563 10173.940
> 81920.0 81920.0  0.0   4097.0 655360.0 205844.6 7569408.0  7202847.8  43280.0 25855.8   5271  268.378 5082  9905.563 10173.940
> 81920.0 81920.0  0.0   4097.0 655360.0 205846.6 7569408.0  7202847.8  43280.0 25855.8   5271  268.378 5083  9905.797 10174.174
> 81920.0 81920.0  0.0   4097.0 655360.0 206115.6 7569408.0  7202847.8  43280.0 25855.8   5271  268.378 5083  9905.797 10174.174
> 81920.0 81920.0  0.0   4097.0 655360.0 206406.6 7569408.0  7202847.8  43280.0 25855.8   5271  268.378 5084  9905.992 10174.369
> 81920.0 81920.0  0.0   4097.0 655360.0 207640.6 7569408.0  7202847.8  43280.0 25855.8   5271  268.378 5084  9905.992 10174.369
> 81920.0 81920.0  0.0   4097.0 655360.0 207650.8 7569408.0  7202847.8  43280.0 25855.8   5271  268.378 5085  9906.220 10174.598
> 81920.0 81920.0  0.0   4097.0 655360.0 207908.0 7569408.0  7202847.7  43280.0 25855.8   5271  268.378 5085  9906.220 10174.598
> 81920.0 81920.0  0.0   4097.0 655360.0 208149.9 7569408.0  7202847.7  43280.0 25855.8   5271  268.378 5086  9906.417 10174.794
> 81920.0 81920.0  0.0   4097.0 655360.0 208201.3 7569408.0  7202847.7  43280.0 25855.8   5271  268.378 5086  9906.417 10174.794
> 81920.0 81920.0  0.0   4097.0 655360.0 208240.0 7569408.0  7202847.7  43280.0 25855.8   5271  268.378 5087  9906.417 10174.794
> 81920.0 81920.0  0.0   4097.0 655360.0 208240.0 7569408.0  7202847.6  43280.0 25855.8   5271  268.378 5087  9906.654 10175.031
> 81920.0 81920.0  0.0   4097.0 655360.0 208501.2 7569408.0  7202847.6  43280.0 25855.8   5271  268.378 5088  9906.853 10175.230
> 81920.0 81920.0  0.0   4097.0 655360.0 208726.8 7569408.0  7202847.6  43280.0 25855.8   5271  268.378 5088  9906.853 10175.230
> 81920.0 81920.0  0.0   4097.0 655360.0 208726.8 7569408.0  7202847.6  43280.0 25855.8   5271  268.378 5088  9906.853 10175.230
> 81920.0 81920.0  0.0   4097.0 655360.0 208821.1 7569408.0  7202847.6  43280.0 25855.8   5271  268.378 5089  9907.164 10175.542
> 81920.0 81920.0  0.0   4097.0 655360.0 209059.2 7569408.0  7202847.6  43280.0 25855.8   5271  268.378 5090  9907.365 10175.743
> 81920.0 81920.0  0.0   4097.0 655360.0 209398.1 7569408.0  7202847.6  43280.0 25855.8   5271  268.378 5090  9907.365 10175.743
> 81920.0 81920.0  0.0   4097.0 655360.0 209400.4 7569408.0  7202847.6  43280.0 25855.8   5271  268.378 5090  9907.365 10175.743
> 81920.0 81920.0  0.0   4097.0 655360.0 209408.6 7569408.0  7202847.6  43280.0 25855.8   5271  268.378 5091  9907.596 10175.973
> 81920.0 81920.0  0.0   4097.0 655360.0 209427.3 7569408.0  7202847.5  43280.0 25855.8   5271  268.378 5092  9907.793 10176.171
> 81920.0 81920.0  0.0   4097.0 655360.0 209684.5 7569408.0  7202847.5  43280.0 25855.8   5271  268.378 5092  9907.793 10176.171
> 81920.0 81920.0  0.0   4097.0 655360.0 209910.1 7569408.0  7202847.5  43280.0 25855.8   5271  268.378 5092  9907.793 10176.171
> 81920.0 81920.0  0.0   4097.0 655360.0 209932.1 7569408.0  7202847.5  43280.0 25855.8   5271  268.378 5093  9908.028 10176.405
> 81920.0 81920.0  0.0   4097.0 655360.0 209952.8 7569408.0  7202847.4  43280.0 25855.8   5271  268.378 5094  9908.225 10176.603
> 81920.0 81920.0  0.0   4097.0 655360.0 210190.9 7569408.0  7202847.4  43280.0 25855.8   5271  268.378 5094  9908.225 10176.603
> 81920.0 81920.0  0.0   4097.0 655360.0 210500.9 7569408.0  7202847.4  43280.0 25855.8   5271  268.378 5094  9908.225 10176.603
> 81920.0 81920.0  0.0   4097.0 655360.0 210500.9 7569408.0  7202847.4  43280.0 25855.8   5271  268.378 5095  9908.470 10176.848
> 81920.0 81920.0  0.0   4097.0 655360.0 210509.1 7569408.0  7202847.4  43280.0 25855.8   5271  268.378 5096  9908.668 10177.046
> 81920.0 81920.0  0.0   4097.0 655360.0 210785.8 7569408.0  7202847.4  43280.0 25855.8   5271  268.378 5096  9908.668 10177.046
> 81920.0 81920.0  0.0   4097.0 655360.0 210818.8 7569408.0  7202847.4  43280.0 25855.8   5271  268.378 5096  9908.668 10177.046
> 81920.0 81920.0  0.0   4097.0 655360.0 211109.8 7569408.0  7202847.4  43280.0 25855.8   5271  268.378 5097  9908.902 10177.279
> 81920.0 81920.0  0.0   4097.0 655360.0 211175.3 7569408.0  7202847.4  43280.0 25855.8   5271  268.378 5098  9909.103 10177.481
> 81920.0 81920.0  0.0   4097.0 655360.0 211267.7 7569408.0  7202847.4  43280.0 25855.8   5271  268.378 5098  9909.103 10177.481
> 81920.0 81920.0  0.0   4097.0 655360.0 211527.0 7569408.0  7202847.4  43280.0 25855.8   5271  268.378 5098  9909.103 10177.481
> 81920.0 81920.0  0.0   4097.0 655360.0 211754.7 7569408.0  7202847.4  43280.0 25855.8   5271  268.378 5099  9909.341 10177.718
> 81920.0 81920.0  0.0   4097.0 655360.0 211766.4 7569408.0  7202847.2  43280.0 25855.8   5271  268.378 5100  9909.542 10177.920
> 81920.0 81920.0  0.0   4097.0 655360.0 211840.2 7569408.0  7202847.2  43280.0 25855.8   5271  268.378 5100  9909.542 10177.920
> 81920.0 81920.0  0.0   4097.0 655360.0 212116.0 7569408.0  7202847.2  43280.0 25855.8   5271  268.378 5100  9909.542 10177.920
> 81920.0 81920.0  0.0   4097.0 655360.0 212122.8 7569408.0  7202847.2  43280.0 25855.8   5271  268.378 5101  9909.783 10178.161
> 81920.0 81920.0  0.0   4097.0 655360.0 212411.4 7569408.0  7202847.2  43280.0 25855.8   5271  268.378 5102  9909.783 10178.161
> 81920.0 81920.0  0.0   4097.0 655360.0 212424.6 7569408.0  7202847.2  43280.0 25855.8   5271  268.378 5102  9909.987 10178.365
> 81920.0 81920.0  0.0   4097.0 655360.0 212432.8 7569408.0  7202847.2  43280.0 25855.8   5271  268.378 5102  9909.987 10178.365
> 81920.0 81920.0  0.0   4097.0 655360.0 212720.2 7569408.0  7202847.2  43280.0 25855.8   5271  268.378 5103  9910.313 10178.690
> 81920.0 81920.0  0.0   4097.0 655360.0 212945.8 7569408.0  7202847.1  43280.0 25855.8   5271  268.378 5103  9910.313 10178.690
> 81920.0 81920.0  0.0   4097.0 655360.0 212945.8 7569408.0  7202847.1  43280.0 25855.8   5271  268.378 5104  9910.516 10178.893
> 81920.0 81920.0  0.0   4097.0 655360.0 212958.1 7569408.0  7202847.1  43280.0 25855.8   5271  268.378 5104  9910.516 10178.893
> 81920.0 81920.0  0.0   4097.0 655360.0 213301.3 7569408.0  7202847.1  43280.0 25855.8   5271  268.378 5105  9910.851 10179.229
> 81920.0 81920.0  0.0   4097.0 655360.0 213482.3 7569408.0  7202847.0  43280.0 25855.8   5271  268.378 5105  9910.851 10179.229
> 81920.0 81920.0  0.0   4097.0 655360.0 213527.0 7569408.0  7202847.0  43280.0 25855.8   5271  268.378 5106  9911.056 10179.433
> 81920.0 81920.0  0.0   4097.0 655360.0 213547.0 7569408.0  7202847.0  43280.0 25855.8   5271  268.378 5106  9911.056 10179.433
> 81920.0 81920.0  0.0   4097.0 655360.0 213612.5 7569408.0  7202847.0  43280.0 25855.8   5271  268.378 5107  9911.307 10179.684
> 81920.0 81920.0  0.0   4097.0 655360.0 213892.6 7569408.0  7202847.0  43280.0 25855.8   5271  268.378 5107  9911.307 10179.684
> 81920.0 81920.0  0.0   4097.0 655360.0 214133.8 7569408.0  7202847.0  43280.0 25855.8   5271  268.378 5108  9911.513 10179.891
> 81920.0 81920.0  0.0   4097.0 655360.0 214133.8 7569408.0  7202847.0  43280.0 25855.8   5271  268.378 5108  9911.513 10179.891
> 81920.0 81920.0  0.0   4097.0 655360.0 215407.3 7569408.0  7202847.0  43280.0 25855.8   5271  268.378 5108  9911.513 10179.891
> 81920.0 81920.0  0.0   4097.0 655360.0 215677.9 7569408.0  7202846.9  43280.0 25855.8   5271  268.378 5109  9911.822 10180.199
> 81920.0 81920.0  0.0   4097.0 655360.0 215879.9 7569408.0  7202846.9  43280.0 25855.8   5271  268.378 5110  9912.026 10180.404
> 81920.0 81920.0  0.0   4097.0 655360.0 215924.5 7569408.0  7202846.9  43280.0 25855.8   5271  268.378 5110  9912.026 10180.404
> 81920.0 81920.0  0.0   4097.0 655360.0 215932.7 7569408.0  7202846.9  43280.0 25855.8   5271  268.378 5110  9912.026 10180.404
> 81920.0 81920.0  0.0   4097.0 655360.0 216189.6 7569408.0  7202846.9  43280.0 25855.8   5271  268.378 5111  9912.347 10180.725
> 81920.0 81920.0  0.0   4097.0 655360.0 216208.6 7569408.0  7202846.9  43280.0 25855.8   5271  268.378 5112  9912.556 10180.934
> 81920.0 81920.0  0.0   4097.0 655360.0 216504.1 7569408.0  7202846.9  43280.0 25855.8   5271  268.378 5112  9912.556 10180.934
> 81920.0 81920.0  0.0   4097.0 655360.0 216517.9 7569408.0  7202846.9  43280.0 25855.8   5271  268.378 5112  9912.556 10180.934
> {quote}
> This continues even after we shut down our app. Nothing is connected to Cassandra any more, yet it is still stuck in full GC and cannot free up memory.
> Running nodetool tpstats shows that nothing is pending, all seems OK:
> {quote}
> Pool Name                    Active   Pending      Completed   Blocked  All time blocked
> ReadStage                         0         0       69555935         0                 0
> RequestResponseStage              0         0              0         0                 0
> MutationStage                     0         0       73123690         0                 0
> ReadRepairStage                   0         0              0         0                 0
> ReplicateOnWriteStage             0         0              0         0                 0
> GossipStage                       0         0              0         0                 0
> CacheCleanupExecutor              0         0              0         0                 0
> MigrationStage                    0         0             46         0                 0
> MemoryMeter                       0         0           1125         0                 0
> FlushWriter                       0         0            824         0                30
> ValidationExecutor                0         0              0         0                 0
> InternalResponseStage             0         0             23         0                 0
> AntiEntropyStage                  0         0              0         0                 0
> MemtablePostFlusher               0         0           1783         0                 0
> MiscStage                         0         0              0         0                 0
> PendingRangeCalculator            0         0              1         0                 0
> CompactionExecutor                0         0          74330         0                 0
> commitlog_archiver                0         0              0         0                 0
> HintedHandoff                     0         0              0         0                 0
> Message type           Dropped
> RANGE_SLICE                  0
> READ_REPAIR                  0
> PAGED_RANGE                  0
> BINARY                       0
> READ                       585
> MUTATION                 75775
> _TRACE                       0
> REQUEST_RESPONSE             0
> COUNTER_MUTATION             0
> {quote}
> We had this happen on 2 separate boxes, one with 2.0.6, the other with 2.0.8.
> Right now this is a total blocker for us. We are unable to process the customer data and have to abort in the middle of large processing.
> This is a new customer, so we did not have a chance to see if this occurred with 1.1 or 1.2 in the past (we moved to 2.0 recently).
> We have the Cassandra process still running, pls let us know if there is anything else we could run to give you more insight.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message