Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cassandra.apache.org
Date: Wed, 22 Feb 2012 14:51:49 +0000 (UTC)
From: "Jonathan Ellis (Commented) (JIRA)" <jira@apache.org>
To: commits@cassandra.apache.org
Message-ID: 
 <1584584903.4399.1329922309404.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: 
 <498876066.21265.1322551540224.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (CASSANDRA-3537) ExpiringMap timer is not
 exception-proof
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


    [ https://issues.apache.org/jira/browse/CASSANDRA-3537?page=3Dcom.atlas=
sian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D=
13213670#comment-13213670 ]=20

Jonathan Ellis commented on CASSANDRA-3537:
-------------------------------------------

catching Throwable or Error is generally Bad because most Errors are not re=
coverable. OutOfMemoryError is the most common. We have a global exception =
hook set up to shut down the process for those.
               =20
> ExpiringMap timer is not exception-proof
> ----------------------------------------
>
>                 Key: CASSANDRA-3537
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3537
>             Project: Cassandra
>          Issue Type: Bug
>         Environment: =E5=BD=93=E5=89=8D=E5=A0=86=E5=A4=A7=E5=B0=8F:=C2=A0
> 5,815,955 Kb
> =E5=A0=86=E5=A4=A7=E5=B0=8F=E7=9A=84=E6=9C=80=E5=A4=A7=E5=80=BC:=C2=A0
> 6,045,696 Kb
> =E5=88=86=E9=85=8D=E7=9A=84=E5=86=85=E5=AD=98:=C2=A0
> 6,045,696 Kb
> =E6=9A=82=E6=8C=82=E7=BB=93=E6=9D=9F=E6=93=8D=E4=BD=9C:=C2=A0
> 0 =E4=B8=AA=E5=AF=B9=E8=B1=A1
> =E5=9E=83=E5=9C=BE=E6=94=B6=E9=9B=86=E5=99=A8:=C2=A0
> Name =3D 'ParNew', Collections =3D 3,294, Total time spent =3D 2 minutes
> =E5=9E=83=E5=9C=BE=E6=94=B6=E9=9B=86=E5=99=A8:=C2=A0
> Name =3D 'ConcurrentMarkSweep', Collections =3D 5,909, Total time spent =
=3D 2 hours 17 minutes
> =20
> =E6=93=8D=E4=BD=9C=E7=B3=BB=E7=BB=9F:=C2=A0
> Linux 2.6.32.12-0.7-default
> =E4=BD=93=E7=B3=BB=E7=BB=93=E6=9E=84:=C2=A0
> amd64
> =E5=A4=84=E7=90=86=E5=99=A8=E7=9A=84=E6=95=B0=E7=9B=AE:=C2=A0
> 16
> =E5=88=86=E9=85=8D=E7=9A=84=E8=99=9A=E6=8B=9F=E5=86=85=E5=AD=98:=C2=A0
> 42,748,416 Kb
> =E7=89=A9=E7=90=86=E5=86=85=E5=AD=98=E6=80=BB=E9=87=8F:=C2=A0
> 24,568,836 Kb
> =E5=8F=AF=E7=94=A8=E7=89=A9=E7=90=86=E5=86=85=E5=AD=98:=C2=A0
> =C2=A07,136,380 Kb
> =E4=BA=A4=E6=8D=A2=E7=A9=BA=E9=97=B4=E6=80=BB=E9=87=8F:=C2=A0
> =C2=A02,104,472 Kb
> =E5=8F=AF=E7=94=A8=E4=BA=A4=E6=8D=A2=E7=A9=BA=E9=97=B4:=C2=A0
> =C2=A01,970,800 Kb
> =20
> VM =E5=8F=82=E6=95=B0:=C2=A0
> -ea -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=3D42 -Xms6G -Xmx6G =
-Xmn2400M -XX:+HeapDumpOnOutOfMemoryError -Xss128k -XX:+UseParNewGC -XX:+Us=
eConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=3D8 -XX:Ma=
xTenuringThreshold=3D1 -XX:CMSInitiatingOccupancyFraction=3D75 -XX:+UseCMSI=
nitiatingOccupancyOnly -Djava.net.preferIPv4Stack=3Dtrue -Dcom.sun.manageme=
nt.jmxremote.port=3D9000 -Dcom.sun.management.jmxremote.ssl=3Dfalse -Dcom.s=
un.management.jmxremote.authenticate=3Dfalse -Dpasswd.properties=3D/opt/obs=
/cassandra/conf/passwd.properties -Dpasswd.mode=3DMD5 -Dlog4j.configuration=
=3Dlog4j-server.properties -Dlog4j.defaultInitOverride=3Dtrue=20
> =E7=B1=BB=E8=B7=AF=E5=BE=84:=C2=A0
> /opt/obs/cassandra/conf:/opt/obs/cassandra/build/classes/main:/opt/obs/ca=
ssandra/build/classes/thrift:/opt/obs/cassandra/lib/antlr-3.2.jar:/opt/obs/=
cassandra/lib/apache-cassandra-1.0.0.jar:/opt/obs/cassandra/lib/apache-cass=
andra-clientutil-1.0.0.jar:/opt/obs/cassandra/lib/apache-cassandra-thrift-1=
.0.0.jar:/opt/obs/cassandra/lib/avro-1.4.0-fixes.jar:/opt/obs/cassandra/lib=
/avro-1.4.0-sources-fixes.jar:/opt/obs/cassandra/lib/cassandra_simple_authe=
ntication.jar:/opt/obs/cassandra/lib/commons-cli-1.1.jar:/opt/obs/cassandra=
/lib/commons-codec-1.2.jar:/opt/obs/cassandra/lib/commons-lang-2.4.jar:/opt=
/obs/cassandra/lib/compress-lzf-0.8.4.jar:/opt/obs/cassandra/lib/concurrent=
linkedhashmap-lru-1.2.jar:/opt/obs/cassandra/lib/guava-r08.jar:/opt/obs/cas=
sandra/lib/high-scale-lib-1.1.2.jar:/opt/obs/cassandra/lib/jackson-core-asl=
-1.4.0.jar:/opt/obs/cassandra/lib/jackson-mapper-asl-1.4.0.jar:/opt/obs/cas=
sandra/lib/jamm-0.2.5.jar:/opt/obs/cassandra/lib/jline-0.9.94.jar:/opt/obs/=
cassandra/lib/json-simple-1.1.jar:/opt/obs/cassandra/lib/libthrift-0.6.jar:=
/opt/obs/cassandra/lib/log4j-1.2.16.jar:/opt/obs/cassandra/lib/servlet-api-=
2.5-20081211.jar:/opt/obs/cassandra/lib/slf4j-api-1.6.1.jar:/opt/obs/cassan=
dra/lib/slf4j-log4j12-1.6.1.jar:/opt/obs/cassandra/lib/snakeyaml-1.6.jar:/o=
pt/obs/cassandra/lib/snappy-java-1.0.3.jar
>            Reporter: MaHaiyang
>            Assignee: Sylvain Lebresne
>            Priority: Minor
>             Fix For: 1.0.9
>
>         Attachments: 3537.txt
>
>
> I have 4 cassandra nodes ,and I put about 30G data to db for every nodes =
. It's just 4 days before I start the cluster ,but now every 4 nodes have t=
he same problem ,JVM heap is full  ,and  GC take no effect ,There must be s=
ome memory leak . Jmap the memory as follow:
> Object Histogram:
> num =09  #instances=09#bytes=09Class description
> -------------------------------------------------------------------------=
-
> 1:=09=0915793606=09758093088=09java.nio.HeapByteBuffer
> 2:=09=092153811=09320138208=09java.lang.Object[]
> 3:=09=096163192=09197222144=09org.apache.cassandra.db.Column
> 4:=09=092543836=09175890256=09int[]
> 5:=09=092168816=09155397192=09long[]
> 6:=09=092078123=09116374888=09org.cliffc.high_scale_lib.ConcurrentAutoTab=
le$CAT
> 7:=09=091847111=0973884440=09java.math.BigInteger
> 8:=09=091234243=0959243664=09java.util.Hashtable
> 9:=09=091770829=0958233000=09char[]
> 10:=09=091770627=0956660064=09java.lang.String
> 11:=09=091665886=0939981264=09org.apache.cassandra.db.DecoratedKey
> 12:=09=09692706=0938791536=09org.cliffc.high_scale_lib.NonBlockingHashMap=
$CHM
> 13:=09=091234274=0937172088=09java.util.Hashtable$Entry[]
> 14:=09=091133541=0936273312=09java.net.Inet4Address
> 15:=09=09738528=0935449344=09org.apache.cassandra.service.ReadCallback
> 16:=09=092078118=0933249888=09org.cliffc.high_scale_lib.Counter
> 17:=09=091373886=0932973264=09org.apache.cassandra.db.ReadResponse
> 18:=09=091234023=0929616552=09org.apache.cassandra.net.Message
> 19:=09=091234019=0929616456=09org.apache.cassandra.net.Header
> 20:=09=091846185=0929538960=09org.apache.cassandra.dht.BigIntegerToken
> 21:=09=09891378=0928524096=09org.apache.cassandra.utils.ExpiringMap$Cache=
ableObject
> 22:=09=09692706=0927708240=09org.cliffc.high_scale_lib.NonBlockingHashMap
> 23:=09=091148252=0927558048=09java.util.Collections$SynchronizedSet
> 24:=09=09541977=0926014896=09org.apache.cassandra.db.SliceFromReadCommand
> 25:=09=09998001=0923952024=09java.util.concurrent.ConcurrentSkipListMap$N=
ode
> 26:=09=09928792=0922291008=09java.util.ArrayList
> 27:=09=09692715=0922166880=09java.util.concurrent.atomic.AtomicReferenceF=
ieldUpdater$AtomicReferenceFieldUpdaterImpl
> 28:=09=09891378=0921393072=09org.apache.cassandra.net.CallbackInfo
> 29:=09=091148247=0918371952=09java.util.Hashtable$KeySet
> 30:=09=09731859=0917564616=09org.apache.cassandra.db.Row
> 31:=09=09529991=0916959712=09org.apache.cassandra.db.ArrayBackedSortedCol=
umns
> 32:=09=09691425=0916594200=09org.apache.cassandra.db.AbstractColumnContai=
ner$DeletionInfo
> 33:=09=09648580=0915565920=09org.apache.cassandra.db.filter.QueryPath
> 34:=09=09648338=0915560112=09org.apache.cassandra.service.RowDigestResolv=
er
> 35:=09=09971376=0915542016=09java.util.concurrent.atomic.AtomicInteger
> 36:=09=09837418=0913398688=09org.apache.cassandra.utils.SimpleCondition
> 37:=09=09535614=0912854736=09org.apache.cassandra.db.ColumnFamily
> 38:=09=09725634=0911610144=09java.util.concurrent.atomic.AtomicReference
> 39:=09=09195117=099365616=09org.apache.cassandra.db.ThreadSafeSortedColum=
ns
> 40:=09=09281921=099021472=09java.util.concurrent.ConcurrentSkipListMap$He=
adIndex
> 41:=09=09277679=098885728=09java.util.concurrent.locks.ReentrantLock$Nonf=
airSync
> 42:=09=09314424=097546176=09java.util.concurrent.ConcurrentSkipListMap$In=
dex
> 43:=09=09275186=096604464=09java.util.concurrent.locks.AbstractQueuedSync=
hronizer$ConditionObject
> 44:=09=09270280=096486720=09java.util.concurrent.LinkedBlockingQueue$Node
> 45:=09=09219553=095269272=09org.apache.cassandra.io.sstable.IndexSummary$=
KeyPosition
> 46:=09=09106436=095108928=09java.util.TreeMap
> 47:=09=09122185=094887400=09org.apache.cassandra.db.ExpiringColumn
> 48:=09=09189968=094559232=09org.apache.cassandra.db.SuperColumn
> 49:=09=09275659=094410544=09java.util.concurrent.locks.ReentrantLock
> 50:=09=0990213=094330224=09java.util.concurrent.LinkedBlockingQueue
> 51:=09=09107026=094281040=09java.util.TreeMap$Entry
> 52:=09=0930501=094222056=09* ConstMethodKlass

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrato=
rs: https://issues.apache.org/jira/secure/ContactAdministrators!default.jsp=
a
For more information on JIRA, see: http://www.atlassian.com/software/jira