cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dan Larsen <...@techba.se>
Subject Re: Help! Serious crash!
Date Wed, 07 Oct 2009 18:23:40 GMT
I don't know why... But for some reason, the commitlogs grew bigger  
than 128MB - so I suppose a manual compact was needed...?!?!
Anyways... I restarted with the 2GB value in cassandra.in.sh,  
everything got compacted, the commitlog rotated - and I could stop,  
change the settings back and start everything up, with no probles...

Thanks for your suggestions Igor! :-)

Best regards
Dan

On 07/10/2009, at 20.16, Dan Larsen wrote:

> Sorry about that... running 0.4 taken from trunk.
> I didn't change CommitLogRotationThresholdInMB - it's still 128
>
> Thanks!
>
> On 07/10/2009, at 20.12, Igor Katkov wrote:
>
>> What version are you running? Take 0.4 from trunk.
>> Default config has
>>   <CommitLogRotationThresholdInMB>128</ 
>> CommitLogRotationThresholdInMB>
>> Did you change it allow your commit log chunks be 1Gb?
>>
>> On Wed, Oct 7, 2009 at 1:38 PM, Dan Larsen <dan@techba.se> wrote:
>> Hi guys,
>>
>> I have a small setup with 3 macs running os x snow leopard and the  
>> native java 1.6.0.something.something ;-)
>> I am doing a lot of inserts, without any problems and I can read  
>> the data again.
>>
>> But....
>> At some point, around 1G Commitlog size, cassandra starts a rise in  
>> ram usage, untill it just stops responding - if I restart the  
>> server, with or without -b, it will just usae 100% cpu for 10-15  
>> mins and then stop with this message:
>> Another problem, is that it seems to kill other servers too, when I  
>> try to restart :-(
>>
>> java.lang.OutOfMemoryError: Java heap space
>> Dumping heap to java_pid53353.hprof ...
>> Heap dump file created [1164687070 bytes in 89.912 secs]
>>
>> ERROR - Fatal exception in thread Thread[RMI TCP Connection(idle), 
>> 5,RMI Runtime]
>> java.lang.OutOfMemoryError: Java heap space
>>        at java.io.BufferedInputStream.<init> 
>> (BufferedInputStream.java:178)
>>        at java.io.BufferedInputStream.<init> 
>> (BufferedInputStream.java:158)
>>        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0 
>> (TCPTransport.java:679)
>>        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run 
>> (TCPTransport.java:649)
>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask 
>> (ThreadPoolExecutor.java:886)
>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run 
>> (ThreadPoolExecutor.java:908)
>>        at java.lang.Thread.run(Thread.java:637)
>> ERROR - Fatal exception in thread Thread[main,5,main]
>> java.lang.OutOfMemoryError: Java heap space
>> ERROR - Fatal exception in thread Thread[RMI TCP Connection(idle), 
>> 5,RMI Runtime]
>> java.lang.OutOfMemoryError: Java heap space
>>        at java.lang.Class.getDeclaredMethods0(Native Method)
>>        at java.lang.Class.privateGetDeclaredMethods(Class.java:2427)
>>        at java.lang.Class.getDeclaredMethod(Class.java:1935)
>>        at java.io.ObjectStreamClass.getPrivateMethod 
>> (ObjectStreamClass.java:1382)
>>        at java.io.ObjectStreamClass.access$1700 
>> (ObjectStreamClass.java:52)
>>        at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:438)
>>        at java.security.AccessController.doPrivileged(Native Method)
>>        at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java: 
>> 413)
>>        at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java: 
>> 310)
>>        at java.io.ObjectOutputStream.writeObject0 
>> (ObjectOutputStream.java:1106)
>>        at java.io.ObjectOutputStream.defaultWriteFields 
>> (ObjectOutputStream.java:1509)
>>        at java.io.ObjectOutputStream.writeSerialData 
>> (ObjectOutputStream.java:1474)
>>        at java.io.ObjectOutputStream.writeOrdinaryObject 
>> (ObjectOutputStream.java:1392)
>>        at java.io.ObjectOutputStream.writeObject0 
>> (ObjectOutputStream.java:1150)
>>        at java.io.ObjectOutputStream.writeObject 
>> (ObjectOutputStream.java:326)
>>        at java.util.ArrayList.writeObject(ArrayList.java:570)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke 
>> (NativeMethodAccessorImpl.java:39)
>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke 
>> (DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at java.io.ObjectStreamClass.invokeWriteObject 
>> (ObjectStreamClass.java:945)
>>        at java.io.ObjectOutputStream.writeSerialData 
>> (ObjectOutputStream.java:1461)
>>        at java.io.ObjectOutputStream.writeOrdinaryObject 
>> (ObjectOutputStream.java:1392)
>>        at java.io.ObjectOutputStream.writeObject0 
>> (ObjectOutputStream.java:1150)
>>        at java.io.ObjectOutputStream.writeObject 
>> (ObjectOutputStream.java:326)
>>        at sun.rmi.server.UnicastRef.marshalValue(UnicastRef.java:274)
>>        at sun.rmi.server.UnicastServerRef.dispatch 
>> (UnicastServerRef.java:315)
>>        at sun.rmi.transport.Transport$1.run(Transport.java:159)
>>        at java.security.AccessController.doPrivileged(Native Method)
>>        at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
>>        at sun.rmi.transport.tcp.TCPTransport.handleMessages 
>> (TCPTransport.java:535)
>>        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0 
>> (TCPTransport.java:790)
>> ERROR - Fatal exception in thread Thread[RMI TCP Connection(idle), 
>> 5,RMI Runtime]
>>
>> [ABOVE REPEATS A COUPLE OF TIMES]
>>
>> If I try to connect with nodeprobe, after it halts, I will just get  
>> this:
>>
>> Error connecting to remote JMX agent!
>> java.io.IOException: Failed to retrieve RMIServer stub:  
>> javax.naming.ServiceUnavailableException [Root exception is  
>> java.rmi.ConnectException: Connection refused to host:  
>> 192.168.100.203; nested exception is:
>>        java.net.ConnectException: Connection refused]
>>        at javax.management.remote.rmi.RMIConnector.connect 
>> (RMIConnector.java:338)
>>        at javax.management.remote.JMXConnectorFactory.connect 
>> (JMXConnectorFactory.java:248)
>>        at org.apache.cassandra.tools.NodeProbe.connect 
>> (NodeProbe.java:152)
>>        at org.apache.cassandra.tools.NodeProbe.<init> 
>> (NodeProbe.java:114)
>>        at org.apache.cassandra.tools.NodeProbe.main(NodeProbe.java: 
>> 545)
>> Caused by: javax.naming.ServiceUnavailableException [Root exception  
>> is java.rmi.ConnectException: Connection refused to host:  
>> 192.168.100.203; nested exception is:
>>        java.net.ConnectException: Connection refused]
>>        at com.sun.jndi.rmi.registry.RegistryContext.lookup 
>> (RegistryContext.java:101)
>>        at com.sun.jndi.toolkit.url.GenericURLContext.lookup 
>> (GenericURLContext.java:185)
>>        at javax.naming.InitialContext.lookup(InitialContext.java:392)
>>        at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI 
>> (RMIConnector.java:1886)
>>        at javax.management.remote.rmi.RMIConnector.findRMIServer 
>> (RMIConnector.java:1856)
>>        at javax.management.remote.rmi.RMIConnector.connect 
>> (RMIConnector.java:257)
>>        ... 4 more
>> Caused by: java.rmi.ConnectException: Connection refused to host:  
>> 192.168.100.203; nested exception is:
>>        java.net.ConnectException: Connection refused
>>        at sun.rmi.transport.tcp.TCPEndpoint.newSocket 
>> (TCPEndpoint.java:601)
>>        at sun.rmi.transport.tcp.TCPChannel.createConnection 
>> (TCPChannel.java:198)
>>        at sun.rmi.transport.tcp.TCPChannel.newConnection 
>> (TCPChannel.java:184)
>>        at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:322)
>>        at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
>>        at com.sun.jndi.rmi.registry.RegistryContext.lookup 
>> (RegistryContext.java:97)
>>        ... 9 more
>> Caused by: java.net.ConnectException: Connection refused
>>        at java.net.PlainSocketImpl.socketConnect(Native Method)
>>        at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java: 
>> 333)
>>        at java.net.PlainSocketImpl.connectToAddress 
>> (PlainSocketImpl.java:195)
>>        at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
>>        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:432)
>>        at java.net.Socket.connect(Socket.java:525)
>>        at java.net.Socket.connect(Socket.java:475)
>>        at java.net.Socket.<init>(Socket.java:372)
>>        at java.net.Socket.<init>(Socket.java:186)
>>        at  
>> sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket 
>> (RMIDirectSocketFactory.java:22)
>>        at  
>> sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket 
>> (RMIMasterSocketFactory.java:128)
>>        at sun.rmi.transport.tcp.TCPEndpoint.newSocket 
>> (TCPEndpoint.java:595)
>>        ... 14 more
>>
>>
>> I know i responds as supposed, before the crash.
>>
>> Hope somebody can help! :-)
>>
>> Best regards
>> Dan Larsen
>>
>


Mime
View raw message