cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dan Larsen <...@techba.se>
Subject Re: Help! Serious crash!
Date Wed, 07 Oct 2009 17:50:53 GMT
Forgot to say - It will eat up 1.13GB of RAM and 1.32GB VRAM, and  
still throw and OutOfMemory, though 700MB left...

On 07/10/2009, at 19.38, Dan Larsen wrote:

> Hi guys,
>
> I have a small setup with 3 macs running os x snow leopard and the  
> native java 1.6.0.something.something ;-)
> I am doing a lot of inserts, without any problems and I can read the  
> data again.
>
> But....
> At some point, around 1G Commitlog size, cassandra starts a rise in  
> ram usage, untill it just stops responding - if I restart the  
> server, with or without -b, it will just usae 100% cpu for 10-15  
> mins and then stop with this message:
> Another problem, is that it seems to kill other servers too, when I  
> try to restart :-(
>
> java.lang.OutOfMemoryError: Java heap space
> Dumping heap to java_pid53353.hprof ...
> Heap dump file created [1164687070 bytes in 89.912 secs]
>
> ERROR - Fatal exception in thread Thread[RMI TCP Connection(idle), 
> 5,RMI Runtime]
> java.lang.OutOfMemoryError: Java heap space
> 	at java.io.BufferedInputStream.<init>(BufferedInputStream.java:178)
> 	at java.io.BufferedInputStream.<init>(BufferedInputStream.java:158)
> 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0 
> (TCPTransport.java:679)
> 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run 
> (TCPTransport.java:649)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask 
> (ThreadPoolExecutor.java:886)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run 
> (ThreadPoolExecutor.java:908)
> 	at java.lang.Thread.run(Thread.java:637)
> ERROR - Fatal exception in thread Thread[main,5,main]
> java.lang.OutOfMemoryError: Java heap space
> ERROR - Fatal exception in thread Thread[RMI TCP Connection(idle), 
> 5,RMI Runtime]
> java.lang.OutOfMemoryError: Java heap space
> 	at java.lang.Class.getDeclaredMethods0(Native Method)
> 	at java.lang.Class.privateGetDeclaredMethods(Class.java:2427)
> 	at java.lang.Class.getDeclaredMethod(Class.java:1935)
> 	at java.io.ObjectStreamClass.getPrivateMethod 
> (ObjectStreamClass.java:1382)
> 	at java.io.ObjectStreamClass.access$1700(ObjectStreamClass.java:52)
> 	at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:438)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:413)
> 	at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:310)
> 	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java: 
> 1106)
> 	at java.io.ObjectOutputStream.defaultWriteFields 
> (ObjectOutputStream.java:1509)
> 	at java.io.ObjectOutputStream.writeSerialData 
> (ObjectOutputStream.java:1474)
> 	at java.io.ObjectOutputStream.writeOrdinaryObject 
> (ObjectOutputStream.java:1392)
> 	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java: 
> 1150)
> 	at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java: 
> 326)
> 	at java.util.ArrayList.writeObject(ArrayList.java:570)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke 
> (NativeMethodAccessorImpl.java:39)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke 
> (DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at java.io.ObjectStreamClass.invokeWriteObject 
> (ObjectStreamClass.java:945)
> 	at java.io.ObjectOutputStream.writeSerialData 
> (ObjectOutputStream.java:1461)
> 	at java.io.ObjectOutputStream.writeOrdinaryObject 
> (ObjectOutputStream.java:1392)
> 	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java: 
> 1150)
> 	at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java: 
> 326)
> 	at sun.rmi.server.UnicastRef.marshalValue(UnicastRef.java:274)
> 	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java: 
> 315)
> 	at sun.rmi.transport.Transport$1.run(Transport.java:159)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
> 	at sun.rmi.transport.tcp.TCPTransport.handleMessages 
> (TCPTransport.java:535)
> 	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0 
> (TCPTransport.java:790)
> ERROR - Fatal exception in thread Thread[RMI TCP Connection(idle), 
> 5,RMI Runtime]
>
> [ABOVE REPEATS A COUPLE OF TIMES]
>
> If I try to connect with nodeprobe, after it halts, I will just get  
> this:
>
> Error connecting to remote JMX agent!
> java.io.IOException: Failed to retrieve RMIServer stub:  
> javax.naming.ServiceUnavailableException [Root exception is  
> java.rmi.ConnectException: Connection refused to host:  
> 192.168.100.203; nested exception is:
> 	java.net.ConnectException: Connection refused]
> 	at javax.management.remote.rmi.RMIConnector.connect 
> (RMIConnector.java:338)
> 	at javax.management.remote.JMXConnectorFactory.connect 
> (JMXConnectorFactory.java:248)
> 	at org.apache.cassandra.tools.NodeProbe.connect(NodeProbe.java:152)
> 	at org.apache.cassandra.tools.NodeProbe.<init>(NodeProbe.java:114)
> 	at org.apache.cassandra.tools.NodeProbe.main(NodeProbe.java:545)
> Caused by: javax.naming.ServiceUnavailableException [Root exception  
> is java.rmi.ConnectException: Connection refused to host:  
> 192.168.100.203; nested exception is:
> 	java.net.ConnectException: Connection refused]
> 	at com.sun.jndi.rmi.registry.RegistryContext.lookup 
> (RegistryContext.java:101)
> 	at com.sun.jndi.toolkit.url.GenericURLContext.lookup 
> (GenericURLContext.java:185)
> 	at javax.naming.InitialContext.lookup(InitialContext.java:392)
> 	at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI 
> (RMIConnector.java:1886)
> 	at javax.management.remote.rmi.RMIConnector.findRMIServer 
> (RMIConnector.java:1856)
> 	at javax.management.remote.rmi.RMIConnector.connect 
> (RMIConnector.java:257)
> 	... 4 more
> Caused by: java.rmi.ConnectException: Connection refused to host:  
> 192.168.100.203; nested exception is:
> 	java.net.ConnectException: Connection refused
> 	at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:601)
> 	at sun.rmi.transport.tcp.TCPChannel.createConnection 
> (TCPChannel.java:198)
> 	at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java: 
> 184)
> 	at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:322)
> 	at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
> 	at com.sun.jndi.rmi.registry.RegistryContext.lookup 
> (RegistryContext.java:97)
> 	... 9 more
> Caused by: java.net.ConnectException: Connection refused
> 	at java.net.PlainSocketImpl.socketConnect(Native Method)
> 	at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
> 	at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java: 
> 195)
> 	at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
> 	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:432)
> 	at java.net.Socket.connect(Socket.java:525)
> 	at java.net.Socket.connect(Socket.java:475)
> 	at java.net.Socket.<init>(Socket.java:372)
> 	at java.net.Socket.<init>(Socket.java:186)
> 	at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket 
> (RMIDirectSocketFactory.java:22)
> 	at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket 
> (RMIMasterSocketFactory.java:128)
> 	at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:595)
> 	... 14 more
>
>
> I know i responds as supposed, before the crash.
>
> Hope somebody can help! :-)
>
> Best regards
> Dan Larsen
>


Mime
View raw message