cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dan Larsen <...@techba.se>
Subject Re: Help! Serious crash!
Date Wed, 07 Oct 2009 20:52:47 GMT
I will check that thread out! Thanks :-)

Maybe I should try to refresh the code...

I am starting to think, it might be the "native" java of snow leopard,  
that is causing some of my problems...
This is what has kept me from coding Java for more than a decade...  
Java is never just Java...
I know C is not just C either and C++ is not just C++... But it is so  
much more true for Java...

Anyways... Not trying to start a flame, in the middle of an Apache  
Incubator project... :-P
So... Thanks for you hints - I'll do some testing and post it here, if  
I find out anything interesting.

Good to know, you had that much data! I'll be attempting something  
similar... ;-)


On 07/10/2009, at 20.20, Igor Katkov wrote:

> I had similar OutOfMemory issues, see "commit logs are not deleted"   
> thread, but it was fixed/commited to trunk.
> What I don't get is how your commit log segment grows beyond  
> threshold.
>
> >Is it really nescessary to have as much RAM as you have data?!?!?
> No, I was able to insert 50Gb worth of data with 3Gb cap
>
>
> On Wed, Oct 7, 2009 at 2:16 PM, Dan Larsen <dan@techba.se> wrote:
> Sorry about that... running 0.4 taken from trunk.
> I didn't change CommitLogRotationThresholdInMB - it's still 128
>
> Thanks!
>
> On 07/10/2009, at 20.12, Igor Katkov wrote:
>
>> What version are you running? Take 0.4 from trunk.
>> Default config has
>>   <CommitLogRotationThresholdInMB>128</ 
>> CommitLogRotationThresholdInMB>
>> Did you change it allow your commit log chunks be 1Gb?
>>
>> On Wed, Oct 7, 2009 at 1:38 PM, Dan Larsen <dan@techba.se> wrote:
>> Hi guys,
>>
>> I have a small setup with 3 macs running os x snow leopard and the  
>> native java 1.6.0.something.something ;-)
>> I am doing a lot of inserts, without any problems and I can read  
>> the data again.
>>
>> But....
>> At some point, around 1G Commitlog size, cassandra starts a rise in  
>> ram usage, untill it just stops responding - if I restart the  
>> server, with or without -b, it will just usae 100% cpu for 10-15  
>> mins and then stop with this message:
>> Another problem, is that it seems to kill other servers too, when I  
>> try to restart :-(
>>
>> java.lang.OutOfMemoryError: Java heap space
>> Dumping heap to java_pid53353.hprof ...
>> Heap dump file created [1164687070 bytes in 89.912 secs]
>>
>> ERROR - Fatal exception in thread Thread[RMI TCP Connection(idle), 
>> 5,RMI Runtime]
>> java.lang.OutOfMemoryError: Java heap space
>>        at java.io.BufferedInputStream.<init> 
>> (BufferedInputStream.java:178)
>>        at java.io.BufferedInputStream.<init> 
>> (BufferedInputStream.java:158)
>>        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0 
>> (TCPTransport.java:679)
>>        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run 
>> (TCPTransport.java:649)
>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask 
>> (ThreadPoolExecutor.java:886)
>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run 
>> (ThreadPoolExecutor.java:908)
>>        at java.lang.Thread.run(Thread.java:637)
>> ERROR - Fatal exception in thread Thread[main,5,main]
>> java.lang.OutOfMemoryError: Java heap space
>> ERROR - Fatal exception in thread Thread[RMI TCP Connection(idle), 
>> 5,RMI Runtime]
>> java.lang.OutOfMemoryError: Java heap space
>>        at java.lang.Class.getDeclaredMethods0(Native Method)
>>        at java.lang.Class.privateGetDeclaredMethods(Class.java:2427)
>>        at java.lang.Class.getDeclaredMethod(Class.java:1935)
>>        at java.io.ObjectStreamClass.getPrivateMethod 
>> (ObjectStreamClass.java:1382)
>>        at java.io.ObjectStreamClass.access$1700 
>> (ObjectStreamClass.java:52)
>>        at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:438)
>>        at java.security.AccessController.doPrivileged(Native Method)
>>        at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java: 
>> 413)
>>        at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java: 
>> 310)
>>        at java.io.ObjectOutputStream.writeObject0 
>> (ObjectOutputStream.java:1106)
>>        at java.io.ObjectOutputStream.defaultWriteFields 
>> (ObjectOutputStream.java:1509)
>>        at java.io.ObjectOutputStream.writeSerialData 
>> (ObjectOutputStream.java:1474)
>>        at java.io.ObjectOutputStream.writeOrdinaryObject 
>> (ObjectOutputStream.java:1392)
>>        at java.io.ObjectOutputStream.writeObject0 
>> (ObjectOutputStream.java:1150)
>>        at java.io.ObjectOutputStream.writeObject 
>> (ObjectOutputStream.java:326)
>>        at java.util.ArrayList.writeObject(ArrayList.java:570)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>        at sun.reflect.NativeMethodAccessorImpl.invoke 
>> (NativeMethodAccessorImpl.java:39)
>>        at sun.reflect.DelegatingMethodAccessorImpl.invoke 
>> (DelegatingMethodAccessorImpl.java:25)
>>        at java.lang.reflect.Method.invoke(Method.java:597)
>>        at java.io.ObjectStreamClass.invokeWriteObject 
>> (ObjectStreamClass.java:945)
>>        at java.io.ObjectOutputStream.writeSerialData 
>> (ObjectOutputStream.java:1461)
>>        at java.io.ObjectOutputStream.writeOrdinaryObject 
>> (ObjectOutputStream.java:1392)
>>        at java.io.ObjectOutputStream.writeObject0 
>> (ObjectOutputStream.java:1150)
>>        at java.io.ObjectOutputStream.writeObject 
>> (ObjectOutputStream.java:326)
>>        at sun.rmi.server.UnicastRef.marshalValue(UnicastRef.java:274)
>>        at sun.rmi.server.UnicastServerRef.dispatch 
>> (UnicastServerRef.java:315)
>>        at sun.rmi.transport.Transport$1.run(Transport.java:159)
>>        at java.security.AccessController.doPrivileged(Native Method)
>>        at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
>>        at sun.rmi.transport.tcp.TCPTransport.handleMessages 
>> (TCPTransport.java:535)
>>        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0 
>> (TCPTransport.java:790)
>> ERROR - Fatal exception in thread Thread[RMI TCP Connection(idle), 
>> 5,RMI Runtime]
>>
>> [ABOVE REPEATS A COUPLE OF TIMES]
>>
>> If I try to connect with nodeprobe, after it halts, I will just get  
>> this:
>>
>> Error connecting to remote JMX agent!
>> java.io.IOException: Failed to retrieve RMIServer stub:  
>> javax.naming.ServiceUnavailableException [Root exception is  
>> java.rmi.ConnectException: Connection refused to host:  
>> 192.168.100.203; nested exception is:
>>        java.net.ConnectException: Connection refused]
>>        at javax.management.remote.rmi.RMIConnector.connect 
>> (RMIConnector.java:338)
>>        at javax.management.remote.JMXConnectorFactory.connect 
>> (JMXConnectorFactory.java:248)
>>        at org.apache.cassandra.tools.NodeProbe.connect 
>> (NodeProbe.java:152)
>>        at org.apache.cassandra.tools.NodeProbe.<init> 
>> (NodeProbe.java:114)
>>        at org.apache.cassandra.tools.NodeProbe.main(NodeProbe.java: 
>> 545)
>> Caused by: javax.naming.ServiceUnavailableException [Root exception  
>> is java.rmi.ConnectException: Connection refused to host:  
>> 192.168.100.203; nested exception is:
>>        java.net.ConnectException: Connection refused]
>>        at com.sun.jndi.rmi.registry.RegistryContext.lookup 
>> (RegistryContext.java:101)
>>        at com.sun.jndi.toolkit.url.GenericURLContext.lookup 
>> (GenericURLContext.java:185)
>>        at javax.naming.InitialContext.lookup(InitialContext.java:392)
>>        at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI 
>> (RMIConnector.java:1886)
>>        at javax.management.remote.rmi.RMIConnector.findRMIServer 
>> (RMIConnector.java:1856)
>>        at javax.management.remote.rmi.RMIConnector.connect 
>> (RMIConnector.java:257)
>>        ... 4 more
>> Caused by: java.rmi.ConnectException: Connection refused to host:  
>> 192.168.100.203; nested exception is:
>>        java.net.ConnectException: Connection refused
>>        at sun.rmi.transport.tcp.TCPEndpoint.newSocket 
>> (TCPEndpoint.java:601)
>>        at sun.rmi.transport.tcp.TCPChannel.createConnection 
>> (TCPChannel.java:198)
>>        at sun.rmi.transport.tcp.TCPChannel.newConnection 
>> (TCPChannel.java:184)
>>        at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:322)
>>        at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
>>        at com.sun.jndi.rmi.registry.RegistryContext.lookup 
>> (RegistryContext.java:97)
>>        ... 9 more
>> Caused by: java.net.ConnectException: Connection refused
>>        at java.net.PlainSocketImpl.socketConnect(Native Method)
>>        at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java: 
>> 333)
>>        at java.net.PlainSocketImpl.connectToAddress 
>> (PlainSocketImpl.java:195)
>>        at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
>>        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:432)
>>        at java.net.Socket.connect(Socket.java:525)
>>        at java.net.Socket.connect(Socket.java:475)
>>        at java.net.Socket.<init>(Socket.java:372)
>>        at java.net.Socket.<init>(Socket.java:186)
>>        at  
>> sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket 
>> (RMIDirectSocketFactory.java:22)
>>        at  
>> sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket 
>> (RMIMasterSocketFactory.java:128)
>>        at sun.rmi.transport.tcp.TCPEndpoint.newSocket 
>> (TCPEndpoint.java:595)
>>        ... 14 more
>>
>>
>> I know i responds as supposed, before the crash.
>>
>> Hope somebody can help! :-)
>>
>> Best regards
>> Dan Larsen
>>
>
>


Mime
View raw message