I will check that thread out! Thanks :-)

Maybe I should try to refresh the code...

I am starting to think, it might be the "native" java of snow leopard, that is causing some of my problems...
This is what has kept me from coding Java for more than a decade... Java is never just Java...
I know C is not just C either and C++ is not just C++... But it is so much more true for Java...

Anyways... Not trying to start a flame, in the middle of an Apache Incubator project... :-P
So... Thanks for you hints - I'll do some testing and post it here, if I find out anything interesting.

Good to know, you had that much data! I'll be attempting something similar... ;-)


On 07/10/2009, at 20.20, Igor Katkov wrote:

I had similar OutOfMemory issues, see "commit logs are not deleted"  thread, but it was fixed/commited to trunk.
What I don't get is how your commit log segment grows beyond threshold.

>Is it really nescessary to have as much RAM as you have data?!?!?
No, I was able to insert 50Gb worth of data with 3Gb cap


On Wed, Oct 7, 2009 at 2:16 PM, Dan Larsen <dan@techba.se> wrote:
Sorry about that... running 0.4 taken from trunk.
I didn't change CommitLogRotationThresholdInMB - it's still 128

Thanks!

On 07/10/2009, at 20.12, Igor Katkov wrote:

What version are you running? Take 0.4 from trunk.
Default config has
  <CommitLogRotationThresholdInMB>128</CommitLogRotationThresholdInMB>
Did you change it allow your commit log chunks be 1Gb?

On Wed, Oct 7, 2009 at 1:38 PM, Dan Larsen <dan@techba.se> wrote:
Hi guys,

I have a small setup with 3 macs running os x snow leopard and the native java 1.6.0.something.something ;-)
I am doing a lot of inserts, without any problems and I can read the data again.

But....
At some point, around 1G Commitlog size, cassandra starts a rise in ram usage, untill it just stops responding - if I restart the server, with or without -b, it will just usae 100% cpu for 10-15 mins and then stop with this message:
Another problem, is that it seems to kill other servers too, when I try to restart :-(

java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid53353.hprof ...
Heap dump file created [1164687070 bytes in 89.912 secs]

ERROR - Fatal exception in thread Thread[RMI TCP Connection(idle),5,RMI Runtime]
java.lang.OutOfMemoryError: Java heap space
       at java.io.BufferedInputStream.<init>(BufferedInputStream.java:178)
       at java.io.BufferedInputStream.<init>(BufferedInputStream.java:158)
       at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:679)
       at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
       at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
       at java.lang.Thread.run(Thread.java:637)
ERROR - Fatal exception in thread Thread[main,5,main]
java.lang.OutOfMemoryError: Java heap space
ERROR - Fatal exception in thread Thread[RMI TCP Connection(idle),5,RMI Runtime]
java.lang.OutOfMemoryError: Java heap space
       at java.lang.Class.getDeclaredMethods0(Native Method)
       at java.lang.Class.privateGetDeclaredMethods(Class.java:2427)
       at java.lang.Class.getDeclaredMethod(Class.java:1935)
       at java.io.ObjectStreamClass.getPrivateMethod(ObjectStreamClass.java:1382)
       at java.io.ObjectStreamClass.access$1700(ObjectStreamClass.java:52)
       at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:438)
       at java.security.AccessController.doPrivileged(Native Method)
       at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:413)
       at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:310)
       at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1106)
       at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1509)
       at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1474)
       at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1392)
       at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1150)
       at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:326)
       at java.util.ArrayList.writeObject(ArrayList.java:570)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
       at java.lang.reflect.Method.invoke(Method.java:597)
       at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:945)
       at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1461)
       at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1392)
       at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1150)
       at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:326)
       at sun.rmi.server.UnicastRef.marshalValue(UnicastRef.java:274)
       at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:315)
       at sun.rmi.transport.Transport$1.run(Transport.java:159)
       at java.security.AccessController.doPrivileged(Native Method)
       at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
       at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
       at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
ERROR - Fatal exception in thread Thread[RMI TCP Connection(idle),5,RMI Runtime]

[ABOVE REPEATS A COUPLE OF TIMES]

If I try to connect with nodeprobe, after it halts, I will just get this:

Error connecting to remote JMX agent!
java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 192.168.100.203; nested exception is:
       java.net.ConnectException: Connection refused]
       at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:338)
       at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:248)
       at org.apache.cassandra.tools.NodeProbe.connect(NodeProbe.java:152)
       at org.apache.cassandra.tools.NodeProbe.<init>(NodeProbe.java:114)
       at org.apache.cassandra.tools.NodeProbe.main(NodeProbe.java:545)
Caused by: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 192.168.100.203; nested exception is:
       java.net.ConnectException: Connection refused]
       at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:101)
       at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:185)
       at javax.naming.InitialContext.lookup(InitialContext.java:392)
       at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1886)
       at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1856)
       at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:257)
       ... 4 more
Caused by: java.rmi.ConnectException: Connection refused to host: 192.168.100.203; nested exception is:
       java.net.ConnectException: Connection refused
       at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:601)
       at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:198)
       at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:184)
       at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:322)
       at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
       at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:97)
       ... 9 more
Caused by: java.net.ConnectException: Connection refused
       at java.net.PlainSocketImpl.socketConnect(Native Method)
       at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
       at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
       at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
       at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:432)
       at java.net.Socket.connect(Socket.java:525)
       at java.net.Socket.connect(Socket.java:475)
       at java.net.Socket.<init>(Socket.java:372)
       at java.net.Socket.<init>(Socket.java:186)
       at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:22)
       at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:128)
       at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:595)
       ... 14 more


I know i responds as supposed, before the crash.

Hope somebody can help! :-)

Best regards
Dan Larsen