incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From earltbj <earl...@gmail.com>
Subject Snapshot Memory Allocation Error
Date Wed, 10 Aug 2011 13:49:33 GMT
While performing a snapshot on a node earlier this morning, the following
error was encountered executing the snapshot command:

Exception in thread "main" java.io.IOError: java.io.IOException: Cannot run
program "ln": java.io.IOException: error=12, Cannot allocate memory
        at
org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1660)
        at
org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:1686)
        at org.apache.cassandra.db.Table.snapshot(Table.java:198)
        at
org.apache.cassandra.service.StorageService.takeSnapshot(StorageService.java:1393)
        at sun.reflect.GeneratedMethodAccessor123.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
        at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
        at
com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
        at
com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
        at
com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
        at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
        at
com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
        at
javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
        at
javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
        at
javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
        at
javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
        at
javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
        at sun.reflect.GeneratedMethodAccessor139.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at
sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
        at sun.rmi.transport.Transport$1.run(Transport.java:177)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
        at
sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
        at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
        at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:636)
Caused by: java.io.IOException: Cannot run program "ln":
java.io.IOException: error=12, Cannot allocate memory
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:475)
        at
org.apache.cassandra.utils.CLibrary.createHardLinkWithExec(CLibrary.java:181)
        at
org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:147)
        at
org.apache.cassandra.io.sstable.SSTableReader.createLinks(SSTableReader.java:730)
        at
org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1653)
        ... 31 more
Caused by: java.io.IOException: java.io.IOException: error=12, Cannot
allocate memory
        at java.lang.UNIXProcess.<init>(UNIXProcess.java:164)
        at java.lang.ProcessImpl.start(ProcessImpl.java:81)
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:468)
        ... 35 more

Logging is in debug mode; however, there were no error messages written to
the system.log around the time of the snapshot. The database is up, I can
log into it and view column families, so the only thing affected at this
point that I can see is the ability to perform snapshots.

Note: this same thing happened to a node within the same cluster a few weeks
ago and the work around at that time was to bounce the database, afterwards
we were able to proceed with snapshots.

Any idea what could be causing this?

BACKGROUND
Cassandra ReleaseVersion: 0.8.1
OS Level: Linux cassandra1 2.6.32-32-server #62-Ubuntu SMP Wed Apr 20
22:07:43 UTC 2011 x86_64 GNU/Linux

 

--
View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Snapshot-Memory-Allocation-Error-tp6672437p6672437.html
Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.

Mime
View raw message