incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stu King <...@stuartrexking.com>
Subject Re: Exception when running a clean up
Date Thu, 10 Mar 2011 23:54:48 GMT
I have upgraded from 0.7.0 to 0.7.3. I then run nodetool scrub on my
keyspace and now see this exception:

Exception in thread "main" java.io.IOError: java.io.IOException: Cannot run
program "ln": java.io.IOException: error=24, Too many open files
at
org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1658)
at
org.apache.cassandra.db.ColumnFamilyStore.scrub(ColumnFamilyStore.java:962)
at
org.apache.cassandra.service.StorageService.scrub(StorageService.java:1256)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
at
com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
at
javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
at
javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
at
javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
at
javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
at
javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
at sun.rmi.transport.Transport$1.run(Transport.java:177)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Caused by: java.io.IOException: Cannot run program "ln":
java.io.IOException: error=24, Too many open files
at java.lang.ProcessBuilder.start(ProcessBuilder.java:475)
at
org.apache.cassandra.utils.CLibrary.createHardLinkWithExec(CLibrary.java:181)
at org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:147)
at
org.apache.cassandra.io.sstable.SSTableReader.createLinks(SSTableReader.java:617)
at
org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1651)
... 32 more
Caused by: java.io.IOException: java.io.IOException: error=24, Too many open
files
at java.lang.UNIXProcess.<init>(UNIXProcess.java:164)
at java.lang.ProcessImpl.start(ProcessImpl.java:81)
at java.lang.ProcessBuilder.start(ProcessBuilder.java:468)
... 36 more


On Thu, Mar 10, 2011 at 5:49 PM, aaron morton <aaron@thelastpickle.com>wrote:

> What version of cassandra are you using and what is the upgrade history for
> the cluster?
> Aaron
>
> On 10/03/2011, at 8:24 PM, Stu King wrote:
>
> > I am seeing this exception when I am trying to run a cleanup. I want to
> decommission the node after the cleanup.
> >
> > java.util.concurrent.ExecutionException: java.io.IOError:
> java.io.EOFException
> >       at
> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> >       at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> >       at
> org.apache.cassandra.db.CompactionManager.performCleanup(CompactionManager.java:180)
> >       at
> org.apache.cassandra.db.ColumnFamilyStore.forceCleanup(ColumnFamilyStore.java:909)
> >       at
> org.apache.cassandra.service.StorageService.forceTableCleanup(StorageService.java:1127)
> >       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >       at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> >       at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >       at java.lang.reflect.Method.invoke(Method.java:616)
> >       at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> >       at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> >       at
> com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> >       at
> com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> >       at
> com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> >       at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> >       at
> com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> >       at
> javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> >       at
> javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> >       at
> javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> >       at
> javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> >       at
> javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> >       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >       at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> >       at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >       at java.lang.reflect.Method.invoke(Method.java:616)
> >       at
> sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> >       at sun.rmi.transport.Transport$1.run(Transport.java:177)
> >       at java.security.AccessController.doPrivileged(Native Method)
> >       at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> >       at
> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> >       at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> >       at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> >       at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> >       at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> >       at java.lang.Thread.run(Thread.java:636)
> > Caused by: java.io.IOError: java.io.EOFException
> >       at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:105)
> >       at
> org.apache.cassandra.db.CompactionManager.doCleanupCompaction(CompactionManager.java:418)
> >       at
> org.apache.cassandra.db.CompactionManager.access$400(CompactionManager.java:54)
> >       at
> org.apache.cassandra.db.CompactionManager$2.call(CompactionManager.java:171)
> >       at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> >       at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> >       ... 3 more
> > Caused by: java.io.EOFException
> >       at java.io.RandomAccessFile.readFully(RandomAccessFile.java:416)
> >       at
> org.apache.cassandra.utils.FBUtilities.readByteArray(FBUtilities.java:280)
> >       at
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:76)
> >       at
> org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:364)
> >       at
> org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:313)
> >       at
> org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:101)
> >       ... 8 more
>
>

Mime
View raw message