incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Ellis <jbel...@gmail.com>
Subject Re: Exception when running a clean up
Date Fri, 11 Mar 2011 02:34:22 GMT
Unrelated to either upgrade or scrub.  That just means you need to
install JNA to get native linking instead of having to fork to run ln.

On Thu, Mar 10, 2011 at 5:54 PM, Stu King <stu@stuartrexking.com> wrote:
> I have upgraded from 0.7.0 to 0.7.3. I then run nodetool scrub on my
> keyspace and now see this exception:
> Exception in thread "main" java.io.IOError: java.io.IOException: Cannot run
> program "ln": java.io.IOException: error=24, Too many open files
> at
> org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1658)
> at
> org.apache.cassandra.db.ColumnFamilyStore.scrub(ColumnFamilyStore.java:962)
> at
> org.apache.cassandra.service.StorageService.scrub(StorageService.java:1256)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
> at
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
> at
> com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
> at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
> at
> javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
> at
> javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
> at
> javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
> at
> javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
> at
> javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
> at sun.rmi.transport.Transport$1.run(Transport.java:177)
> at java.security.AccessController.doPrivileged(Native Method)
> at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
> at
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
> Caused by: java.io.IOException: Cannot run program "ln":
> java.io.IOException: error=24, Too many open files
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:475)
> at
> org.apache.cassandra.utils.CLibrary.createHardLinkWithExec(CLibrary.java:181)
> at org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:147)
> at
> org.apache.cassandra.io.sstable.SSTableReader.createLinks(SSTableReader.java:617)
> at
> org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1651)
> ... 32 more
> Caused by: java.io.IOException: java.io.IOException: error=24, Too many open
> files
> at java.lang.UNIXProcess.<init>(UNIXProcess.java:164)
> at java.lang.ProcessImpl.start(ProcessImpl.java:81)
> at java.lang.ProcessBuilder.start(ProcessBuilder.java:468)
> ... 36 more
>
> On Thu, Mar 10, 2011 at 5:49 PM, aaron morton <aaron@thelastpickle.com>
> wrote:
>>
>> What version of cassandra are you using and what is the upgrade history
>> for the cluster?
>> Aaron
>>
>> On 10/03/2011, at 8:24 PM, Stu King wrote:
>>
>> > I am seeing this exception when I am trying to run a cleanup. I want to
>> > decommission the node after the cleanup.
>> >
>> > java.util.concurrent.ExecutionException: java.io.IOError:
>> > java.io.EOFException
>> >       at
>> > java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
>> >       at java.util.concurrent.FutureTask.get(FutureTask.java:111)
>> >       at
>> > org.apache.cassandra.db.CompactionManager.performCleanup(CompactionManager.java:180)
>> >       at
>> > org.apache.cassandra.db.ColumnFamilyStore.forceCleanup(ColumnFamilyStore.java:909)
>> >       at
>> > org.apache.cassandra.service.StorageService.forceTableCleanup(StorageService.java:1127)
>> >       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> >       at
>> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> >       at
>> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> >       at java.lang.reflect.Method.invoke(Method.java:616)
>> >       at
>> > com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:111)
>> >       at
>> > com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:45)
>> >       at
>> > com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:226)
>> >       at
>> > com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
>> >       at
>> > com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:251)
>> >       at
>> > com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:857)
>> >       at
>> > com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:795)
>> >       at
>> > javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1450)
>> >       at
>> > javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:90)
>> >       at
>> > javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1285)
>> >       at
>> > javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1383)
>> >       at
>> > javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:807)
>> >       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> >       at
>> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> >       at
>> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> >       at java.lang.reflect.Method.invoke(Method.java:616)
>> >       at
>> > sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:322)
>> >       at sun.rmi.transport.Transport$1.run(Transport.java:177)
>> >       at java.security.AccessController.doPrivileged(Native Method)
>> >       at sun.rmi.transport.Transport.serviceCall(Transport.java:173)
>> >       at
>> > sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:553)
>> >       at
>> > sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
>> >       at
>> > sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
>> >       at
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>> >       at
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>> >       at java.lang.Thread.run(Thread.java:636)
>> > Caused by: java.io.IOError: java.io.EOFException
>> >       at
>> > org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:105)
>> >       at
>> > org.apache.cassandra.db.CompactionManager.doCleanupCompaction(CompactionManager.java:418)
>> >       at
>> > org.apache.cassandra.db.CompactionManager.access$400(CompactionManager.java:54)
>> >       at
>> > org.apache.cassandra.db.CompactionManager$2.call(CompactionManager.java:171)
>> >       at
>> > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>> >       at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>> >       ... 3 more
>> > Caused by: java.io.EOFException
>> >       at java.io.RandomAccessFile.readFully(RandomAccessFile.java:416)
>> >       at
>> > org.apache.cassandra.utils.FBUtilities.readByteArray(FBUtilities.java:280)
>> >       at
>> > org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:76)
>> >       at
>> > org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:364)
>> >       at
>> > org.apache.cassandra.db.SuperColumnSerializer.deserialize(SuperColumn.java:313)
>> >       at
>> > org.apache.cassandra.io.sstable.SSTableIdentityIterator.next(SSTableIdentityIterator.java:101)
>> >       ... 8 more
>>
>
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com

Mime
View raw message