incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bryan Fernandez <bfernande...@gmail.com>
Subject Re: Snapshot failing on JSON files in 1.1.0
Date Wed, 16 May 2012 23:35:07 GMT
Does anyone know when 1.1.1 will be released?

Thanks.

On Tue, May 15, 2012 at 5:40 PM, Brandon Williams <driftx@gmail.com> wrote:

> Probably https://issues.apache.org/jira/browse/CASSANDRA-4230
>
> On Tue, May 15, 2012 at 4:08 PM, Bryan Fernandez <bfernandezsa@gmail.com>
> wrote:
> > Greetings,
> >
> > We recently upgraded from 1.0.8 to 1.1.0. Everything has been running
> fine
> > with the exception of snapshots. When attempting to snapshot any of the
> > nodes in our six node cluster we are seeing the following error.
> >
> > [root@cassandra-n6 blotter]# /opt/apache-cassandra-1.1.0/bin/nodetool -h
> > 10.20.50.58 snapshot
> > Requested snapshot for: all keyspaces
> > Exception in thread "main" java.io.IOError: java.io.IOException: Unable
> to
> > create hard link from
> > /var/lib/cassandra/data/blotter/twitter_users/twitter_users.json to
> >
> /var/lib/cassandra/data/blotter/twitter_users/snapshots/1337115022389/twitter_users.json
> > (errno 17)
> > at
> >
> org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1454)
> > at
> >
> org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:1483)
> > at org.apache.cassandra.db.Table.snapshot(Table.java:205)
> > at
> >
> org.apache.cassandra.service.StorageService.takeSnapshot(StorageService.java:1793)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
> > at
> >
> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
> > at
> >
> com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
> > at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120)
> > at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262)
> > at
> >
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
> > at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
> > at
> >
> javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427)
> > at
> >
> javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
> > at
> >
> javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265)
> > at
> >
> javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360)
> > at
> >
> javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:303)
> > at sun.rmi.transport.Transport$1.run(Transport.java:159)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
> > at
> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
> > at
> >
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
> > at
> >
> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> > at java.lang.Thread.run(Thread.java:662)
> > Caused by: java.io.IOException: Unable to create hard link from
> > /var/lib/cassandra/data/blotter/twitter_users/twitter_users.json to
> >
> /var/lib/cassandra/data/blotter/twitter_users/snapshots/1337115022389/twitter_users.json
> > (errno 17)
> > at org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:163)
> > at
> >
> org.apache.cassandra.db.Directories.snapshotLeveledManifest(Directories.java:343)
> > at
> >
> org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1450)
> > ... 33 more
> >
> >
> > However, an LS shows that both of these JSON files exist on the
> filesystem
> > (although slightly different sizes).
> >
> > [root@cassandra-n6 blotter]# ls -al
> > /var/lib/cassandra/data/blotter/twitter_users/twitter_users.json
> > -rw-r--r-- 1 root root 38786 May 15 20:51
> > /var/lib/cassandra/data/blotter/twitter_users/twitter_users.json
> >
> > [root@cassandra-n6 blotter]# ls -al
> >
> /var/lib/cassandra/data/blotter/twitter_users/snapshots/1337115022389/twitter_users.json
> > -rw-r--r-- 1 root root 38778 May 15 20:50
> >
> /var/lib/cassandra/data/blotter/twitter_users/snapshots/1337115022389/twitter_users.json
> >
> >
> > We are using Leveled Compaction on the twitter_users CF with I assume is
> > creating the JSON files.
> >
> > [root@cassandra-n6 blotter]# ls -al
> > /var/lib/cassandra/data/blotter/twitter_users/*.json
> > -rw-r--r-- 1 root root 38779 May 15 20:51
> > /var/lib/cassandra/data/blotter/twitter_users/twitter_users.json
> > -rw-r--r-- 1 root root 38779 May 15 20:51
> > /var/lib/cassandra/data/blotter/twitter_users/twitter_users-old.json
> > -rw-r--r-- 1 root root  1040 May 15 20:51
> >
> /var/lib/cassandra/data/blotter/twitter_users/twitter_users.twitter_user_attributes_screenname_idx.json
> > -rw-r--r-- 1 root root  1046 May 15 20:50
> >
> /var/lib/cassandra/data/blotter/twitter_users/twitter_users.twitter_user_attributes_screenname_idx-old.json
> >
> >
> > The other column families which are not using Leveled Compaction seem to
> > have their snapshots created successfully.
> >
> > Any ideas other than turning off Leveled Compaction?
> >
> >
> > Thanks,
> >
> > Brian
> >
> >
> >
> >
>

Mime
View raw message