incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bryan Fernandez <bfernande...@gmail.com>
Subject Snapshot failing on JSON files in 1.1.0
Date Tue, 15 May 2012 21:08:27 GMT
Greetings,

We recently upgraded from 1.0.8 to 1.1.0. Everything has been running fine
with the exception of snapshots. When attempting to snapshot any of the
nodes in our six node cluster we are seeing the following error.

[root@cassandra-n6 blotter]# /opt/apache-cassandra-1.1.0/bin/nodetool -h
10.20.50.58 snapshot
Requested snapshot for: all keyspaces
Exception in thread "main" java.io.IOError: java.io.IOException: Unable to
create hard link from
/var/lib/cassandra/data/blotter/twitter_users/twitter_users.json to
/var/lib/cassandra/data/blotter/twitter_users/snapshots/1337115022389/twitter_users.json
(errno 17)
at
org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1454)
at
org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.java:1483)
at org.apache.cassandra.db.Table.snapshot(Table.java:205)
at
org.apache.cassandra.service.StorageService.takeSnapshot(StorageService.java:1793)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93)
at
com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27)
at
com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208)
at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120)
at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
at
javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427)
at
javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
at
javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265)
at
javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360)
at
javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:303)
at sun.rmi.transport.Transport$1.run(Transport.java:159)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Unable to create hard link from
/var/lib/cassandra/data/blotter/twitter_users/twitter_users.json to
/var/lib/cassandra/data/blotter/twitter_users/snapshots/1337115022389/twitter_users.json
(errno 17)
at org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:163)
at
org.apache.cassandra.db.Directories.snapshotLeveledManifest(Directories.java:343)
at
org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnFamilyStore.java:1450)
... 33 more


However, an LS shows that both of these JSON files exist on the filesystem
(although slightly different sizes).

[root@cassandra-n6 blotter]# ls -al
/var/lib/cassandra/data/blotter/twitter_users/twitter_users.json
-rw-r--r-- 1 root root 38786 May 15 20:51
/var/lib/cassandra/data/blotter/twitter_users/twitter_users.json

[root@cassandra-n6 blotter]# ls -al
/var/lib/cassandra/data/blotter/twitter_users/snapshots/1337115022389/twitter_users.json
-rw-r--r-- 1 root root 38778 May 15 20:50
/var/lib/cassandra/data/blotter/twitter_users/snapshots/1337115022389/twitter_users.json


We are using Leveled Compaction on the twitter_users CF with I assume is
creating the JSON files.

[root@cassandra-n6 blotter]# ls -al
/var/lib/cassandra/data/blotter/twitter_users/*.json
-rw-r--r-- 1 root root 38779 May 15 20:51
/var/lib/cassandra/data/blotter/twitter_users/twitter_users.json
-rw-r--r-- 1 root root 38779 May 15 20:51
/var/lib/cassandra/data/blotter/twitter_users/twitter_users-old.json
-rw-r--r-- 1 root root  1040 May 15 20:51
/var/lib/cassandra/data/blotter/twitter_users/twitter_users.twitter_user_attributes_screenname_idx.json
-rw-r--r-- 1 root root  1046 May 15 20:50
/var/lib/cassandra/data/blotter/twitter_users/twitter_users.twitter_user_attributes_screenname_idx-old.json


The other column families which are not using Leveled Compaction seem to
have their snapshots created successfully.

Any ideas other than turning off Leveled Compaction?


Thanks,

Brian

Mime
View raw message