From user-return-27083-apmail-cassandra-user-archive=cassandra.apache.org@cassandra.apache.org Tue Jun 19 09:56:42 2012 Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 9BEEFDFB8 for ; Tue, 19 Jun 2012 09:56:42 +0000 (UTC) Received: (qmail 5185 invoked by uid 500); 19 Jun 2012 09:56:40 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 4805 invoked by uid 500); 19 Jun 2012 09:56:40 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 4784 invoked by uid 99); 19 Jun 2012 09:56:39 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Jun 2012 09:56:39 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=FSL_RCVD_USER,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of arodrime@gmail.com designates 209.85.217.172 as permitted sender) Received: from [209.85.217.172] (HELO mail-lb0-f172.google.com) (209.85.217.172) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 19 Jun 2012 09:56:34 +0000 Received: by lbbgo11 with SMTP id go11so6052642lbb.31 for ; Tue, 19 Jun 2012 02:56:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type:content-transfer-encoding; bh=9uIUgtbFyX4Y/aLytc1o0DhxbfzguESxAysVHwviySw=; b=S96wCWE4IDW2nbnjO6olwyztXJDD11wZZ3sgHLthScPApJfCz6c/kJbXosRN/2ZHAn s6X01kboI2rDC64KoexUWVPRevQqMZqpMhIrkyKjYV453mxGa83WMEYIqLvf/F6OVj31 Fy+1mNGxpwYSC9REHkjtyvMh3vgB1xAG3XvftsYVC/UwZEBwdqGXxDpq+nf1Mx/AMOLz Hf6Epait10A/4ElEAPeiP1qIaQhvqCF+GQq1sKx1SNtLy/CM8D0Dy4tomKJZ4zvpInLn Ydk4pDnZWO/D1k4h7WQo5kDIMTK0tXx+dVJ62e6bYvkYlxbdCnYQu24nPawMIjy8Ngzx HTDw== Received: by 10.112.11.38 with SMTP id n6mr7662547lbb.82.1340099772523; Tue, 19 Jun 2012 02:56:12 -0700 (PDT) MIME-Version: 1.0 Received: by 10.114.21.134 with HTTP; Tue, 19 Jun 2012 02:55:52 -0700 (PDT) In-Reply-To: References: <00B4C98F-E351-4CB6-923B-F5E19A0303F2@thelastpickle.com> From: Alain RODRIGUEZ Date: Tue, 19 Jun 2012 11:55:52 +0200 Message-ID: Subject: Re: Snapshot failing on JSON files in 1.1.0 To: user@cassandra.apache.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org Hi again, apt-get install libjna-java installed nothing, I was already up to date. I made the symbolic link jna.jar to target jna-3.4.1.jar (downloaded @ the given link) instead of jna-3.2.4.jar. I could restart with the 'JNA mlockall successful' message. I am still unable to snapshot my data. I got the following output : Exception in thread "main" java.io.IOError: java.io.IOException: Unable to create hard link from /raid0/cassandra/data/cassa_teads/stats_product-hc-233-Data.db to /raid0/cassandra/data/cassa_teads/snapshots/1340099026781/stats_product-hc-= 233-Data.db (errno 1) at org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(C= olumnFamilyStore.java:1433) at org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyS= tore.java:1462) at org.apache.cassandra.db.Table.snapshot(Table.java:210) at org.apache.cassandra.service.StorageService.takeSnapshot(Storage= Service.java:1710) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessor= Impl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethod= AccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Stand= ardMBeanIntrospector.java:93) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Stand= ardMBeanIntrospector.java:27) at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospec= tor.java:208) at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:12= 0) at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:26= 2) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Def= aultMBeanServerInterceptor.java:836) at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.jav= a:761) at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMICon= nectionImpl.java:1427) at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConn= ectionImpl.java:72) at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperatio= n.run(RMIConnectionImpl.java:1265) at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperat= ion(RMIConnectionImpl.java:1360) at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnecti= onImpl.java:788) at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethod= AccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:3= 03) at sun.rmi.transport.Transport$1.run(Transport.java:159) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:155) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.j= ava:535) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTra= nsport.java:790) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTran= sport.java:649) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoo= lExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExe= cutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.IOException: Unable to create hard link from /raid0/cassandra/data/cassa_teads/stats_product-hc-233-Data.db to /raid0/cassandra/data/cassa_teads/snapshots/1340099026781/stats_product-hc-= 233-Data.db (errno 1) at org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java= :158) at org.apache.cassandra.io.sstable.SSTableReader.createLinks(SSTabl= eReader.java:857) at org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(C= olumnFamilyStore.java:1412) ... 32 more Logs tell me this : ERROR 09:43:46,840 Unable to create hard link com.sun.jna.LastErrorException: [1]=C3X at org.apache.cassandra.utils.CLibrary.link(Native Method) at org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java= :145) at org.apache.cassandra.io.sstable.SSTableReader.createLinks(SSTabl= eReader.java:857) at org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(C= olumnFamilyStore.java:1412) at org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyS= tore.java:1462) at org.apache.cassandra.db.Table.snapshot(Table.java:210) at org.apache.cassandra.service.StorageService.takeSnapshot(Storage= Service.java:1710) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessor= Impl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethod= AccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Stand= ardMBeanIntrospector.java:93) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(Stand= ardMBeanIntrospector.java:27) at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospec= tor.java:208) at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:12= 0) at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:26= 2) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Def= aultMBeanServerInterceptor.java:836) at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.jav= a:761) at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMICon= nectionImpl.java:1427) at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConn= ectionImpl.java:72) at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperatio= n.run(RMIConnectionImpl.java:1265) at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperat= ion(RMIConnectionImpl.java:1360) at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnecti= onImpl.java:788) at sun.reflect.GeneratedMethodAccessor42.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethod= AccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:3= 03) at sun.rmi.transport.Transport$1.run(Transport.java:159) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:155) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.j= ava:535) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTra= nsport.java:790) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTran= sport.java:649) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoo= lExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExe= cutor.java:908) at java.lang.Thread.run(Thread.java:662) The weird thing is that I have the folder "snapshot" created and an sstable inside : $ ll -h /raid0/cassandra/data/cassa_teads/snapshots/1340099026781/ total 56M drwxr-xr-x 2 cassandra cassandra 4.0K 2012-06-19 09:43 ./ drwxr-xr-x 3 cassandra cassandra 26 2012-06-19 09:43 ../ -rw-r--r-- 2 cassandra cassandra 56M 2012-05-24 17:58 stats_product-hc-2372-Data.db -rw-r--r-- 2 cassandra cassandra 71 2012-05-24 17:58 stats_product-hc-2372-Digest.sha1 -rw-r--r-- 2 cassandra cassandra 3.3K 2012-05-24 17:58 stats_product-hc-2372-Filter.db -rw-r--r-- 2 cassandra cassandra 11K 2012-05-24 17:58 stats_product-hc-2372-Index.db -rw-r--r-- 2 cassandra cassandra 4.3K 2012-05-24 17:58 stats_product-hc-2372-Statistics.db Not being able to create snapshot is really annoying, any idea ? Alain 2012/6/13 Joaquin Casares : > Hello Alain, > > Yes, the AMI is geared to working "out of the box" for most dev purposes. > > We recently spotted an issue with JNA 3.2.7 on Ubuntu 10.10 not being pic= ked > up. > > You can try running `apt-get install libjna-java` but in order for the > change to be activated, you must restart your Cassandra service. > > Could you try removing the older version in the=A0/usr/share/java/ folder= and > restarting your Cassandra service? You should see a 'JNA mlockall > successful' message in your system.log. If not, then remove the remaining > jna.jar and replace it with the newest jna.jar as found > here:=A0https://github.com/twall/jna. > > Upon another restart of your node, you should see the 'JNA mlockall > successful' message as well as the ability to run the snapshots. > > I was unable to replicate this on a new instance so all new launches shou= ld > include the patched code. > > Do let me know if anyone else sees this issue. > > Thanks, > > Joaquin Casares > DataStax > Software Engineer/Support > > > > On Wed, Jun 13, 2012 at 10:28 AM, Alain RODRIGUEZ > wrote: >> >> if I do : locate jna >> >> /opt/java/64/jdk1.6.0_31/db/docs/html/ref/rrefsqljnaturaljoin.html >> /root/.m2/repository/net/java/dev/jna >> /root/.m2/repository/net/java/dev/jna/jna >> /root/.m2/repository/net/java/dev/jna/jna/3.2.7 >> /root/.m2/repository/net/java/dev/jna/jna/3.2.7/jna-3.2.7-sources.jar >> /root/.m2/repository/net/java/dev/jna/jna/3.2.7/jna-3.2.7-sources.jar.sh= a1 >> /root/.m2/repository/net/java/dev/jna/jna/3.2.7/jna-3.2.7.jar >> /root/.m2/repository/net/java/dev/jna/jna/3.2.7/jna-3.2.7.jar.sha1 >> /root/.m2/repository/net/java/dev/jna/jna/3.2.7/jna-3.2.7.pom >> /root/.m2/repository/net/java/dev/jna/jna/3.2.7/jna-3.2.7.pom.sha1 >> /usr/share/doc/libjna-java >> /usr/share/doc/libjna-java/README.Debian >> /usr/share/doc/libjna-java/changelog.Debian.gz >> /usr/share/doc/libjna-java/copyright >> /usr/share/java/jna-3.2.4.jar >> /usr/share/java/jna.jar >> /usr/share/maven-repo/net/java/dev/jna >> /usr/share/maven-repo/net/java/dev/jna/jna >> /usr/share/maven-repo/net/java/dev/jna/jna/3.2.4 >> /usr/share/maven-repo/net/java/dev/jna/jna/debian >> /usr/share/maven-repo/net/java/dev/jna/jna/3.2.4/jna-3.2.4.jar >> /usr/share/maven-repo/net/java/dev/jna/jna/3.2.4/jna-3.2.4.pom >> /usr/share/maven-repo/net/java/dev/jna/jna/debian/jna-debian.jar >> /usr/share/maven-repo/net/java/dev/jna/jna/debian/jna-debian.pom >> /var/cache/apt/archives/libjna-java_3.2.4-2_amd64.deb >> /var/lib/dpkg/info/libjna-java.list >> /var/lib/dpkg/info/libjna-java.md5sums >> >> So what version am I using (jna 3.2.7 or 3.2.4 ?) >> >> Should I do an "apt-get install libjna-java" and does this need a restar= t >> ? >> >> Alain >> >> 2012/6/13 Alain RODRIGUEZ : >> > Hi Aaron, >> > >> > We are using Ubuntu (AMI Datastax 1.0.9 as I said). >> > >> > Release: =A0 =A0 =A0 =A010.10 >> > Codename: =A0 =A0 =A0 maverick >> > >> > ERROR [RMI TCP Connection(37732)-10.248.10.94] 2012-06-13 15:00:17,157 >> > CLibrary.java (line 153) Unable to create hard link >> > com.sun.jna.LastErrorException: errno was 1 >> > =A0 =A0 =A0 =A0at org.apache.cassandra.utils.CLibrary.link(Native Meth= od) >> > =A0 =A0 =A0 =A0at >> > org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:145) >> > =A0 =A0 =A0 =A0at >> > org.apache.cassandra.io.sstable.SSTableReader.createLinks(SSTableReade= r.java:857) >> > =A0 =A0 =A0 =A0at >> > org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(ColumnF= amilyStore.java:1412) >> > =A0 =A0 =A0 =A0at >> > org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.j= ava:1462) >> > =A0 =A0 =A0 =A0at org.apache.cassandra.db.Table.snapshot(Table.java:21= 0) >> > =A0 =A0 =A0 =A0at >> > org.apache.cassandra.service.StorageService.takeSnapshot(StorageServic= e.java:1710) >> > =A0 =A0 =A0 =A0at sun.reflect.NativeMethodAccessorImpl.invoke0(Native = Method) >> > =A0 =A0 =A0 =A0at >> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j= ava:39) >> > =A0 =A0 =A0 =A0at >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess= orImpl.java:25) >> > =A0 =A0 =A0 =A0at java.lang.reflect.Method.invoke(Method.java:597) >> > =A0 =A0 =A0 =A0at >> > com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBe= anIntrospector.java:93) >> > =A0 =A0 =A0 =A0at >> > com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBe= anIntrospector.java:27) >> > =A0 =A0 =A0 =A0at >> > com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.ja= va:208) >> > =A0 =A0 =A0 =A0at >> > com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120) >> > =A0 =A0 =A0 =A0at >> > com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262) >> > =A0 =A0 =A0 =A0at >> > com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMB= eanServerInterceptor.java:836) >> > =A0 =A0 =A0 =A0at >> > com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761) >> > =A0 =A0 =A0 =A0at >> > javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectio= nImpl.java:1427) >> > =A0 =A0 =A0 =A0at >> > javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnection= Impl.java:72) >> > =A0 =A0 =A0 =A0at >> > javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(= RMIConnectionImpl.java:1265) >> > =A0 =A0 =A0 =A0at >> > javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RM= IConnectionImpl.java:1360) >> > =A0 =A0 =A0 =A0at >> > javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl= .java:788) >> > =A0 =A0 =A0 =A0at sun.reflect.GeneratedMethodAccessor46.invoke(Unknown= Source) >> > =A0 =A0 =A0 =A0at >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess= orImpl.java:25) >> > =A0 =A0 =A0 =A0at java.lang.reflect.Method.invoke(Method.java:597) >> > =A0 =A0 =A0 =A0at >> > sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:303) >> > =A0 =A0 =A0 =A0at sun.rmi.transport.Transport$1.run(Transport.java:159= ) >> > =A0 =A0 =A0 =A0at java.security.AccessController.doPrivileged(Native M= ethod) >> > =A0 =A0 =A0 =A0at sun.rmi.transport.Transport.serviceCall(Transport.ja= va:155) >> > =A0 =A0 =A0 =A0at >> > sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:53= 5) >> > =A0 =A0 =A0 =A0at >> > sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport= .java:790) >> > =A0 =A0 =A0 =A0at >> > sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.= java:649) >> > =A0 =A0 =A0 =A0at >> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecu= tor.java:886) >> > =A0 =A0 =A0 =A0at >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.= java:908) >> > =A0 =A0 =A0 =A0at java.lang.Thread.run(Thread.java:662) >> > =A0WARN [RMI TCP Connection(37732)-10.248.10.94] 2012-06-13 15:00:17,1= 58 >> > CLibrary.java (line 97) Obsolete version of JNA present; unable to >> > read errno. Upgrade to JNA 3.2.7 or later >> > >> > I see that a JNA update seems necessary. Shouldn't this AMI be fully >> > worrking "out of the box" ? >> > >> > I am going to search how to upgrade JNA. >> > >> > Alain >> > >> > 2012/5/31 aaron morton : >> >> CASSANDRA-4230 is a bug in 1.1 >> >> >> >> I am not aware of issues using snapshot on 1.0.9. But errno 0 is a bi= t >> >> odd. >> >> >> >> On the server side there should be a log message at ERROR level that >> >> contains the string "Unable to create hard link" and the error messag= e. >> >> What >> >> does that say ? >> >> >> >> Can you also include the OS version. >> >> >> >> Cheers >> >> >> >> ----------------- >> >> Aaron Morton >> >> Freelance Developer >> >> @aaronmorton >> >> http://www.thelastpickle.com >> >> >> >> On 28/05/2012, at 9:27 PM, Alain RODRIGUEZ wrote: >> >> >> >> I have the same error with the last Datastax AMI (1.0.9). Is that the >> >> same >> >> bug ? >> >> >> >> Requested snapshot for: cassa_teads >> >> Exception in thread "main" java.io.IOError: java.io.IOException: >> >> Unable to create hard link from >> >> /raid0/cassandra/data/cassa_teads/stats_product-hc-233-Index.db to >> >> >> >> /raid0/cassandra/data/cassa_teads/snapshots/20120528/stats_product-hc= -233-Index.db >> >> (errno 0) >> >> =A0=A0=A0=A0=A0=A0=A0at >> >> >> >> org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(Column= FamilyStore.java:1433) >> >> =A0=A0=A0=A0=A0=A0=A0at >> >> >> >> org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.= java:1462) >> >> =A0=A0=A0=A0=A0=A0=A0at org.apache.cassandra.db.Table.snapshot(Table.= java:210) >> >> =A0=A0=A0=A0=A0=A0=A0at >> >> >> >> org.apache.cassandra.service.StorageService.takeSnapshot(StorageServi= ce.java:1710) >> >> =A0=A0=A0=A0=A0=A0=A0at sun.reflect.NativeMethodAccessorImpl.invoke0(= Native Method) >> >> =A0=A0=A0=A0=A0=A0=A0at >> >> >> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.= java:39) >> >> =A0=A0=A0=A0=A0=A0=A0at >> >> >> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces= sorImpl.java:25) >> >> =A0=A0=A0=A0=A0=A0=A0at java.lang.reflect.Method.invoke(Method.java:5= 97) >> >> =A0=A0=A0=A0=A0=A0=A0at >> >> >> >> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMB= eanIntrospector.java:93) >> >> =A0=A0=A0=A0=A0=A0=A0at >> >> >> >> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMB= eanIntrospector.java:27) >> >> =A0=A0=A0=A0=A0=A0=A0at >> >> >> >> com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.j= ava:208) >> >> =A0=A0=A0=A0=A0=A0=A0at >> >> com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120) >> >> =A0=A0=A0=A0=A0=A0=A0at >> >> com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262) >> >> =A0=A0=A0=A0=A0=A0=A0at >> >> >> >> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultM= BeanServerInterceptor.java:836) >> >> =A0=A0=A0=A0=A0=A0=A0at >> >> com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761= ) >> >> =A0=A0=A0=A0=A0=A0=A0at >> >> >> >> javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnecti= onImpl.java:1427) >> >> =A0=A0=A0=A0=A0=A0=A0at >> >> >> >> javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectio= nImpl.java:72) >> >> =A0=A0=A0=A0=A0=A0=A0at >> >> >> >> javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run= (RMIConnectionImpl.java:1265) >> >> =A0=A0=A0=A0=A0=A0=A0at >> >> >> >> javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(R= MIConnectionImpl.java:1360) >> >> =A0=A0=A0=A0=A0=A0=A0at >> >> >> >> javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImp= l.java:788) >> >> =A0=A0=A0=A0=A0=A0=A0at sun.reflect.GeneratedMethodAccessor50.invoke(= Unknown Source) >> >> =A0=A0=A0=A0=A0=A0=A0at >> >> >> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces= sorImpl.java:25) >> >> =A0=A0=A0=A0=A0=A0=A0at java.lang.reflect.Method.invoke(Method.java:5= 97) >> >> =A0=A0=A0=A0=A0=A0=A0at >> >> sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:303) >> >> =A0=A0=A0=A0=A0=A0=A0at sun.rmi.transport.Transport$1.run(Transport.j= ava:159) >> >> =A0=A0=A0=A0=A0=A0=A0at java.security.AccessController.doPrivileged(N= ative Method) >> >> =A0=A0=A0=A0=A0=A0=A0at sun.rmi.transport.Transport.serviceCall(Trans= port.java:155) >> >> =A0=A0=A0=A0=A0=A0=A0at >> >> >> >> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:5= 35) >> >> =A0=A0=A0=A0=A0=A0=A0at >> >> >> >> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTranspor= t.java:790) >> >> =A0=A0=A0=A0=A0=A0=A0at >> >> >> >> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport= .java:649) >> >> =A0=A0=A0=A0=A0=A0=A0at >> >> >> >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExec= utor.java:886) >> >> =A0=A0=A0=A0=A0=A0=A0at >> >> >> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor= .java:908) >> >> =A0=A0=A0=A0=A0=A0=A0at java.lang.Thread.run(Thread.java:662) >> >> Caused by: java.io.IOException: Unable to create hard link from >> >> /raid0/cassandra/data/cassa_teads/stats_product-hc-233-Index.db to >> >> >> >> /raid0/cassandra/data/cassa_teads/snapshots/20120528/stats_product-hc= -233-Index.db >> >> (errno 0) >> >> =A0=A0=A0=A0=A0=A0=A0at >> >> org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:158) >> >> =A0=A0=A0=A0=A0=A0=A0at >> >> >> >> org.apache.cassandra.io.sstable.SSTableReader.createLinks(SSTableRead= er.java:857) >> >> =A0=A0=A0=A0=A0=A0=A0at >> >> >> >> org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(Column= FamilyStore.java:1412) >> >> =A0=A0=A0=A0=A0=A0=A0... 32 more >> >> >> >> Can we do a snapshot manually (like flushing and after copying all th= e >> >> file into the snapshot folder) ? >> >> >> >> Alain >> >> >> >> 2012/5/19 Jonathan Ellis : >> >> >> >> When these bugs are fixed: >> >> >> >> >> >> https://issues.apache.org/jira/secure/IssueNavigator.jspa?reset=3Dtru= e&jqlQuery=3Dproject+%3D+CASSANDRA+AND+fixVersion+%3D+%221.1.1%22+AND+resol= ution+%3D+Unresolved+ORDER+BY+due+ASC%2C+priority+DESC%2C+created+ASC&mode= =3Dhide >> >> >> >> >> >> On Wed, May 16, 2012 at 6:35 PM, Bryan Fernandez >> >> >> >> wrote: >> >> >> >> Does anyone know when 1.1.1 will be released? >> >> >> >> >> >> Thanks. >> >> >> >> >> >> On Tue, May 15, 2012 at 5:40 PM, Brandon Williams >> >> wrote: >> >> >> >> >> >> Probably https://issues.apache.org/jira/browse/CASSANDRA-4230 >> >> >> >> >> >> On Tue, May 15, 2012 at 4:08 PM, Bryan Fernandez >> >> >> >> >> >> wrote: >> >> >> >> Greetings, >> >> >> >> >> >> We recently upgraded from 1.0.8 to 1.1.0. Everything has been running >> >> >> >> fine >> >> >> >> with the exception of snapshots. When attempting to snapshot any of t= he >> >> >> >> nodes in our six node cluster we are seeing the following error. >> >> >> >> >> >> [root@cassandra-n6 blotter]# /opt/apache-cassandra-1.1.0/bin/nodetool >> >> -h >> >> >> >> 10.20.50.58 snapshot >> >> >> >> Requested snapshot for: all keyspaces >> >> >> >> Exception in thread "main" java.io.IOError: java.io.IOException: Unab= le >> >> >> >> to >> >> >> >> create hard link from >> >> >> >> /var/lib/cassandra/data/blotter/twitter_users/twitter_users.json to >> >> >> >> >> >> >> >> /var/lib/cassandra/data/blotter/twitter_users/snapshots/1337115022389= /twitter_users.json >> >> >> >> (errno 17) >> >> >> >> at >> >> >> >> >> >> >> >> org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(Column= FamilyStore.java:1454) >> >> >> >> at >> >> >> >> >> >> >> >> org.apache.cassandra.db.ColumnFamilyStore.snapshot(ColumnFamilyStore.= java:1483) >> >> >> >> at org.apache.cassandra.db.Table.snapshot(Table.java:205) >> >> >> >> at >> >> >> >> >> >> >> >> org.apache.cassandra.service.StorageService.takeSnapshot(StorageServi= ce.java:1793) >> >> >> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> >> >> >> at >> >> >> >> >> >> >> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.= java:39) >> >> >> >> at >> >> >> >> >> >> >> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces= sorImpl.java:25) >> >> >> >> at java.lang.reflect.Method.invoke(Method.java:597) >> >> >> >> at >> >> >> >> >> >> >> >> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMB= eanIntrospector.java:93) >> >> >> >> at >> >> >> >> >> >> >> >> com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMB= eanIntrospector.java:27) >> >> >> >> at >> >> >> >> >> >> >> >> com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.j= ava:208) >> >> >> >> at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120) >> >> >> >> at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262) >> >> >> >> at >> >> >> >> >> >> >> >> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultM= BeanServerInterceptor.java:836) >> >> >> >> at >> >> >> >> com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761= ) >> >> >> >> at >> >> >> >> >> >> >> >> javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnecti= onImpl.java:1427) >> >> >> >> at >> >> >> >> >> >> >> >> javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectio= nImpl.java:72) >> >> >> >> at >> >> >> >> >> >> >> >> javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run= (RMIConnectionImpl.java:1265) >> >> >> >> at >> >> >> >> >> >> >> >> javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(R= MIConnectionImpl.java:1360) >> >> >> >> at >> >> >> >> >> >> >> >> javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImp= l.java:788) >> >> >> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> >> >> >> at >> >> >> >> >> >> >> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.= java:39) >> >> >> >> at >> >> >> >> >> >> >> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces= sorImpl.java:25) >> >> >> >> at java.lang.reflect.Method.invoke(Method.java:597) >> >> >> >> at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:303= ) >> >> >> >> at sun.rmi.transport.Transport$1.run(Transport.java:159) >> >> >> >> at java.security.AccessController.doPrivileged(Native Method) >> >> >> >> at sun.rmi.transport.Transport.serviceCall(Transport.java:155) >> >> >> >> at >> >> >> >> >> >> sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:5= 35) >> >> >> >> at >> >> >> >> >> >> >> >> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTranspor= t.java:790) >> >> >> >> at >> >> >> >> >> >> >> >> sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport= .java:649) >> >> >> >> at >> >> >> >> >> >> >> >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExec= utor.java:886) >> >> >> >> at >> >> >> >> >> >> >> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor= .java:908) >> >> >> >> at java.lang.Thread.run(Thread.java:662) >> >> >> >> Caused by: java.io.IOException: Unable to create hard link from >> >> >> >> /var/lib/cassandra/data/blotter/twitter_users/twitter_users.json to >> >> >> >> >> >> >> >> /var/lib/cassandra/data/blotter/twitter_users/snapshots/1337115022389= /twitter_users.json >> >> >> >> (errno 17) >> >> >> >> at >> >> org.apache.cassandra.utils.CLibrary.createHardLink(CLibrary.java:163) >> >> >> >> at >> >> >> >> >> >> >> >> org.apache.cassandra.db.Directories.snapshotLeveledManifest(Directori= es.java:343) >> >> >> >> at >> >> >> >> >> >> >> >> org.apache.cassandra.db.ColumnFamilyStore.snapshotWithoutFlush(Column= FamilyStore.java:1450) >> >> >> >> ... 33 more >> >> >> >> >> >> >> >> However, an LS shows that both of these JSON files exist on the >> >> >> >> filesystem >> >> >> >> (although slightly different sizes). >> >> >> >> >> >> [root@cassandra-n6 blotter]# ls -al >> >> >> >> /var/lib/cassandra/data/blotter/twitter_users/twitter_users.json >> >> >> >> -rw-r--r-- 1 root root 38786 May 15 20:51 >> >> >> >> /var/lib/cassandra/data/blotter/twitter_users/twitter_users.json >> >> >> >> >> >> [root@cassandra-n6 blotter]# ls -al >> >> >> >> >> >> >> >> /var/lib/cassandra/data/blotter/twitter_users/snapshots/1337115022389= /twitter_users.json >> >> >> >> -rw-r--r-- 1 root root 38778 May 15 20:50 >> >> >> >> >> >> >> >> /var/lib/cassandra/data/blotter/twitter_users/snapshots/1337115022389= /twitter_users.json >> >> >> >> >> >> >> >> We are using Leveled Compaction on the twitter_users CF with I assume >> >> is >> >> >> >> creating the JSON files. >> >> >> >> >> >> [root@cassandra-n6 blotter]# ls -al >> >> >> >> /var/lib/cassandra/data/blotter/twitter_users/*.json >> >> >> >> -rw-r--r-- 1 root root 38779 May 15 20:51 >> >> >> >> /var/lib/cassandra/data/blotter/twitter_users/twitter_users.json >> >> >> >> -rw-r--r-- 1 root root 38779 May 15 20:51 >> >> >> >> /var/lib/cassandra/data/blotter/twitter_users/twitter_users-old.json >> >> >> >> -rw-r--r-- 1 root root =A01040 May 15 20:51 >> >> >> >> >> >> >> >> /var/lib/cassandra/data/blotter/twitter_users/twitter_users.twitter_u= ser_attributes_screenname_idx.json >> >> >> >> -rw-r--r-- 1 root root =A01046 May 15 20:50 >> >> >> >> >> >> >> >> /var/lib/cassandra/data/blotter/twitter_users/twitter_users.twitter_u= ser_attributes_screenname_idx-old.json >> >> >> >> >> >> >> >> The other column families which are not using Leveled Compaction seem >> >> to >> >> >> >> have their snapshots created successfully. >> >> >> >> >> >> Any ideas other than turning off Leveled Compaction? >> >> >> >> >> >> >> >> Thanks, >> >> >> >> >> >> Brian >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> >> >> Jonathan Ellis >> >> >> >> Project Chair, Apache Cassandra >> >> >> >> co-founder of DataStax, the source for professional Cassandra support >> >> >> >> http://www.datastax.com >> >> >> >> > >