Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E6259D062 for ; Tue, 10 Jul 2012 21:35:54 +0000 (UTC) Received: (qmail 54410 invoked by uid 500); 10 Jul 2012 21:35:52 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 54390 invoked by uid 500); 10 Jul 2012 21:35:52 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 54381 invoked by uid 99); 10 Jul 2012 21:35:52 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Jul 2012 21:35:52 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=FSL_RCVD_USER,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of frank.hsueh@gmail.com designates 209.85.212.178 as permitted sender) Received: from [209.85.212.178] (HELO mail-wi0-f178.google.com) (209.85.212.178) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 10 Jul 2012 21:35:44 +0000 Received: by wibhr14 with SMTP id hr14so326252wib.7 for ; Tue, 10 Jul 2012 14:35:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=jT2krlPc/SHoX8oe2x71qz8ToEnBAbf9QajW2mNJixg=; b=GYiInAWbsidsBK2c0M/Sf6hXAceSp7zpkKez5pUJU5g562gTAjOWbhH0y7wZ5S+/lU l2OHB7kH1jDYKUvmMNhTBvnDRpytLa15yQmf265vRuI1FznlLWIvQnLlH1Giep4skUDc CjR5Vsn2IwJScgucOyrveZf3Su4+AkKny9Dx7d1b2zemQE6SN3KZzO6+vHGDDMuZLFVN teSmJQ58QNlcOVEvbn6jA5is6riFug4bWXPqmCvJj6VJtqiVwQvbc8SmcsNYM1fxj0bN q1yeTZa/tZja7G9luEwzpQJOByhKJZYd1k5zw8iNaoMB/K6FABGaMkMLIabWMbH5380i aYYA== MIME-Version: 1.0 Received: by 10.180.93.99 with SMTP id ct3mr10892861wib.13.1341956123908; Tue, 10 Jul 2012 14:35:23 -0700 (PDT) Received: by 10.180.85.135 with HTTP; Tue, 10 Jul 2012 14:35:23 -0700 (PDT) Date: Tue, 10 Jul 2012 14:35:23 -0700 Message-ID: Subject: failed to delete commitlog, cassandra can't accept writes From: Frank Hsueh To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=f46d043c08e811d9a504c480827a --f46d043c08e811d9a504c480827a Content-Type: text/plain; charset=UTF-8 after reading the JIRA, I decided to use Java 6. with Casandra 1.1.2 on Java 6 x64 on Win7 sp1 x64 (all latest versions), after a several minutes of sustained writes, I see: from system.log: >>>> java.io.IOError: java.io.IOException: Failed to delete C:\var\lib\cassandra\commitlog\CommitLog-948695923996466.log at org.apache.cassandra.db.commitlog.CommitLogSegment.discard(CommitLogSegment.java:176) at org.apache.cassandra.db.commitlog.CommitLogAllocator$4.run(CommitLogAllocator.java:223) at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.IOException: Failed to delete C:\var\lib\cassandra\commitlog\CommitLog-948695923996466.log at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:54) at org.apache.cassandra.db.commitlog.CommitLogSegment.discard(CommitLogSegment.java:172) ... 4 more <<<< anybody seen this before? is this related to 4337 ? On Sat, Jul 7, 2012 at 6:36 PM, Frank Hsueh wrote: > bug already reported: > > https://issues.apache.org/jira/browse/CASSANDRA-4337 > > > > On Sat, Jul 7, 2012 at 6:26 PM, Frank Hsueh wrote: > >> Hi, >> >> I'm running Casandra 1.1.2 on Java 7 x64 on Win7 sp1 x64 (all latest >> versions). If it matters, I'm using a late version of Astyanax as my >> client. >> >> I'm using 4 threads to write a lot of data into a single CF. >> >> After several minutes of load (~ 30m at last incident), Cassandra stops >> accepting writes (client reports an OperationTimeoutException). I looked >> at the logs and I see on the Cassandra server: >> >> >>>> >> ERROR 18:00:42,807 Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main] >> java.io.IOError: java.io.IOException: Rename from >> \var\lib\cassandra\commitlog\CommitLog-701533048437587.log to >> 703272597990002 failed >> at >> org.apache.cassandra.db.commitlog.CommitLogSegment.(CommitLogSegment.java:127) >> at >> org.apache.cassandra.db.commitlog.CommitLogSegment.recycle(CommitLogSegment.java:204) >> at >> org.apache.cassandra.db.commitlog.CommitLogAllocator$2.run(CommitLogAllocator.java:166) >> at >> org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:95) >> at >> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) >> at java.lang.Thread.run(Thread.java:722) >> Caused by: java.io.IOException: Rename from >> \var\lib\cassandra\commitlog\CommitLog-701533048437587.log to >> 703272597990002 failed >> at >> org.apache.cassandra.db.commitlog.CommitLogSegment.(CommitLogSegment.java:105) >> ... 5 more >> <<<< >> >> Anybody else seen this before ? >> >> >> -- >> Frank Hsueh | frank.hsueh@gmail.com >> > > > > -- > Frank Hsueh | frank.hsueh@gmail.com > -- Frank Hsueh | frank.hsueh@gmail.com --f46d043c08e811d9a504c480827a Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
after reading the JIRA, I decided to use Java 6.

<= div>with Casandra 1.1.2 on Java 6 x64 on Win7 sp1 x64=C2=A0(all latest vers= ions), after a several minutes of sustained writes, I see:

from system.log:
>>>>
java.io.IOE= rror: java.io.IOException: Failed to delete C:\var\lib\cassandra\commitlog\= CommitLog-948695923996466.log
at org.apache.cassandra.db.commitlog.CommitLo= gSegment.discard(CommitLogSegment.java:176)
at or= g.apache.cassandra.db.commitlog.CommitLogAllocator$4.run(CommitLogAllocator= .java:223)
at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMay= Throw(CommitLogAllocator.java:95)
at or= g.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
=
at ja= va.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Failed to delete C:\var\lib\cassandra\= commitlog\CommitLog-948695923996466.log
at org.apache.cassandra.io.util.Fil= eUtils.deleteWithConfirm(FileUtils.java:54)
at or= g.apache.cassandra.db.commitlog.CommitLogSegment.discard(CommitLogSegment.j= ava:172)
... 4 more
<<<<

anybody seen this before? = =C2=A0is this related to 4337 ?



On Sat, Jul 7, 2012 at 6:36 PM, Frank= Hsueh <frank.hsueh@gmail.com> wrote:
bug already reported:

https://issues.apache.org/jira/browse/CASSANDRA-4337



On= Sat, Jul 7, 2012 at 6:26 PM, Frank Hsueh <frank.hsueh@gmail.com&g= t; wrote:
Hi,

I'm ru= nning Casandra 1.1.2 on Java 7 x64 on Win7 sp1 x64 (all latest versions). = =C2=A0If it matters, I'm using a late version of Astyanax as my client.=

I'm using 4 threads to write a lot of data into a s= ingle CF.

After several minutes of load (~ 30m at last incident),= Cassandra stops accepting writes (client reports an OperationTimeoutExcept= ion). =C2=A0I looked at the logs and I see on the Cassandra server:

>>>>
ERROR 18:00:42,807 Exception in t= hread Thread[COMMIT-LOG-ALLOCATOR,5,main]
java.io.IOError: java.i= o.IOException: Rename from \var\lib\cassandra\commitlog\CommitLog-701533048= 437587.log to 703272597990002 failed
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.cassandra.db.commitlog.Commi= tLogSegment.<init>(CommitLogSegment.java:127)
=C2=A0 =C2=A0= =C2=A0 =C2=A0 at org.apache.cassandra.db.commitlog.CommitLogSegment.recycl= e(CommitLogSegment.java:204)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.cassandra.db.commitlog.CommitLogA= llocator$2.run(CommitLogAllocator.java:166)
=C2=A0 =C2=A0 =C2=A0 = =C2=A0 at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThro= w(CommitLogAllocator.java:95)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.cassandra.utils.WrappedRunnable.r= un(WrappedRunnable.java:30)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at java.l= ang.Thread.run(Thread.java:722)
Caused by: java.io.IOException: R= ename from \var\lib\cassandra\commitlog\CommitLog-701533048437587.log to 70= 3272597990002 failed
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.cassandra.db.commitlog.Commi= tLogSegment.<init>(CommitLogSegment.java:105)
=C2=A0 =C2=A0= =C2=A0 =C2=A0 ... 5 more
<<<<

Anybody else seen this before ?


--
Frank Hsueh | frank.hsueh@gmail.com



--
Frank Hsueh | = frank.hsueh@gmai= l.com



--
Frank Hsueh = | frank.hsueh@gmail.com
--f46d043c08e811d9a504c480827a--