Return-Path: Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: (qmail 65709 invoked from network); 19 Aug 2010 05:55:55 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 19 Aug 2010 05:55:55 -0000 Received: (qmail 47622 invoked by uid 500); 19 Aug 2010 05:55:53 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 47375 invoked by uid 500); 19 Aug 2010 05:55:49 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 47274 invoked by uid 99); 19 Aug 2010 05:55:47 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 Aug 2010 05:55:47 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of rantav@gmail.com designates 209.85.216.172 as permitted sender) Received: from [209.85.216.172] (HELO mail-qy0-f172.google.com) (209.85.216.172) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 19 Aug 2010 05:55:43 +0000 Received: by qyk1 with SMTP id 1so2679822qyk.10 for ; Wed, 18 Aug 2010 22:55:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:mime-version:received:in-reply-to :references:from:date:message-id:subject:to:content-type; bh=wV0FhjiSu+DZL+DqUyXQmPq4uVPpOy/fjfue34droJ4=; b=mP2sfYR0EL04yZ+V+ytNKvtimQNK5Jy6VsRQiiqG5Jmtl3aPMWzI5aF/KqtD9N3/Xq QZJBieR3f7f85hL9eEUTdmZMsvXp0m+MH+oPBTSkkphkHTwd72VsuATx0OPxQBXe7okU W/gmnLBhX/dRPUCtdRfG4Ng+iAA+YVmHTF6tY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; b=HlXB3C8W7n+FHBNaUytEjbXt9/i4Q66bL1jCTY8tU3W5h9QS/BVipNhU9/VKDdHQJQ FPbRpHkd37eGmlZIzGx+r1R/P6LXWvr2YetpPS4yVrU3oIbtLg7H3X1npXFpS54S7CKa f1zu9uYGY3eoNZMFaqm+ZyspGJ8eHz35bCcqo= Received: by 10.229.184.149 with SMTP id ck21mr6796317qcb.138.1282197322126; Wed, 18 Aug 2010 22:55:22 -0700 (PDT) MIME-Version: 1.0 Received: by 10.229.235.21 with HTTP; Wed, 18 Aug 2010 22:55:02 -0700 (PDT) In-Reply-To: References: From: Ran Tavory Date: Thu, 19 Aug 2010 08:55:02 +0300 Message-ID: Subject: Re: File write errors but cassandra isn't crashing To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=0016e65bbaf4ead5af048e26d3b5 --0016e65bbaf4ead5af048e26d3b5 Content-Type: text/plain; charset=UTF-8 I opened as an improvement suggestion: https://issues.apache.org/jira/browse/CASSANDRA-1409 On Mon, Aug 16, 2010 at 8:26 PM, Benjamin Black wrote: > Useful config option, perhaps? > > On Mon, Aug 16, 2010 at 8:51 AM, Jonathan Ellis wrote: > > That's a tough call -- you can also come up with scenarios where you'd > > rather have it read-only than completely dead. > > > > On Wed, Aug 11, 2010 at 12:38 PM, Ran Tavory wrote: > >> Due to administrative error one of the hosts in the cluster lost > permission > >> to write to it's data directory. > >> So I started seeing errors in the log, however, the server continued > serving > >> traffic. It wasn't able to compact and do other write operations but it > >> didn't crash. > >> I was wondering wether that's by design and if so, is this a good one... > I > >> guess I want to know if really bad things happen to my cluster... > >> logs look like that... > >> > >> INFO [FLUSH-TIMER] 2010-08-11 07:53:14,683 ColumnFamilyStore.java (line > >> 357) KvAds has reached its threshold; switching in a fresh Memtable at > >> CommitLogContext(file='/outbrain/cassandra/commitlog/Commi > >> tLog-1281505164614.log', position=88521163) > >> INFO [FLUSH-TIMER] 2010-08-11 07:53:14,683 ColumnFamilyStore.java (line > >> 609) Enqueuing flush of Memtable(KvAds)@851225759 > >> INFO [FLUSH-WRITER-POOL:1] 2010-08-11 07:53:14,684 Memtable.java (line > 148) > >> Writing Memtable(KvAds)@851225759 > >> ERROR [FLUSH-WRITER-POOL:1] 2010-08-11 07:53:14,688 > >> DebuggableThreadPoolExecutor.java (line 94) Error in executor futuretask > >> java.util.concurrent.ExecutionException: java.lang.RuntimeException: > >> java.io.FileNotFoundException: > >> /outbrain/cassandra/data/outbrain_kvdb/KvAds-tmp-249-Data.db (Permission > >> denied) > >> at > >> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) > >> at java.util.concurrent.FutureTask.get(FutureTask.java:83) > >> at > >> > org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:86) > >> at > >> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888) > >> at > >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > >> at java.lang.Thread.run(Thread.java:619) > >> Caused by: java.lang.RuntimeException: java.io.FileNotFoundException: > >> /outbrain/cassandra/data/outbrain_kvdb/KvAds-tmp-249-Data.db (Permission > >> denied) > >> at > >> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34) > >> at > >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > >> at > >> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > >> at java.util.concurrent.FutureTask.run(FutureTask.java:138) > >> at > >> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > >> ... more > > > > > > > > -- > > Jonathan Ellis > > Project Chair, Apache Cassandra > > co-founder of Riptano, the source for professional Cassandra support > > http://riptano.com > > > --0016e65bbaf4ead5af048e26d3b5 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
I opened as an improvement suggestion:=C2=A0https://issues.apache.or= g/jira/browse/CASSANDRA-1409

On Mon, = Aug 16, 2010 at 8:26 PM, Benjamin Black <b@b3k.us> wrote:
Useful config option, perhaps?

On Mon, Aug 16, 2010 at 8:51 AM, Jonathan Ellis <jbellis@gmail.com> wrote:
> That's a tough call -- you can also come up with scenarios where y= ou'd
> rather have it read-only than completely dead.
>
> On Wed, Aug 11, 2010 at 12:38 PM, Ran Tavory <rantav@gmail.com> wrote:
>> Due to administrative error one of the hosts in the cluster lost p= ermission
>> to write to it's data directory.
>> So I started seeing errors in the log, however, the server continu= ed serving
>> traffic. It wasn't able to compact and do other write operatio= ns but it
>> didn't crash.
>> I was wondering wether that's by design and if so, is this a g= ood one... I
>> guess I want to know if really bad things happen to my cluster...<= br> >> logs look like that...
>>
>> =C2=A0INFO [FLUSH-TIMER] 2010-08-11 07:53:14,683 ColumnFamilyStore= .java (line
>> 357) KvAds has reached its threshold; switching in a fresh Memtabl= e at
>> CommitLogContext(file=3D'/outbrain/cassandra/commitlog/Commi >> tLog-1281505164614.log', position=3D88521163)
>> =C2=A0INFO [FLUSH-TIMER] 2010-08-11 07:53:14,683 ColumnFamilyStore= .java (line
>> 609) Enqueuing flush of Memtable(KvAds)@851225759
>> =C2=A0INFO [FLUSH-WRITER-POOL:1] 2010-08-11 07:53:14,684 Memtable.= java (line 148)
>> Writing Memtable(KvAds)@851225759
>> ERROR [FLUSH-WRITER-POOL:1] 2010-08-11 07:53:14,688
>> DebuggableThreadPoolExecutor.java (line 94) Error in executor futu= retask
>> java.util.concurrent.ExecutionException: java.lang.RuntimeExceptio= n:
>> java.io.FileNotFoundException:
>> /outbrain/cassandra/data/outbrain_kvdb/KvAds-tmp-249-Data.db (Perm= ission
>> denied)
>> =C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at
>> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)=
>> =C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at java.util.concurrent.FutureTas= k.get(FutureTask.java:83)
>> =C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at
>> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.after= Execute(DebuggableThreadPoolExecutor.java:86)
>> =C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolE= xecutor.java:888)
>> =C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecu= tor.java:908)
>> =C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at java.lang.Thread.run(Thread.ja= va:619)
>> Caused by: java.lang.RuntimeException: java.io.FileNotFoundExcepti= on:
>> /outbrain/cassandra/data/outbrain_kvdb/KvAds-tmp-249-Data.db (Perm= ission
>> denied)
>> =C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at
>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.jav= a:34)
>> =C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java= :441)
>> =C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at
>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)=
>> =C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at java.util.concurrent.FutureTas= k.run(FutureTask.java:138)
>> =C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolE= xecutor.java:886)
>> ... more
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of Riptano, the source for professional Cassandra support > http://riptano.com
>

--0016e65bbaf4ead5af048e26d3b5--