Return-Path: Delivered-To: apmail-incubator-cassandra-user-archive@minotaur.apache.org Received: (qmail 95302 invoked from network); 5 Oct 2009 23:30:24 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 5 Oct 2009 23:30:24 -0000 Received: (qmail 80096 invoked by uid 500); 5 Oct 2009 23:30:23 -0000 Delivered-To: apmail-incubator-cassandra-user-archive@incubator.apache.org Received: (qmail 80051 invoked by uid 500); 5 Oct 2009 23:30:23 -0000 Mailing-List: contact cassandra-user-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: cassandra-user@incubator.apache.org Delivered-To: mailing list cassandra-user@incubator.apache.org Received: (qmail 80042 invoked by uid 99); 5 Oct 2009 23:30:23 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Oct 2009 23:30:23 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of chris.were@gmail.com designates 209.85.211.182 as permitted sender) Received: from [209.85.211.182] (HELO mail-yw0-f182.google.com) (209.85.211.182) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 05 Oct 2009 23:30:13 +0000 Received: by ywh12 with SMTP id 12so3489511ywh.21 for ; Mon, 05 Oct 2009 16:28:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:reply-to:in-reply-to :references:from:date:message-id:subject:to:cc:content-type; bh=/G2apnQ5NPuGhGhgH0rRWrLPppMuSKxsa+ah08zFjmc=; b=wbucNfs1/iNxN7/r2hN45kPszhGWXSGHNUWIqdyh5oqBUeL6IvW4VfO4fKBJC/v1ws g1nkO6z5KJIOQIre8V4Er9Jwavoj0Jzl4dlx+TP8WZJ2pyVEAeqCenfu+TqUkow09hpF DWcKcXyQuiu940MOXsMRb0Vcbt+L5mJiqjCTk= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:reply-to:in-reply-to:references:from:date:message-id :subject:to:cc:content-type; b=aIfVmZ5MO6ncPqjsHyPpE0uT91PsG66sg/JOww7auhpR+5p4padnQNjrb6ye1Kqfr3 kU5PIk46PBB5qm4JL00zoNglCYbj6I6LkCFHOtKz+LYKt3sRmJixojFkgpMifZ3CTWDS lXxmvgs7Xe6CJgHMQsxZhjuafFgosVNgB54zM= MIME-Version: 1.0 Received: by 10.91.142.8 with SMTP id u8mr297827agn.102.1254785332318; Mon, 05 Oct 2009 16:28:52 -0700 (PDT) Reply-To: chris@chriswere.com In-Reply-To: References: <35bb42690910051532i58a84e0cr95b7d64a10b39bb@mail.gmail.com> From: Chris Were Date: Tue, 6 Oct 2009 09:58:32 +1030 Message-ID: <35bb42690910051628t3229d62csccac10af46b68e8b@mail.gmail.com> Subject: Re: Cassandra Fatal Exceptions To: Jonathan Ellis Cc: cassandra-user@incubator.apache.org Content-Type: multipart/alternative; boundary=0016e64603900092200475387a97 X-Virus-Checked: Checked by ClamAV on apache.org --0016e64603900092200475387a97 Content-Type: text/plain; charset=ISO-8859-1 Thanks -- I'll head back to 0.4.0 then :) Cheers. On Tue, Oct 6, 2009 at 9:11 AM, Jonathan Ellis wrote: > Dan beat you to it: http://issues.apache.org/jira/browse/CASSANDRA-471 > > On Mon, Oct 5, 2009 at 5:39 PM, Jonathan Ellis wrote: > > looks like you ran into a regression. that's the risk you run on trunk. > :) > > > > Could you open a ticket for this on > > https://issues.apache.org/jira/browse/CASSANDRA ? > > > > -Jonathan > > > > On Mon, Oct 5, 2009 at 5:32 PM, Chris Were wrote: > >> Hi, > >> I updated from trunk two days ago, but ever since I've been receiving a > >> series of Fatal Exceptions. My logs show that about an hour after > starting > >> Cassandra I receive a RejectedExecutionException and two Assertion > >> exceptions. From that point on, the log files just fill up with repeated > >> TimeoutExceptions (each one followed by two Assertion exceptions). I am > in > >> no way familar with Cassandra internals so am at a loss to determine > what's > >> happening here. > >> Here is an excerpt of the first 6 exceptions (the last three continue ad > >> infinitum): > >> ERROR [FLUSH-TIMER] 2009-10-05 05:04:14,465 CassandraDaemon.java (line > >> org.apache.cassandra.service.CassandraDaemon) Fatal exception in thread > >> Thread[FLUSH-TIMER,5,main] > >> java.util.concurrent.RejectedExecutionException > >> at > >> > java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1956) > >> at > >> > java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:816) > >> at > >> > java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1337) > >> at > >> > java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:106) > >> at > >> > org.apache.cassandra.db.ColumnFamilyStore.submitFlush(ColumnFamilyStore.java:926) > >> at > >> > org.apache.cassandra.db.ColumnFamilyStore.switchMemtable(ColumnFamilyStore.java:367) > >> at > >> > org.apache.cassandra.db.ColumnFamilyStore.forceFlush(ColumnFamilyStore.java:415) > >> at > >> > org.apache.cassandra.db.ColumnFamilyStore.forceFlushIfExpired(ColumnFamilyStore.java:407) > >> at org.apache.cassandra.db.Table$1.run(Table.java:529) > >> at java.util.TimerThread.mainLoop(Timer.java:534) > >> at java.util.TimerThread.run(Timer.java:484) > >> ERROR [ROW-MUTATION-STAGE:6] 2009-10-05 05:04:14,574 > >> DebuggableThreadPoolExecutor.java (line > >> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor) Error in > >> ThreadPoolExecutor > >> java.lang.AssertionError > >> at org.apache.cassandra.db.Memtable.put(Memtable.java:143) > >> at > >> > org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:442) > >> at org.apache.cassandra.db.Table.apply(Table.java:626) > >> at > org.apache.cassandra.db.RowMutation.apply(RowMutation.java:205) > >> at > >> > org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:61) > >> at > >> > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:39) > >> at > >> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > >> at > >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > >> at java.lang.Thread.run(Thread.java:636) > >> ERROR [ROW-MUTATION-STAGE:6] 2009-10-05 05:04:14,574 > CassandraDaemon.java > >> (line org.apache.cassandra.service.CassandraDaemon) Fatal exception > >> in thread Thread[ROW-MUTATION-STAGE:6,5,main] > >> java.lang.AssertionError > >> at org.apache.cassandra.db.Memtable.put(Memtable.java:143) > >> at > >> > org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:442) > >> at org.apache.cassandra.db.Table.apply(Table.java:626) > >> at > org.apache.cassandra.db.RowMutation.apply(RowMutation.java:205) > >> at > >> > org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:61) > >> at > >> > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:39) > >> at > >> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > >> at > >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > >> at java.lang.Thread.run(Thread.java:636) > >> ERROR [pool-1-thread-1] 2009-10-05 05:04:19,525 StorageProxy.java (line > >> org.apache.cassandra.service.StorageProxy) error writing key > >> ec9787399d444098819180163b16abc4 > >> java.util.concurrent.TimeoutException: Operation timed out - received > only 0 > >> responses from . > >> at > >> > org.apache.cassandra.service.QuorumResponseHandler.get(QuorumResponseHandler.java:92) > >> at > >> > org.apache.cassandra.service.StorageProxy.insertBlocking(StorageProxy.java:165) > >> at > >> > org.apache.cassandra.service.CassandraServer.doInsert(CassandraServer.java:468) > >> at > >> > org.apache.cassandra.service.CassandraServer.batch_insert(CassandraServer.java:448) > >> at > >> > org.apache.cassandra.service.Cassandra$Processor$batch_insert.process(Cassandra.java:854) > >> at > >> > org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:627) > >> at > >> > org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253) > >> at > >> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > >> at > >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > >> at java.lang.Thread.run(Thread.java:636) > >> ERROR [ROW-MUTATION-STAGE:2] 2009-10-05 05:04:19,784 > >> DebuggableThreadPoolExecutor.java (line > >> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor) Error in > >> ThreadPoolExecutor > >> java.lang.AssertionError > >> at org.apache.cassandra.db.Memtable.put(Memtable.java:143) > >> at > >> > org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:442) > >> at org.apache.cassandra.db.Table.apply(Table.java:626) > >> at > org.apache.cassandra.db.RowMutation.apply(RowMutation.java:205) > >> at > >> > org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:61) > >> at > >> > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:39) > >> at > >> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > >> at > >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > >> at java.lang.Thread.run(Thread.java:636) > >> ERROR [ROW-MUTATION-STAGE:2] 2009-10-05 05:04:19,784 > CassandraDaemon.java > >> (line org.apache.cassandra.service.CassandraDaemon) Fatal exception in > >> thread Thread[ROW-MUTATION-STAGE:2,5,main] > >> java.lang.AssertionError > >> at org.apache.cassandra.db.Memtable.put(Memtable.java:143) > >> at > >> > org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:442) > >> at org.apache.cassandra.db.Table.apply(Table.java:626) > >> at > org.apache.cassandra.db.RowMutation.apply(RowMutation.java:205) > >> at > >> > org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:61) > >> at > >> > org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:39) > >> at > >> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > >> at > >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > >> at java.lang.Thread.run(Thread.java:636) > >> Cheers, > >> Chris > > > --0016e64603900092200475387a97 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Thanks -- I'll head back to 0.4.0 then :)

Cheers.
On Tue, Oct 6, 2009 at 9:11 AM, Jonathan El= lis <jbellis@gmai= l.com> wrote:
Dan beat you to it: http://issues.apa= che.org/jira/browse/CASSANDRA-471

On Mon, Oct 5, 2009 at 5:39 PM, Jonathan Ellis <jbellis@gmail.com> wrote:
> looks like you ran into a regression. =A0that's the risk you run o= n trunk. :)
>
> Could you open a ticket for this on
> https://issues.apache.org/jira/browse/CASSANDRA ?
>
> -Jonathan
>
> On Mon, Oct 5, 2009 at 5:32 PM, Chris Were <chris.were@gmail.com> wrote:
>> Hi,
>> I updated from trunk two days ago, but ever since I've been re= ceiving a
>> series of Fatal Exceptions. My logs show that about an hour after = starting
>> Cassandra I receive a=A0RejectedExecutionException and two Asserti= on
>> exceptions. From that point on, the log files just fill up with re= peated
>> TimeoutExceptions (each one followed by two Assertion exceptions).= I am in
>> no way familar with Cassandra internals so am at a loss to determi= ne what's
>> happening here.
>> Here is an excerpt of the first 6 exceptions (the last three conti= nue ad
>> infinitum):
>> ERROR [FLUSH-TIMER] 2009-10-05 05:04:14,465 CassandraDaemon.java (= line
>> org.apache.cassandra.service.CassandraDaemon) Fatal exception in t= hread
>> Thread[FLUSH-TIMER,5,main]
>> java.util.concurrent.RejectedExecutionException
>> =A0=A0 =A0 =A0 =A0at
>> java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecut= ion(ThreadPoolExecutor.java:1956)
>> =A0=A0 =A0 =A0 =A0at
>> java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.= java:816)
>> =A0=A0 =A0 =A0 =A0at
>> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor= .java:1337)
>> =A0=A0 =A0 =A0 =A0at
>> java.util.concurrent.AbstractExecutorService.submit(AbstractExecut= orService.java:106)
>> =A0=A0 =A0 =A0 =A0at
>> org.apache.cassandra.db.ColumnFamilyStore.submitFlush(ColumnFamily= Store.java:926)
>> =A0=A0 =A0 =A0 =A0at
>> org.apache.cassandra.db.ColumnFamilyStore.switchMemtable(ColumnFam= ilyStore.java:367)
>> =A0=A0 =A0 =A0 =A0at
>> org.apache.cassandra.db.ColumnFamilyStore.forceFlush(ColumnFamilyS= tore.java:415)
>> =A0=A0 =A0 =A0 =A0at
>> org.apache.cassandra.db.ColumnFamilyStore.forceFlushIfExpired(Colu= mnFamilyStore.java:407)
>> =A0=A0 =A0 =A0 =A0at org.apache.cassandra.db.Table$1.run(Table.jav= a:529)
>> =A0=A0 =A0 =A0 =A0at java.util.TimerThread.mainLoop(Timer.java:534= )
>> =A0=A0 =A0 =A0 =A0at java.util.TimerThread.run(Timer.java:484)
>> ERROR [ROW-MUTATION-STAGE:6] 2009-10-05 05:04:14,574
>> DebuggableThreadPoolExecutor.java (line
>> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor) Erro= r in
>> ThreadPoolExecutor
>> java.lang.AssertionError
>> =A0=A0 =A0 =A0 =A0at org.apache.cassandra.db.Memtable.put(Memtable= .java:143)
>> =A0=A0 =A0 =A0 =A0at
>> org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.= java:442)
>> =A0=A0 =A0 =A0 =A0at org.apache.cassandra.db.Table.apply(Table.jav= a:626)
>> =A0=A0 =A0 =A0 =A0at org.apache.cassandra.db.RowMutation.apply(Row= Mutation.java:205)
>> =A0=A0 =A0 =A0 =A0at
>> org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationV= erbHandler.java:61)
>> =A0=A0 =A0 =A0 =A0at
>> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTa= sk.java:39)
>> =A0=A0 =A0 =A0 =A0at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecut= or.java:1110)
>> =A0=A0 =A0 =A0 =A0at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecu= tor.java:603)
>> =A0=A0 =A0 =A0 =A0at java.lang.Thread.run(Thread.java:636)
>> ERROR [ROW-MUTATION-STAGE:6] 2009-10-05 05:04:14,574 CassandraDaem= on.java
>> (line org.apache.cassandra.service.CassandraDaemon) Fatal exceptio= n
>> =A0in thread Thread[ROW-MUTATION-STAGE:6,5,main]
>> java.lang.AssertionError
>> =A0=A0 =A0 =A0 =A0at org.apache.cassandra.db.Memtable.put(Memtable= .java:143)
>> =A0=A0 =A0 =A0 =A0at
>> org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.= java:442)
>> =A0=A0 =A0 =A0 =A0at org.apache.cassandra.db.Table.apply(Table.jav= a:626)
>> =A0=A0 =A0 =A0 =A0at org.apache.cassandra.db.RowMutation.apply(Row= Mutation.java:205)
>> =A0=A0 =A0 =A0 =A0at
>> org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationV= erbHandler.java:61)
>> =A0=A0 =A0 =A0 =A0at
>> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTa= sk.java:39)
>> =A0=A0 =A0 =A0 =A0at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecut= or.java:1110)
>> =A0=A0 =A0 =A0 =A0at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecu= tor.java:603)
>> =A0=A0 =A0 =A0 =A0at java.lang.Thread.run(Thread.java:636)
>> ERROR [pool-1-thread-1] 2009-10-05 05:04:19,525 StorageProxy.java = (line
>> org.apache.cassandra.service.StorageProxy) error writing key
>> ec9787399d444098819180163b16abc4
>> java.util.concurrent.TimeoutException: Operation timed out - recei= ved only 0
>> responses from =A0.
>> =A0=A0 =A0 =A0 =A0at
>> org.apache.cassandra.service.QuorumResponseHandler.get(QuorumRespo= nseHandler.java:92)
>> =A0=A0 =A0 =A0 =A0at
>> org.apache.cassandra.service.StorageProxy.insertBlocking(StoragePr= oxy.java:165)
>> =A0=A0 =A0 =A0 =A0at
>> org.apache.cassandra.service.CassandraServer.doInsert(CassandraSer= ver.java:468)
>> =A0=A0 =A0 =A0 =A0at
>> org.apache.cassandra.service.CassandraServer.batch_insert(Cassandr= aServer.java:448)
>> =A0=A0 =A0 =A0 =A0at
>> org.apache.cassandra.service.Cassandra$Processor$batch_insert.proc= ess(Cassandra.java:854)
>> =A0=A0 =A0 =A0 =A0at
>> org.apache.cassandra.service.Cassandra$Processor.process(Cassandra= .java:627)
>> =A0=A0 =A0 =A0 =A0at
>> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThre= adPoolServer.java:253)
>> =A0=A0 =A0 =A0 =A0at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecut= or.java:1110)
>> =A0=A0 =A0 =A0 =A0at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecu= tor.java:603)
>> =A0=A0 =A0 =A0 =A0at java.lang.Thread.run(Thread.java:636)
>> ERROR [ROW-MUTATION-STAGE:2] 2009-10-05 05:04:19,784
>> DebuggableThreadPoolExecutor.java (line
>> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor) Erro= r in
>> ThreadPoolExecutor
>> java.lang.AssertionError
>> =A0=A0 =A0 =A0 =A0at org.apache.cassandra.db.Memtable.put(Memtable= .java:143)
>> =A0=A0 =A0 =A0 =A0at
>> org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.= java:442)
>> =A0=A0 =A0 =A0 =A0at org.apache.cassandra.db.Table.apply(Table.jav= a:626)
>> =A0=A0 =A0 =A0 =A0at org.apache.cassandra.db.RowMutation.apply(Row= Mutation.java:205)
>> =A0=A0 =A0 =A0 =A0at
>> org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationV= erbHandler.java:61)
>> =A0=A0 =A0 =A0 =A0at
>> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTa= sk.java:39)
>> =A0=A0 =A0 =A0 =A0at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecut= or.java:1110)
>> =A0=A0 =A0 =A0 =A0at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecu= tor.java:603)
>> =A0=A0 =A0 =A0 =A0at java.lang.Thread.run(Thread.java:636)
>> ERROR [ROW-MUTATION-STAGE:2] 2009-10-05 05:04:19,784 CassandraDaem= on.java
>> (line org.apache.cassandra.service.CassandraDaemon) Fatal exceptio= n in
>> thread Thread[ROW-MUTATION-STAGE:2,5,main]
>> java.lang.AssertionError
>> =A0=A0 =A0 =A0 =A0at org.apache.cassandra.db.Memtable.put(Memtable= .java:143)
>> =A0=A0 =A0 =A0 =A0at
>> org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.= java:442)
>> =A0=A0 =A0 =A0 =A0at org.apache.cassandra.db.Table.apply(Table.jav= a:626)
>> =A0=A0 =A0 =A0 =A0at org.apache.cassandra.db.RowMutation.apply(Row= Mutation.java:205)
>> =A0=A0 =A0 =A0 =A0at
>> org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationV= erbHandler.java:61)
>> =A0=A0 =A0 =A0 =A0at
>> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTa= sk.java:39)
>> =A0=A0 =A0 =A0 =A0at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecut= or.java:1110)
>> =A0=A0 =A0 =A0 =A0at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecu= tor.java:603)
>> =A0=A0 =A0 =A0 =A0at java.lang.Thread.run(Thread.java:636)
>> Cheers,
>> Chris
>

--0016e64603900092200475387a97--