Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id BA1E776A5 for ; Fri, 22 Jul 2011 10:11:42 +0000 (UTC) Received: (qmail 38948 invoked by uid 500); 22 Jul 2011 10:11:40 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 38422 invoked by uid 500); 22 Jul 2011 10:11:30 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 38398 invoked by uid 99); 22 Jul 2011 10:11:28 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Jul 2011 10:11:28 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of jonathan.colby@gmail.com designates 74.125.82.70 as permitted sender) Received: from [74.125.82.70] (HELO mail-ww0-f70.google.com) (74.125.82.70) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 22 Jul 2011 10:11:21 +0000 Received: by wwg11 with SMTP id 11so734936wwg.1 for ; Fri, 22 Jul 2011 03:11:01 -0700 (PDT) MIME-Version: 1.0 Received: by 10.216.68.203 with SMTP id l53mr123423wed.0.1311329460809; Fri, 22 Jul 2011 03:11:00 -0700 (PDT) In-Reply-To: <56E2DE81-72B9-4193-9319-C627731E93CF@thelastpickle.com> Message-ID: <000e0ce0cc28b1f87c04a8a5aedb@google.com> Date: Fri, 22 Jul 2011 10:11:00 +0000 Subject: Re: Re: eliminate need to repair by using column TTL?? From: jonathan.colby@gmail.com To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=000e0ce0cc28b1f86304a8a5aed8 X-Virus-Checked: Checked by ClamAV on apache.org --000e0ce0cc28b1f86304a8a5aed8 Content-Type: text/plain; charset=ISO-8859-1; format=flowed; delsp=yes good points Aaron. I realize now how expensive repair on reads are. I'm going to keep doing repairs regularly but still have a max TTL on all columns to make sure we don't have really old data we no longer need getting buried in the cluster. On , aaron morton wrote: > Read repair will only repair data that is read on the nodes that are up > at that time, and does not guarantee that any changes it detects will be > written back to the nodes. The diff mutations are async fire and forget > messages which may go missing or be dropped or ignored by the recipient > just like any other message. > Also getting hit with a bunch of read repair operations is pretty > painful. The normal read runs, the coordinator detects the digest > mis-match, the read runs again from all nodes and they all have to return > their full data (no digests this time), the coordinator detects the > diffs, mutations are sent back to each node that needs them. All this > happens sync to the read request when the CL > ONE. Thats 2 reads with > more network IO and up to RF mutations . > The delete thing is important but repair also reduces the chance of reads > getting hit with RR and gives me confidence when it's necessary to nuke a > bad node. > Your plan may work but it feels risky to me. You may end up with worse > read performance and unpleasent emotions if you ever have to nuke a node. > Others may disagree. > Not ignoring the fact the repair can take a long time, fail, hurt > performance etc. There are plans to improve it though. > Cheers > ----------------- > Aaron Morton > Freelance Cassandra Developer > @aaronmorton > http://www.thelastpickle.com > On 22 Jul 2011, at 19:55, jonathan.colby@gmail.com wrote: > > One of the main reasons for regularly running repair is to make sure > deletes are propagated in the cluster, ie, data is not resurrected if a > node never received the delete call. > > > > And repair-on-read takes care of repairing inconsistencies "on-the-fly". > > > > So if I were to set a universal TTL on all columns - so everything > would only live for a certain age, would I be able to get away without > having to do regular repairs with nodetool? > > > > I realize this scenario would not be applicable for everyone, but our > data model would allow us to do this. > > > > So could this be an alternative to running the (resource-intensive, > long-running) repairs with nodetool? > > > > Thanks. --000e0ce0cc28b1f86304a8a5aed8 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable good points Aaron. I realize now how expensive repair on reads are. I= 9;m going to keep doing repairs regularly but still have a max TTL on all c= olumns to make sure we don't have really old data we no longer need get= ting buried in the cluster.

On , aaron morton <aaron@thelastp= ickle.com> wrote:
> Read repair will only repair data that is re= ad on the nodes that are up at that time, and does not guarantee that any c= hanges it detects will be written back to the nodes. The diff mutations are= async fire and forget messages which may go missing or be dropped or ignor= ed by the recipient just like any other message.
>
>
>
> Also getting hit with a bunch of read repair operations is= pretty painful. The normal read runs, the coordinator detects the digest m= is-match, the read runs again from all nodes and they all have to return th= eir full data (no digests this time), the coordinator detects the diffs, mu= tations are sent back to each node that needs them. All this happens sync t= o the read request when the CL > ONE. Thats 2 reads with more network IO= and up to RF mutations .
>
>
>
> The de= lete thing is important but repair also reduces the chance of reads getting= hit with RR and gives me confidence when it's necessary to nuke a bad = node.
>
>
>
> Your plan may work but it = feels risky to me. You may end up with worse read performance and unpleasen= t emotions if you ever have to nuke a node. Others may disagree.
> =
>
>
> Not ignoring the fact the repair can take = a long time, fail, hurt performance etc. There are plans to improve it thou= gh.
>
>
>
> Cheers
>
> =
>
> -----------------
>
> Aaron Morton>
> Freelance Cassandra Developer
>
> @aar= onmorton
>
> http://www.thelastpickle.com
>
= >
>
> On 22 Jul 2011, at 19:55, jonathan.colby@gmail.c= om wrote:
>
>
>
> > One of the main r= easons for regularly running repair is to make sure deletes are propagated = in the cluster, i.e., data is not resurrected if a node never received the = delete call.
>
> >
>
> > And repair= -on-read takes care of repairing inconsistencies "on-the-fly".>
> >
>
> > So if I were to set a uni= versal TTL on all columns - so everything would only live for a certain age= , would I be able to get away without having to do regular repairs with nod= etool?
>
> >
>
> > I realize this s= cenario would not be applicable for everyone, but our data model would allo= w us to do this.
>
> >
>
> > So cou= ld this be an alternative to running the (resource-intensive, long-running)= repairs with nodetool?
>
> >
>
> >= Thanks.
>
>
> --000e0ce0cc28b1f86304a8a5aed8--