cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Haddad <...@jonhaddad.com>
Subject Re: Batch size warnings
Date Wed, 07 Dec 2016 17:30:32 GMT
@Ed, what you just said reminded me a lot of RAMP transactions.  I did a
blog post on it here: http://rustyrazorblade.com/2015/11/ramp-made-easy/

I've been considering doing a follow up on how to do a Cassandra data model
enabling RAMP transactions, but that takes time, and I have almost zero of
that.

On Wed, Dec 7, 2016 at 9:16 AM Edward Capriolo <edlinuxguru@gmail.com>
wrote:

> I have been circling around a thought process over batches. Now that
> Cassandra has aggregating functions, it might be possible write a type of
> record that has an END_OF_BATCH type marker and the data can be suppressed
> from view until it was all there.
>
> IE you write something like a checksum record that an intelligent client
> can use to tell if the rest of the batch is complete.
>
> On Wed, Dec 7, 2016 at 11:58 AM, Voytek Jarnot <voytek.jarnot@gmail.com>
> wrote:
>
> Been about a month since I have up on it, but it was very much related to
> the stuff you're dealing with ... Basically Cassandra just stepping on its
> own.... errrrr, tripping over its own feet streaming MVs.
>
> On Dec 7, 2016 10:45 AM, "Benjamin Roth" <benjamin.roth@jaumo.com> wrote:
>
> I meant the mv thing
>
> Am 07.12.2016 17:27 schrieb "Voytek Jarnot" <voytek.jarnot@gmail.com>:
>
> Sure, about which part?
>
> default batch size warning is 5kb
> I've increased it to 30kb, and will need to increase to 40kb (8x default
> setting) to avoid WARN log messages about batch sizes.  I do realize it's
> just a WARNing, but may as well avoid those if I can configure it out.
> That said, having to increase it so substantially (and we're only dealing
> with 5 tables) is making me wonder if I'm not taking the correct approach
> in terms of using batches to guarantee atomicity.
>
> On Wed, Dec 7, 2016 at 10:13 AM, Benjamin Roth <benjamin.roth@jaumo.com>
> wrote:
>
> Could you please be more specific?
>
> Am 07.12.2016 17:10 schrieb "Voytek Jarnot" <voytek.jarnot@gmail.com>:
>
> Should've mentioned - running 3.9.  Also - please do not recommend MVs: I
> tried, they're broken, we punted.
>
> On Wed, Dec 7, 2016 at 10:06 AM, Voytek Jarnot <voytek.jarnot@gmail.com>
> wrote:
>
> The low default value for batch_size_warn_threshold_in_kb is making me
> wonder if I'm perhaps approaching the problem of atomicity in a non-ideal
> fashion.
>
> With one data set duplicated/denormalized into 5 tables to support
> queries, we use batches to ensure inserts make it to all or 0 tables.  This
> works fine, but I've had to bump the warn threshold and fail threshold
> substantially (8x higher for the warn threshold).  This - in turn - makes
> me wonder, with a default setting so low, if I'm not solving this problem
> in the canonical/standard way.
>
> Mostly just looking for confirmation that we're not unintentionally doing
> something weird...
>
>
>
>
>

Mime
View raw message