cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jacek Furmankiewicz (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-7088) Zero length BigInteger exception causing processing to freeze
Date Fri, 25 Apr 2014 17:01:18 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-7088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13981248#comment-13981248
] 

Jacek Furmankiewicz commented on CASSANDRA-7088:
------------------------------------------------

Hi Brandon, we have good news. I am not sure if you still want to treat it as an issue or
not.

The core underlying problem is that we had a mismatch between a composite column type in our
Astyanax code (long,long,String) 
vs how the the actual CF was created (CompositeType(LongType,LongType,IntegerType)).

Now one ever noticed because it worked just fine in 1.2 (how I am not sure....it should have
bombed on the first query....I suspect Astyanax did some raw binary serialization that somehow
hid this underlying problem).

However, once we moved to 2.0, Cassandra must have detected that something does not make sense
in the query vs the underlying data representation and hence the zero length BigInteger error.

So it is definitely an issue on our side.

The only potential issue on the Cassandra side is that this error caused the entire query
to freeze  until we got an operation timeout exception, which sort of hid the underlying issue
for a while.

I will leave  it up to you if you want to leave the issue open for further investigation on
your side or close it.

Thank you for your help along the way.

>  Zero length BigInteger exception causing processing to freeze
> --------------------------------------------------------------
>
>                 Key: CASSANDRA-7088
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-7088
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: Linux Mint 16
>            Reporter: Jacek Furmankiewicz
>         Attachments: cassandra.yaml
>
>
> We attempted to migrate our developers to Cassandra 2.0.7 from 1.2.
> Everything worked perfectly, but we have experienced a massive drop in developer velocity.
> We run integration tests with Cucumber BDD and 1000 BDDs went from 7 minutes (Cassandra
1.2) to 15 minutes (2.0.7),
> This is when we run Cassandra of the ramdisk (/dev/shm) to make it run faster on dev
boxes.
> When we tried pointed to actual drives  the difference was dramatic: the entire suite
took over 70 minutes (!) vs 15 in Cassandra 1.2.
> After investigation, we found that most of the time is spent in the truncation logic
between every scenario, where we truncate all the column families and start with a clean DB
for the next test case.
> This used to be super fast in 1.2, is now very slow in 2.0.
> It may not seem important, but upgrading to 2.0 has basically cut down developer velocity
by 100%, just by more than doubling the time it takes to run our BDD suite.
> We truncate the CFs using the Ruby driver:
>   $cassandra.column_families.each do |column_family|
>         name = column_family[0].to_s
>         $cassandra.truncate! name
>   end
> I am attaching our cassandra.yaml. Please note we already switched off auto_compaction
before truncate, just as we did in 1.2 for dev boxes, Made no difference.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message