cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ken Hancock <ken.hanc...@schange.com>
Subject Re: automated CREATE TABLE just nuked my cluster after a 2.0 -> 2.1 upgrade....
Date Tue, 02 Feb 2016 16:14:23 GMT
Just to close the loop on this, but am I correct that the IF NOT EXITS
isn't the real problem?  Even multiple calls to CREATE TABLE cause the same
schema mismatch if done concurrently?  Normally, a CREATE TABLE call will
return an exception that the table already exists.

On Tue, Feb 2, 2016 at 11:06 AM, Jack Krupansky <jack.krupansky@gmail.com>
wrote:

> And CASSANDRA-10699  seems to be the sub-issue of CASSANDRA-9424 to do
> that:
> https://issues.apache.org/jira/browse/CASSANDRA-10699
>
>
> -- Jack Krupansky
>
> On Tue, Feb 2, 2016 at 9:59 AM, Sebastian Estevez <
> sebastian.estevez@datastax.com> wrote:
>
>> Hi Ken,
>>
>> Earlier in this thread I posted a link to
>> https://issues.apache.org/jira/browse/CASSANDRA-9424
>>
>> That is the fix for these schema disagreement issues and as ay commented,
>> the plan is to use CAS. Until then we have to treat schema delicately.
>>
>> all the best,
>>
>> Sebastián
>> On Feb 2, 2016 9:48 AM, "Ken Hancock" <ken.hancock@schange.com> wrote:
>>
>>> So this rings odd to me.  If you can accomplish the same thing by using
>>> a CAS operation, why not fix create table if not exist so that if your are
>>> writing an application that creates the table on startup, that the
>>> application is safe to run on multiple nodes and uses CAS to safeguard
>>> multiple concurrent creations?
>>>
>>>
>>> On Tue, Jan 26, 2016 at 12:32 PM, Eric Stevens <mightye@gmail.com>
>>> wrote:
>>>
>>>> There's still a race condition there, because two clients could SELECT
>>>> at the same time as each other, then both INSERT.
>>>>
>>>> You'd be better served with a CAS operation, and let Paxos guarantee
>>>> at-most-once execution.
>>>>
>>>> On Tue, Jan 26, 2016 at 9:06 AM Francisco Reyes <lists@natserv.net>
>>>> wrote:
>>>>
>>>>> On 01/22/2016 10:29 PM, Kevin Burton wrote:
>>>>>
>>>>> I sort of agree.. but we are also considering migrating to hourly
>>>>> tables.. and what if the single script doesn't run.
>>>>>
>>>>> I like having N nodes make changes like this because in my experience
>>>>> that central / single box will usually fail at the wrong time :-/
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Jan 22, 2016 at 6:47 PM, Jonathan Haddad <jon@jonhaddad.com>
>>>>> wrote:
>>>>>
>>>>>> Instead of using ZK, why not solve your concurrency problem by
>>>>>> removing it?  By that, I mean simply have 1 process that creates
all your
>>>>>> tables instead of creating a race condition intentionally?
>>>>>>
>>>>>> On Fri, Jan 22, 2016 at 6:16 PM Kevin Burton <burton@spinn3r.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Not sure if this is a bug or not or kind of a *fuzzy* area.
>>>>>>>
>>>>>>> In 2.0 this worked fine.
>>>>>>>
>>>>>>> We have a bunch of automated scripts that go through and create
>>>>>>> tables... one per day.
>>>>>>>
>>>>>>> at midnight UTC our entire CQL went offline.. .took down our
whole
>>>>>>> app.  ;-/
>>>>>>>
>>>>>>> The resolution was a full CQL shut down and then a drop table
to
>>>>>>> remove the bad tables...
>>>>>>>
>>>>>>> pretty sure the issue was with schema disagreement.
>>>>>>>
>>>>>>> All our CREATE TABLE use IF NOT EXISTS.... but I think the IF
NOT
>>>>>>> EXISTS only checks locally?
>>>>>>>
>>>>>>> My work around is going to be to use zookeeper to create a mutex
>>>>>>> lock during this operation.
>>>>>>>
>>>>>>> Any other things I should avoid?
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> We’re hiring if you know of any awesome Java Devops or Linux
>>>>>>> Operations Engineers!
>>>>>>>
>>>>>>> Founder/CEO Spinn3r.com
>>>>>>> Location: *San Francisco, CA*
>>>>>>> blog:  <http://burtonator.wordpress.com>
>>>>>>> http://burtonator.wordpress.com
>>>>>>> … or check out my Google+ profile
>>>>>>> <https://plus.google.com/102718274791889610666/posts>
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> We’re hiring if you know of any awesome Java Devops or Linux
>>>>> Operations Engineers!
>>>>>
>>>>> Founder/CEO Spinn3r.com
>>>>> Location: *San Francisco, CA*
>>>>> blog:  <http://burtonator.wordpress.com>
>>>>> http://burtonator.wordpress.com
>>>>> … or check out my Google+ profile
>>>>> <https://plus.google.com/102718274791889610666/posts>
>>>>>
>>>>>
>>>>> One way to accomplish both, a single process doing the work and having
>>>>> multiple machines be able to do it, is to have a control table.
>>>>>
>>>>> You can have a table that lists what tables have been created and
>>>>> force concistency all. In this table you list the names of tables created.
>>>>> If a table name is in there, it doesn't need to be created again.
>>>>>
>>>>
>>>
>>>
>>> --
>>> *Ken Hancock *| System Architect, Advanced Advertising
>>> SeaChange International
>>> 50 Nagog Park
>>> Acton, Massachusetts 01720
>>> ken.hancock@schange.com | www.schange.com | NASDAQ:SEAC
>>> <http://www.schange.com/en-US/Company/InvestorRelations.aspx>
>>> Office: +1 (978) 889-3329 | [image: Google Talk:]
>>> ken.hancock@schange.com | [image: Skype:]hancockks | [image: Yahoo IM:]
>>> hancockks [image: LinkedIn] <http://www.linkedin.com/in/kenhancock>
>>>
>>> [image: SeaChange International]
>>> <http://www.schange.com/>
>>> This e-mail and any attachments may contain information which is
>>> SeaChange International confidential. The information enclosed is intended
>>> only for the addressees herein and may not be copied or forwarded without
>>> permission from SeaChange International.
>>>
>>
>


-- 
*Ken Hancock *| System Architect, Advanced Advertising
SeaChange International
50 Nagog Park
Acton, Massachusetts 01720
ken.hancock@schange.com | www.schange.com | NASDAQ:SEAC
<http://www.schange.com/en-US/Company/InvestorRelations.aspx>
Office: +1 (978) 889-3329 | [image: Google Talk:]
ken.hancock@schange.com | [image:
Skype:]hancockks | [image: Yahoo IM:]hancockks [image: LinkedIn]
<http://www.linkedin.com/in/kenhancock>

[image: SeaChange International]
<http://www.schange.com/>
This e-mail and any attachments may contain information which is SeaChange
International confidential. The information enclosed is intended only for
the addressees herein and may not be copied or forwarded without permission
from SeaChange International.

Mime
View raw message