incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Rothering <jrother...@codojo.me>
Subject Re: Bootstrap Timing
Date Fri, 25 Apr 2014 14:31:56 GMT
What version of C* is this?


On Fri, Apr 25, 2014 at 6:55 AM, Phil Burress <philburresseme@gmail.com>wrote:

> Just a follow-up on this for any interested parties. Ultimately we've
> determined that the bootstrap/join process is broken in Cassandra. We ended
> up creating an entirely new cluster and migrating the data.
>
>
> On Mon, Apr 21, 2014 at 10:32 AM, Phil Burress <philburresseme@gmail.com>wrote:
>
>> The new node has managed to stay up without dying for about 24 hours
>> now... but it still is in JOINING state. A new concern has popped up. Disk
>> usage is at 500GB on the new node. The three original nodes have about 40GB
>> each. Any ideas why this is happening?
>>
>>
>> On Sat, Apr 19, 2014 at 9:19 PM, Phil Burress <philburresseme@gmail.com>wrote:
>>
>>> Thank you all for your advice and good info. The node has died a couple
>>> of times with out of memory errors. I've restarted each time but it starts
>>> re - running compaction and then dies again.
>>>
>>> Is there a better way to do this?
>>> On Apr 18, 2014 6:06 PM, "Steven A Robenalt" <srobenal@stanford.edu>
>>> wrote:
>>>
>>>> That's what I'd be doing, but I wouldn't expect it to run for 3 days
>>>> this time. My guess is that whatever was going wrong with the bootstrap
>>>> when you had 3 nodes starting at once was interfering with the completion
>>>> of the 1 remaining node of those 3. A clean bootstrap of a single node
>>>> should complete eventually, and I would think it'll be a lot less than 3
>>>> days. Our database is much smaller than yours at the moment, so I can't
>>>> really guide you on how long it should take, but I'd think that others on
>>>> the list with similar database sizes might be able to give you a better
>>>> idea.
>>>>
>>>> Steve
>>>>
>>>>
>>>>
>>>> On Fri, Apr 18, 2014 at 1:43 PM, Phil Burress <philburresseme@gmail.com
>>>> > wrote:
>>>>
>>>>> First, I just stopped 2 of the nodes and left one running. But this
>>>>> morning, I stopped that third node, cleared out the data, restarted and
let
>>>>> it rejoin again. It appears streaming is done (according to netstats),
>>>>> right now it appears to be running compaction and building secondary
index
>>>>> (according to compactionstats). Just sit and wait I guess?
>>>>>
>>>>>
>>>>> On Fri, Apr 18, 2014 at 2:23 PM, Steven A Robenalt <
>>>>> srobenal@stanford.edu> wrote:
>>>>>
>>>>>> Looking back through this email chain, it looks like Phil said he
>>>>>> wasn't using vnodes.
>>>>>>
>>>>>> For the record, we are using vnodes since we brought up our first
>>>>>> cluster, and have not seen any issues with bootstrapping new nodes
either
>>>>>> to replace existing nodes, or to grow/shrink the cluster. We did
adhere to
>>>>>> the caveats that new nodes should not be seed nodes, and that we
should
>>>>>> allow each node to join the cluster completely before making any
other
>>>>>> changes.
>>>>>>
>>>>>> Phil, when you dropped to adding just the single node to your
>>>>>> cluster, did you start over with the newly added node (blowing away
the
>>>>>> database created on the previous startup), or did you shut down the
other 2
>>>>>> added nodes and leave the remaining one in progress to continue?
>>>>>>
>>>>>> Steve
>>>>>>
>>>>>>
>>>>>> On Fri, Apr 18, 2014 at 10:40 AM, Robert Coli <rcoli@eventbrite.com>wrote:
>>>>>>
>>>>>>> On Fri, Apr 18, 2014 at 5:05 AM, Phil Burress <
>>>>>>> philburresseme@gmail.com> wrote:
>>>>>>>
>>>>>>>> nodetool netstats shows 84 files. They are all at 100%. Nothing
>>>>>>>> showing in Pending or Active for Read Repair Stats.
>>>>>>>>
>>>>>>>> I'm assuming this means it's done. But it still shows "JOINING".
Is
>>>>>>>> there an undocumented step I'm missing here? This whole process
seems
>>>>>>>> broken to me.
>>>>>>>>
>>>>>>>
>>>>>>> Lately it seems like a lot more people than usual are :
>>>>>>>
>>>>>>> 1) using vnodes
>>>>>>> 2) unable to bootstrap new nodes
>>>>>>>
>>>>>>> If I were you, I would likely file a JIRA detailing your negative
>>>>>>> experience with this core functionality.
>>>>>>>
>>>>>>> =Rob
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Steve Robenalt
>>>>>> Software Architect
>>>>>>  HighWire | Stanford University
>>>>>> 425 Broadway St, Redwood City, CA 94063
>>>>>>
>>>>>> srobenal@stanford.edu
>>>>>> http://highwire.stanford.edu
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Steve Robenalt
>>>> Software Architect
>>>> HighWire | Stanford University
>>>> 425 Broadway St, Redwood City, CA 94063
>>>>
>>>> srobenal@stanford.edu
>>>> http://highwire.stanford.edu
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>
>

Mime
View raw message