cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Aronchick <aronch...@gmail.com>
Subject Re: Issue when node goes away?
Date Wed, 01 Jul 2015 18:36:00 GMT
This helps - so let me understand:

Starting point:
 - 4 nodes running 2.1.4
 - System is healthy

Decide to upgrade:
 - Add 2 nodes running 2.1.5
 - Run nodetool upgradestable
 - Wait until system is healthy
 - Stop 2 nodes running 2.1.4
 - Run nodetool upgradestable
 - Add 2 nodes running 2.1.5
 - Run nodetool upgradestable
 - Stop 2 nodes running 2.1.4
 - Run nodetool upgradestable

Finished?

On Wed, Jul 1, 2015 at 1:49 AM, Alain RODRIGUEZ <arodrime@gmail.com> wrote:

> Basically, when you add nodes, add them on the correct version to avoid
> schema / network issues in your streams. Also, try to update all the node
> using rolling restarts in a reduced time frame after stopping repairs, with
> all the nodes up, etc.
>
> You must have a healthy cluster before performing an upgrade.
>
> Remember running a nodetool upgradesstable after any upgrade (if not
> needed it will end fast, so I would run it always as a best practice, just
> in case).
>
> C*heers,
>
> Alain
>
> 2015-07-01 2:16 GMT+02:00 David Aronchick <aronchick@gmail.com>:
>
>> That is a GREAT lead! So it looks like I can't add a few nodes to the
>> cluster of the new version, have it settle down, and then upgrade the rest?
>>
>> On Tue, Jun 30, 2015 at 11:58 AM, Alain RODRIGUEZ <arodrime@gmail.com>
>> wrote:
>>
>>> "Would it matter that I'm mixing cassandra versions? "
>>>
>>> From:
>>> http://docs.datastax.com/en/upgrade/doc/upgrade/datastax_enterprise/upgrdLim.html
>>>
>>> "General upgrade limitations¶
>>>
>>> Do not run nodetool repair.
>>> Do not enable new features.
>>> Do not issue these types of queries during a rolling restart: DDL,
>>> TRUNCATE
>>> *During upgrades, the nodes on different versions show a schema
>>> disagreement*."
>>>
>>> I think this is a good lead.
>>>
>>> C*heers,
>>>
>>> Alain
>>>
>>> 2015-06-30 20:22 GMT+02:00 David Aronchick <aronchick@gmail.com>:
>>>
>>>> I appreciate the thoughts! My issue is that it seems to work perfectly,
>>>> until the node goes away. Would it matter that I'm mixing cassandra
>>>> versions? (2.1.4 and 2.1.5)?
>>>>
>>>> On Tue, Jun 30, 2015 at 5:23 AM, Alain RODRIGUEZ <arodrime@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi David ?
>>>>>
>>>>> What does a "nodetool describecluster" output look like ?
>>>>>
>>>>> My guess is you might be having a schema version desynchronisation. If
>>>>> you see a node with different schema version you might want to try a
>>>>> "nodetool resetlocal*schema*       - Reset node's local *schema* and
>>>>> resync"
>>>>>
>>>>> You asked for any thoughts, this is a thought, not sure if it will
>>>>> help, I hope so.
>>>>>
>>>>> C*heers,
>>>>>
>>>>> Alain
>>>>>
>>>>> 2015-06-30 1:44 GMT+02:00 Robert Coli <rcoli@eventbrite.com>:
>>>>>
>>>>>> On Mon, Jun 29, 2015 at 2:43 PM, David Aronchick <aronchick@gmail.com
>>>>>> > wrote:
>>>>>>
>>>>>>> Ping--- any thoughts here?
>>>>>>>
>>>>>>
>>>>>> I don't have any thoughts on your specific issue at this time, but
>>>>>> FWIW #cassandra on freenode is sometimes a better forum for interactive
>>>>>> debugging of operational edge cases.
>>>>>>
>>>>>> =Rob
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message