cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alain RODRIGUEZ <arodr...@gmail.com>
Subject Re: Advice in upgrade plan from 1.2.18 to 2.2.8
Date Thu, 22 Dec 2016 13:00:40 GMT
Hi,

Here are some thoughts:

running 1.2.18. I plan to upgrade them to 2.2.latest
>

Going 1 major release at the time is probably the safest way to go indeed.


>    1. Install 2.0.latest on one node at a time, start and wait for it to
>    join the ring.
>    2. Run upgradesstables on this node.
>    3. Repeat Step 1,2 on each node installing cassandra2.0 in a rolling
>    manner and running upgradesstables in parallel. (Please let me know if
>    running upgrades stables in parallel is not right here. My cluster is not
>    under much load really)
>
>
I would:

- Upgrade one node, check for cluster health (monitoring, logs, nodeool
commands), having a special attention to the 2.0 node.
- If everything is ok, then go for more nodes, if using distinct racks I
would go per rack; sequentially, node per node, all the nodes from
DC1-rack1, then DC1-rack2, then DC1-rack3. Then move to the next DC if
everything is fine.
- Start the 'upgradesstables' when the cluster is completely and
successfully running with the new version (2.0.17). It is perfectly fine to
run this in parallel as the last part of the upgrade. As you guessed, it is
good to keep monitoring the cluster load.

4. Now I will have both my DCs running 2.0.latest.


Without really having any strong argument, I would let it run for "some
time" like this, hours at least, maybe days. In any case, you will probably
have some work to prepare before the next upgrade, so you will have time to
check how the cluster is doing.

6. Do I need to run upgradesstables here again after the node has started
> and joined? (I think yes, but seek advice. https://docs.datastax.
> com/en/latest-upgrade/upgrade/cassandra/upgrdCassandra.html)


Yes, every time you run a major upgrade. Anyway, nodetool upgradesstables
will skip any sstable that do not need to be upgraded (as long as you don't
add the option to force it), so it is probably better to run it when you
have a doubt.


As additional information, I would prepare, for each upgrade:


   - The new Cassandra configuration (cassandra.yaml and cassandra-sh.yaml
   mainly but also other configuration files)

   To do that, I use to merge the current file in use (your configuration
   on C* 1.2.18) and the Cassandra version file from github for the new
   version (i.e.
   https://github.com/apache/cassandra/tree/cassandra-2.0.17/conf).

   This allows you to
      - Acknowledge and consider the new and removed configuration settings
      - Keep comments and default values in the configuration files up to
      date
      - Be fully exhaustive, and learn as you parse the files

      - Make sure clients will still work with the new version (see the
   doc, do the tests)
   - Cassandra metrics changed in the latest versions, you might have to
   rework your dashboards. Anticipating the dashboard creation for new
   versions would prevent you from loosing metrics when you need them the most.


Finally keep in mind that you should not perform any streaming while
running multiple version and as long as 'nodetool upgradesstables' is not
completely done. Meaning you should not add, remove, replace, move or
repair a node. Also, I would limit schema changes as much as possible while
running multiple versions as it caused troubles in the past.

During an upgrade, almost nothing else than the normal load due to the
service and the upgrade itself should happen. We always try to keep this
time window as short as possible.

C*heers,
-----------------------
Alain Rodriguez - @arodream - alain@thelastpickle.com
France

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com

2016-12-21 20:36 GMT+01:00 Aiman Parvaiz <aiman@steelhouse.com>:

> Hi everyone,
> I have 2 C* DCs with 12 nodes in each running 1.2.18. I plan to upgrade
> them to 2.2.latest and wanted to run by you experts my plan.
>
>
>    1. Install 2.0.latest on one node at a time, start and wait for it to
>    join the ring.
>    2. Run upgradesstables on this node.
>    3. Repeat Step 1,2 on each node installing cassandra2.0 in a rolling
>    manner and running upgradesstables in parallel. (Please let me know if
>    running upgrades stables in parallel is not right here. My cluster is not
>    under much load really)
>    4. Now I will have both my DCs running 2.0.latest.
>    5. Install cassandra 2.1.latest on one node at a time (same as above)
>    6. Do I need to run upgradesstables here again after the node has
>    started and joined? (I think yes, but seek advice.
>    https://docs.datastax.com/en/latest-upgrade/upgrade/
>    cassandra/upgrdCassandra.html
>    <https://docs.datastax.com/en/latest-upgrade/upgrade/cassandra/upgrdCassandra.html>
>    )
>    7. Following the above pattern, I would install cassandra2.1 in a
>    rolling manner across 2 DCs (depending on response to 6 I might or might
>    not run upgradesstables)
>    8. At this point both DCs would have 2.1.latest and again in rolling
>    manner I install 2.2.8.
>
>
> My assumption is that while this upgrade would be happening C* would still
> be able to serve reads and writes and running different versions at various
> point in the upgrade process will not affect the apps reading/writing from
> C*.
>
> Thanks
>
>

Mime
View raw message