incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Edward Capriolo <edlinuxg...@gmail.com>
Subject Re: Freeing up disk space on Cassandra 1.1.5 with Size-Tiered compaction.
Date Thu, 06 Dec 2012 15:38:58 GMT
http://wiki.apache.org/cassandra/LargeDataSetConsiderations


On Thu, Dec 6, 2012 at 9:53 AM, Poziombka, Wade L <
wade.l.poziombka@intel.com> wrote:

>  “Having so much data on each node is a potential bad day.”****
>
> ** **
>
> Is this discussed somewhere on the Cassandra documentation (limits,
> practices etc)?  We are also trying to load up quite a lot of data and have
> hit memory issues (bloom filter etc.) in 1.0.10.  I would like to read up
> on big data usage of Cassandra.  Meaning terabyte size databases.  ****
>
> ** **
>
> I do get your point about the amount of time required to recover downed
> node. But this 300-400MB business is interesting to me.****
>
> ** **
>
> Thanks in advance.****
>
> ** **
>
> Wade****
>
> ** **
>
> *From:* aaron morton [mailto:aaron@thelastpickle.com]
> *Sent:* Wednesday, December 05, 2012 9:23 PM
> *To:* user@cassandra.apache.org
> *Subject:* Re: Freeing up disk space on Cassandra 1.1.5 with Size-Tiered
> compaction.****
>
> ** **
>
> Basically we were successful on two of the nodes. They both took ~2 days
> and 11 hours to complete and at the end we saw one very large file ~900GB
> and the rest much smaller (the overall size decreased). This is what we
> expected!****
>
> I would recommend having up to 300MB to 400MB per node on a regular HDD
> with 1GB networking. ****
>
> ** **
>
> But on the 3rd node, we suspect major compaction didn't actually finish
> it's job…****
>
> The file list looks odd. Check the time stamps, on the files. You should
> not have files older than when compaction started. ****
>
> ** **
>
> 8GB heap ****
>
> The default is 4GB max now days. ****
>
> ** **
>
> 1) Do you expect problems with the 3rd node during 2 weeks more of
> operations, in the conditions seen below? ****
>
> I cannot answer that. ****
>
> ** **
>
> 2) Should we restart with leveled compaction next year? ****
>
> I would run some tests to see how it works for you workload. ****
>
> ** **
>
> 4) Should we consider increasing the cluster capacity?****
>
> IMHO yes.****
>
> You may also want to do some experiments with turing compression on if it
> not already enabled. ****
>
> ** **
>
> Having so much data on each node is a potential bad day. If instead you
> had to move or repair one of those nodes how long would it take for
> cassandra to stream all the data over ? (Or to rsync the data over.) How
> long does it take to run nodetool repair on the node ?****
>
> ** **
>
> With RF 3, if you lose a node you have lost your redundancy. It's
> important to have a plan about how to get it back and how long it may take.
>   ****
>
> ** **
>
> Hope that helps. ****
>
> ** **
>
> -----------------****
>
> Aaron Morton****
>
> Freelance Cassandra Developer****
>
> New Zealand****
>
> ** **
>
> @aaronmorton****
>
> http://www.thelastpickle.com****
>
> ** **
>
> On 6/12/2012, at 3:40 AM, Alexandru Sicoe <adsicoe@gmail.com> wrote:****
>
>
>
> ****
>
> Hi guys,
> Sorry for the late follow-up but I waited to run major compactions on all
> 3 nodes at a time before replying with my findings.
>
> Basically we were successful on two of the nodes. They both took ~2 days
> and 11 hours to complete and at the end we saw one very large file ~900GB
> and the rest much smaller (the overall size decreased). This is what we
> expected!
>
> But on the 3rd node, we suspect major compaction didn't actually finish
> it's job. First of all nodetool compact returned much earlier than the rest
> - after one day and 15 hrs. Secondly from the 1.4TBs initially on the node
> only about 36GB were freed up (almost the same size as before). Saw nothing
> in the server log (debug not enabled). Below I pasted some more details
> about file sizes before and after compaction on this third node and disk
> occupancy.
>
> The situation is maybe not so dramatic for us because in less than 2 weeks
> we will have a down time till after the new year. During this we can
> completely delete all the data in the cluster and start fresh with TTLs for
> 1 month (as suggested by Aaron and 8GB heap as suggested by Alain - thanks).
>
> Questions:
>
> 1) Do you expect problems with the 3rd node during 2 weeks more of
> operations, in the conditions seen below?
> [Note: we expect the minor compactions to continue building up files but
> never really getting to compacting the large file and thus not needing much
> temporarily extra disk space].
>
> 2) Should we restart with leveled compaction next year?
> [Note: Aaron was right, we have 1 week rows which get deleted after 1
> month which means older rows end up in big files => to free up space with
> SizeTiered we will have no choice but run major compactions which we don't
> know if they will work provided that we get at ~1TB / node / 1 month. You
> can see we are at the limit!]
>
> 3) In case we keep SizeTiered:
>
>     - How can we improve the performance of our major compactions? (we
> left all config parameters as default). Would increasing compactions
> throughput interfere with writes and reads? What about multi-threaded
> compactions?
>
>     - Do we still need to run regular repair operations as well? Do these
> also do a major compaction or are they completely separate operations?
>
> [Note: we have 3 nodes with RF=2 and inserting at consistency level 1 and
> reading at consistency level ALL. We read primarily for exporting reasons -
> we export 1 week worth of data at a time].
>
> 4) Should we consider increasing the cluster capacity?
> [We generate ~5million new rows every week which shouldn't come close to
> the hundreds of millions of rows on a node mentioned by Aaron which are the
> volumes that would create problems with bloom filters and indexes].
>
> Cheers,
> Alex
> ------------------
>
> The situation in the data folder
>
>     before calling nodetool comapact:
>
> du -csh /data_bst/cassandra/data/ATLAS/Data/*-Data.db
> 444G    /data_bst/cassandra/data/ATLAS/Data/ATLAS-Data-he-24370-Data.db
> 376G    /data_bst/cassandra/data/ATLAS/Data/ATLAS-Data-he-46431-Data.db
> 305G    /data_bst/cassandra/data/ATLAS/Data/ATLAS-Data-he-68959-Data.db
> 39G     /data_bst/cassandra/data/ATLAS/Data/ATLAS-Data-he-7352-Data.db
> 78G     /data_bst/cassandra/data/ATLAS/Data/ATLAS-Data-he-74076-Data.db
> 81G     /data_bst/cassandra/data/ATLAS/Data/ATLAS-Data-he-79663-Data.db
> 205M    /data_bst/cassandra/data/ATLAS/Data/ATLAS-Data-he-80370-Data.db
> 20G     /data_bst/cassandra/data/ATLAS/Data/ATLAS-Data-he-80968-Data.db
> 20G     /data_bst/cassandra/data/ATLAS/Data/ATLAS-Data-he-82330-Data.db
> 20G     /data_bst/cassandra/data/ATLAS/Data/ATLAS-Data-he-83710-Data.db
> 4.9G    /data_bst/cassandra/data/ATLAS/Data/ATLAS-Data-he-84015-Data.db
> 4.9G    /data_bst/cassandra/data/ATLAS/Data/ATLAS-Data-he-84356-Data.db
> 4.9G    /data_bst/cassandra/data/ATLAS/Data/ATLAS-Data-he-84696-Data.db
> 333M    /data_bst/cassandra/data/ATLAS/Data/ATLAS-Data-he-84707-Data.db
> 92M     /data_bst/cassandra/data/ATLAS/Data/ATLAS-Data-he-84712-Data.db
> 92M     /data_bst/cassandra/data/ATLAS/Data/ATLAS-Data-he-84717-Data.db
> 99M     /data_bst/cassandra/data/ATLAS/Data/ATLAS-Data-he-84722-Data.db
> 2.5G    /data_bst/cassandra/data/ATLAS/Data/ATLAS-Data-tmp-he-84723-Data.db
> 1.4T    total
>
>     after nodetool comapact returned:
>
> du -csh /data_bst/cassandra/data/ATLAS/Data/*-Data.db
> 444G    /data_bst/cassandra/data/ATLAS/Data/ATLAS-Data-he-24370-Data.db
> 910G    /data_bst/cassandra/data/ATLAS/Data/ATLAS-Data-he-84723-Data.db
> 19G     /data_bst/cassandra/data/ATLAS/Data/ATLAS-Data-he-86229-Data.db
> 19G     /data_bst/cassandra/data/ATLAS/Data/ATLAS-Data-he-87639-Data.db
> 5.0G    /data_bst/cassandra/data/ATLAS/Data/ATLAS-Data-he-87923-Data.db
> 4.8G    /data_bst/cassandra/data/ATLAS/Data/ATLAS-Data-he-88261-Data.db
> 338M    /data_bst/cassandra/data/ATLAS/Data/ATLAS-Data-he-88271-Data.db
> 339M    /data_bst/cassandra/data/ATLAS/Data/ATLAS-Data-he-88292-Data.db
> 339M    /data_bst/cassandra/data/ATLAS/Data/ATLAS-Data-he-88312-Data.db
> 98M
>
>
> Looking at the disk occupancy for the logical partition where the data
> folder is in:
>
> df /data_bst
> Filesystem           1K-blocks      Used Available Use% Mounted on
> /dev/sdb1            2927242720 1482502260 1444740460  51% /data_bst
>
>
> and the situation in the cluster
>
> nodetool -h $HOSTNAME ring (before major compaction)
> Address         DC          Rack        Status State   Load
> Effective-Ownership Token
>
> 113427455640312821154458202477256070484
> 10.146.44.17    datacenter1 rack1       Up     Normal  1.37 TB
> 66.67%              0
> 10.146.44.18    datacenter1 rack1       Up     Normal  1.04 TB
> 66.67%              56713727820156410577229101238628035242
> 10.146.44.32    datacenter1 rack1       Up     Normal  1.14 TB
> 66.67%              113427455640312821154458202477256070484
>
> nodetool -h $HOSTNAME ring (after major compaction) (Note we were
> inserting data in the meantime)
> Address         DC          Rack        Status State   Load
> Effective-Ownership Token
>
> 113427455640312821154458202477256070484
> 10.146.44.17    datacenter1 rack1       Up     Normal  1.38 TB
> 66.67%              0
> 10.146.44.18    datacenter1 rack1       Up     Normal  1.08 TB
> 66.67%              56713727820156410577229101238628035242
> 10.146.44.32    datacenter1 rack1       Up     Normal  1.19 TB
> 66.67%              113427455640312821154458202477256070484
>
> ****
>
> ** **
>
> On Fri, Nov 23, 2012 at 2:16 AM, aaron morton <aaron@thelastpickle.com>
> wrote:****
>
> >  From what I know having too much data on one node is bad, not really
> sure why, but  I think that performance will go down due to the size of
> indexes and bloom filters (I may be wrong on the reasons but I'm quite sure
> you can't store too much data per node).****
>
> If you have many hundreds of millions of rows on a node the memory needed
> for bloom filters and index sampling can be significant. These can both be
> tuned.
>
> If you have 1.1T per node the time to do a compaction, repair or upgrade
> may be very significant. Also the time taken to copy this data should you
> need to remove or replace a node may be prohibitive.****
>
>
> > 2. Switch to Leveled compaction strategy.****
>
> I would avoid making a change like that on an unstable / at risk system.
>
> > - Our usage pattern is write once, read once (export) and delete once!
>
>  The column TTL may be of use to you, it removes the need to do a delete.
>
> > - We were thinking of relying on the automatic minor compactions to free
> up space for us but as..
> There are some usage patterns which make life harder for STS. For example
> if you have very long lived rows that are written to and deleted a lot. Row
> fragments that have been around for a while will end up in bigger files,
> and these files get compacted less often.
>
> In this situation, if you are running low on disk space and you think
> there is a lot of deleted data in there, I would run a major compaction. A
> word or warning though, if do this you will need to continue to do it
> regularly. Major compaction creates a single big file, that will not get
> compaction often. There are ways to resolve this, and moving to LDB may
> help in the future.
>
> If you are stuck and worried about disk space it's what I would do. Once
> you are stable again then look at LDB
> http://www.datastax.com/dev/blog/when-to-use-leveled-compaction
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Cassandra Developer
> New Zealand
>
> @aaronmorton
> http://www.thelastpickle.com****
>
>
> On 23/11/2012, at 9:18 AM, Alain RODRIGUEZ <arodrime@gmail.com> wrote:
>
> > Hi Alexandru,
> >
> > "We are running a 3 node Cassandra 1.1.5 cluster with a 3TB Raid 0 disk
> per node for the data dir and separate disk for the commitlog, 12 cores, 24
> GB RAM"
> >
> > I think you should tune your architecture in a very different way. From
> what I know having too much data on one node is bad, not really sure why,
> but  I think that performance will go down due to the size of indexes and
> bloom filters (I may be wrong on the reasons but I'm quite sure you can't
> store too much data per node).
> >
> > Anyway, I am 6 nodes with half of these resources (6 cores / 12GB) would
> be better if you have the choice.
> >
> > "(12GB to Cassandra heap)."
> >
> > The max heap recommanded is 8GB because if you use more than these 8GB
> the Gc jobs will start decreasing your performance.
> >
> > "We now have 1.1 TB worth of data per node (RF = 2)."
> >
> > You should use RF=3 unless one out of consistency or SPOF  doesn't
> matter to you.
> >
> > With RF=2 you are obliged to write at CL.one to remove the single point
> of failure.
> >
> > "1. Start issuing regular major compactions (nodetool compact).
> >      - This is not recommended:
> >             - Stops minor compactions.
> >             - Major performance hit on node (very bad for us because
> need to be taking data all the time)."
> >
> > Actually, major compaction *does not* stop minor compactions. What
> happens is that due to the size of the size of the sstable that remains
> after your major compaction, it will never be compacted with the upcoming
> new sstables, and because of that, your read performance will go down until
> you run an other major compaction.
> >
> > "2. Switch to Leveled compaction strategy.
> >       - It is mentioned to help with deletes and disk space usage. Can
> someone confirm?"
> >
> > From what I know, Leveled compaction will not free disk space. It will
> allow you to use a greater percentage of your total disk space (50% max for
> sized tier compaction vs about 80% for leveled compaction)
> >
> > "Our usage pattern is write once, read once (export) and delete once! "
> >
> > In this case, I think that leveled compaction fits your needs.
> >
> > "Can anyone suggest which (if any) is better? Are there better
> solutions?"
> >
> > Are your sstable compressed ? You have 2 types of built-in compression
> and you may use them depending on the model of each of your CF.
> >
> > see:
> http://www.datastax.com/docs/1.1/operations/tuning#configure-compression
> >
> > Alain
> >
> > 2012/11/22 Alexandru Sicoe <adsicoe@gmail.com>
> > We are running a 3 node Cassandra 1.1.5 cluster with a 3TB Raid 0 disk
> per node for the data dir and separate disk for the commitlog, 12 cores, 24
> GB RAM (12GB to Cassandra heap).
> >****
>
> ** **
>
> ** **
>

Mime
View raw message