incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hiller, Dean" <Dean.Hil...@nrel.gov>
Subject Re: data clean up problem
Date Tue, 28 May 2013 20:23:57 GMT
Actually, we did a huge investigation into this on astyanax and cassandra.
 Astyanax if I remember worked if configured correctly but casasndra did
not so we patched cassandraŠfor some reason cassandra once on the
co-ordinator who had one copy fo the data would wait for both other nodes
to respond even though we are CL=QUOROM on RF=3Šwe put in patch for that
which my teammate is still supposed to submit.  Cassandra should only wait
for one nodeŠat least I think that is how I remember itŠ.We have it in our
backlog to get the patch into cassandra.

Previously one slow node would slow down our website but this no longer
happens to us such that when compaction kicks off on a single node, our
cluster keeps going strong.

Dean

On 5/28/13 2:12 PM, "Dwight Smith" <Dwight.Smith@genesyslab.com> wrote:

>How do you determine the slow node, client side response latency?
>
>-----Original Message-----
>From: Hiller, Dean [mailto:Dean.Hiller@nrel.gov]
>Sent: Tuesday, May 28, 2013 1:10 PM
>To: user@cassandra.apache.org
>Subject: Re: data clean up problem
>
>How much disk used on each node?  We run the suggested < 300G per node as
>above that compactions can have trouble keeping up.
>
>Ps. We run compactions during peak hours just fine because our client
>reroutes to the 2 of 3 nodes not running compactions based on seeing the
>slow node so performance stays fast.
>
>The easy route is to of course double your cluster and halve the data
>sizes per node so compaction can keep up.
>
>Dean
>
>From: cem <cayiroglu@gmail.com<mailto:cayiroglu@gmail.com>>
>Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>"
><user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
>Date: Tuesday, May 28, 2013 1:45 PM
>To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>"
><user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
>Subject: Re: data clean up problem
>
>Thanks for the answer.
>
>Sorry for the misunderstanding. I tried to say I don't send delete
>request from the client so it safe to set gc_grace to 0. TTL is used for
>data clean up. I am not running a manual compaction. I tried that ones
>but it took a lot of time finish and I will not have this amount of
>off-peek time in the production to run this. I even set the compaction
>throughput to unlimited and it didnt help that much.
>
>Disk size just keeps on growing but I know that there is enough space to
>store 1 day data.
>
>What do you think about time rage partitioning? Creating new column
>family for each partition and drop when you know that all records are
>expired.
>
>I have 5 nodes.
>
>Cem.
>
>
>
>
>On Tue, May 28, 2013 at 9:37 PM, Hiller, Dean
><Dean.Hiller@nrel.gov<mailto:Dean.Hiller@nrel.gov>> wrote:
>Also, how many nodes are you running?
>
>From: cem 
><cayiroglu@gmail.com<mailto:cayiroglu@gmail.com><mailto:cayiroglu@gmail.co
>m<mailto:cayiroglu@gmail.com>>>
>Reply-To: 
>"user@cassandra.apache.org<mailto:user@cassandra.apache.org><mailto:user@c
>assandra.apache.org<mailto:user@cassandra.apache.org>>"
><user@cassandra.apache.org<mailto:user@cassandra.apache.org><mailto:user@c
>assandra.apache.org<mailto:user@cassandra.apache.org>>>
>Date: Tuesday, May 28, 2013 1:17 PM
>To: 
>"user@cassandra.apache.org<mailto:user@cassandra.apache.org><mailto:user@c
>assandra.apache.org<mailto:user@cassandra.apache.org>>"
><user@cassandra.apache.org<mailto:user@cassandra.apache.org><mailto:user@c
>assandra.apache.org<mailto:user@cassandra.apache.org>>>
>Subject: Re: data clean up problem
>
>Thanks for the answer but it is already set to 0 since I don't do any
>delete.
>
>Cem
>
>
>On Tue, May 28, 2013 at 9:03 PM, Edward Capriolo
><edlinuxguru@gmail.com<mailto:edlinuxguru@gmail.com><mailto:edlinuxguru@gm
>ail.com<mailto:edlinuxguru@gmail.com>>> wrote:
>You need to change the gc_grace time of the column family. It defaults to
>10 days. By default the tombstones will not go away for 10 days.
>
>
>On Tue, May 28, 2013 at 2:46 PM, cem
><cayiroglu@gmail.com<mailto:cayiroglu@gmail.com><mailto:cayiroglu@gmail.co
>m<mailto:cayiroglu@gmail.com>>> wrote:
>Hi Experts,
>
>
>We have general problem about cleaning up data from the disk. I need to
>free the disk space after retention period and the customer wants to
>dimension the disk space base on that.
>
>After running multiple performance tests with TTL of 1 day we saw that
>the compaction couldn't keep up with the request rate. Disks were getting
>full after 3 days. There were also a lot of sstables that are older than
>1 day after 3 days.
>
>Things that we tried:
>
>-Change the compaction strategy to leveled. (helped a bit but not much)
>
>-Use big sstable size (10G) with leveled compaction to have more
>aggressive compaction.(helped a bit but not much)
>
>-Upgrade Cassandra from 1.0 to 1.2 to use TTL histograms (didn't help at
>all since it has key overlapping estimation algorithm that generates %100
>match. Although we don't have...)
>
>Our column family structure is like this:
>
>Event_data_cf: (we store event data. Event_id  is randomly generated and
>each event has attributes like location=london)
>
>row                  data
>
>event id          data blob
>
>timeseries_cf: (key is the attribute that we want to index. It can be
>location=london, we didnt use secondary indexes because the indexes are
>dynamic.)
>
>row                  data
>
>index key       time series of event id (event1_id, event2_id....)
>
>timeseries_inv_cf: (this is used for removing event by event row key. )
>
>row                  data
>
>event id          set of index keys
>
>Candidate Solution: Implementing time range partitions.
>
>Each partition will have column family set and will be managed by client.
>
>Suppose that you want to have 7 days retention period. Then you can
>configure the partition size as 1 day and have 7 active partitions at any
>time. Then you can drop inactive partitions (older that 7 days). Dropping
>will immediate remove the data from the disk. (With proper Cassandra.yaml
>configuration)
>
>Storing an event:
>
>Find the current partition p1
>
>store to event_data to Event_data_cf_p1
>
>store to indexes to timeseries_cff_p1
>
>store to inverted indexes to timeseries_inv_cf_p1
>
>
>A time range query with an index:
>
>Find the all partitions belongs to that time range
>
>Do read starting from the first partition until you reach to limit
>
>.....
>
>Could you please provide your comments and concerns ?
>
>Is there any other option that we can try?
>
>What do you think about the candidate solution?
>
>Does anyone have the same issue? How would you solve it in another way?
>
>
>Thanks in advance!
>
>Cem
>
>
>
>


Mime
View raw message