cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Haddad <...@jonhaddad.com>
Subject Re: Restarting nodes and reported load
Date Tue, 30 May 2017 22:28:11 GMT
You're the only one I see in the thread that's made any reference to HDFS.
The OP even noted that his question is about C*, not HDFS.

On Tue, May 30, 2017 at 2:59 PM daemeon reiydelle <daemeonr@gmail.com>
wrote:

> Did you notice that HDFS is the distributed file system used?
>
>
>
>
>
> *Daemeon C.M. ReiydelleUSA (+1) 415.501.0198 <(415)%20501-0198>London
> (+44) (0) 20 8144 9872 <+44%2020%208144%209872>*
>
>
> *“All men dream, but not equally. Those who dream by night in the dusty
> recesses of their minds wake up in the day to find it was vanity, but the
> dreamers of the day are dangerous men, for they may act their dreams with
> open eyes, to make it possible.” — T.E. Lawrence*
>
>
> On Tue, May 30, 2017 at 2:18 PM, Jonathan Haddad <jon@jonhaddad.com>
> wrote:
>
>> This isn't an HDFS mailing list.
>>
>> On Tue, May 30, 2017 at 2:14 PM daemeon reiydelle <daemeonr@gmail.com>
>> wrote:
>>
>>> no, 3tb is small. 30-50tb of hdfs space is typical these days per hdfs
>>> node. Depends somewhat on whether there is a mix of more and less
>>> frequently accessed data. But even storing only hot data, never saw
>>> anything less than 20tb hdfs per node.
>>>
>>>
>>>
>>>
>>>
>>> *Daemeon C.M. ReiydelleUSA (+1) 415.501.0198 <(415)%20501-0198>London
>>> (+44) (0) 20 8144 9872 <+44%2020%208144%209872>*
>>>
>>>
>>> *“All men dream, but not equally. Those who dream by night in the dusty
>>> recesses of their minds wake up in the day to find it was vanity, but the
>>> dreamers of the day are dangerous men, for they may act their dreams with
>>> open eyes, to make it possible.” — T.E. Lawrence*
>>>
>>>
>>> On Tue, May 30, 2017 at 2:00 PM, tommaso barbugli <tbarbugli@gmail.com>
>>> wrote:
>>>
>>>> Am I the only one thinking 3TB is way too much data for a single node
>>>> on a VM?
>>>>
>>>> On Tue, May 30, 2017 at 10:36 PM, Daniel Steuernol <
>>>> daniel@sendwithus.com> wrote:
>>>>
>>>>> I don't believe incremental repair is enabled, I have never enabled it
>>>>> on the cluster, and unless it's the default then it is off. Also I don't
>>>>> see a setting in cassandra.yaml for it.
>>>>>
>>>>>
>>>>>
>>>>> On May 30 2017, at 1:10 pm, daemeon reiydelle <daemeonr@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Unless there is a bug, snapshots are excluded (they are not HDFS
>>>>>> anyway!) from nodetool status.
>>>>>>
>>>>>> Out of curiousity, is incremenatal repair enabled? This is almost
>>>>>> certainly a rat hole, but there was an issue a few releases back
where load
>>>>>> would only increase until the node was restarted. Had been fixed
ages ago,
>>>>>> but wondering what happens if you restart a node, IF you have incremental
>>>>>> enabled.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> *Daemeon C.M. ReiydelleUSA (+1) 415.501.0198
>>>>>> <+1%20415-501-0198>London (+44) (0) 20 8144 9872 <+44%2020%208144%209872>*
>>>>>>
>>>>>>
>>>>>> *“All men dream, but not equally. Those who dream by night in the
>>>>>> dusty recesses of their minds wake up in the day to find it was vanity,
but
>>>>>> the dreamers of the day are dangerous men, for they may act their
dreams
>>>>>> with open eyes, to make it possible.” — T.E. Lawrence*
>>>>>>
>>>>>>
>>>>>> On Tue, May 30, 2017 at 12:15 PM, Varun Gupta <varung@uber.com>
>>>>>> wrote:
>>>>>>
>>>>>> Can you please check if you have incremental backup enabled and
>>>>>> snapshots are occupying the space.
>>>>>>
>>>>>> run nodetool clearsnapshot command.
>>>>>>
>>>>>> On Tue, May 30, 2017 at 11:12 AM, Daniel Steuernol <
>>>>>> daniel@sendwithus.com> wrote:
>>>>>>
>>>>>> It's 3-4TB per node, and by load rises, I'm talking about load as
>>>>>> reported by nodetool status.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On May 30 2017, at 10:25 am, daemeon reiydelle <daemeonr@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> When you say "the load rises ... ", could you clarify what you mean
>>>>>> by "load"? That has a specific Linux term, and in e.g. Cloudera Manager.
>>>>>> But in neither case would that be relevant to transient or persisted
disk.
>>>>>> Am I missing something?
>>>>>>
>>>>>>
>>>>>> On Tue, May 30, 2017 at 10:18 AM, tommaso barbugli <
>>>>>> tbarbugli@gmail.com> wrote:
>>>>>>
>>>>>> 3-4 TB per node or in total?
>>>>>>
>>>>>> On Tue, May 30, 2017 at 6:48 PM, Daniel Steuernol <
>>>>>> daniel@sendwithus.com> wrote:
>>>>>>
>>>>>> I should also mention that I am running cassandra 3.10 on the cluster
>>>>>>
>>>>>>
>>>>>>
>>>>>> On May 29 2017, at 9:43 am, Daniel Steuernol <daniel@sendwithus.com>
>>>>>> wrote:
>>>>>>
>>>>>> The cluster is running with RF=3, right now each node is storing
>>>>>> about 3-4 TB of data. I'm using r4.2xlarge EC2 instances, these have
8
>>>>>> vCPU's, 61 GB of RAM, and the disks attached for the data drive are
gp2 ssd
>>>>>> ebs volumes with 10k iops. I guess this brings up the question of
what's a
>>>>>> good marker to decide on whether to increase disk space vs provisioning
a
>>>>>> new node?
>>>>>>
>>>>>>
>>>>>> On May 29 2017, at 9:35 am, tommaso barbugli <tbarbugli@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> Hi Daniel,
>>>>>>
>>>>>> This is not normal. Possibly a capacity problem. Whats the RF, how
>>>>>> much data do you store per node and what kind of servers do you use
(core
>>>>>> count, RAM, disk, ...)?
>>>>>>
>>>>>> Cheers,
>>>>>> Tommaso
>>>>>>
>>>>>> On Mon, May 29, 2017 at 6:22 PM, Daniel Steuernol <
>>>>>> daniel@sendwithus.com> wrote:
>>>>>>
>>>>>>
>>>>>> I am running a 6 node cluster, and I have noticed that the reported
>>>>>> load on each node rises throughout the week and grows way past the
actual
>>>>>> disk space used and available on each node. Also eventually latency
for
>>>>>> operations suffers and the nodes have to be restarted. A couple questions
>>>>>> on this, is this normal? Also does cassandra need to be restarted
every few
>>>>>> days for best performance? Any insight on this behaviour would be
helpful.
>>>>>>
>>>>>> Cheers,
>>>>>> Daniel
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org For
>>>>>> additional commands, e-mail: user-help@cassandra.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org For
>>>>>> additional commands, e-mail: user-help@cassandra.apache.org
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>
>>>
>

Mime
View raw message