cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alain RODRIGUEZ <arodr...@gmail.com>
Subject Re: cassandra scalability
Date Mon, 07 Sep 2015 16:27:07 GMT
Glad to hear that.

About the Proxy

Putting an HA proxy in front of Cassandra is an anti-pattern as you create
a single point of failure. You can multiply them, but why would you do that
anyway when most (all ?) of the modern Cassandra clients handle this for
you through TokenAware /  RoundRobin / LatencyAware strategies ?

About the tokens:

You can use vnodes (set "num_tokens" to 64 or 128 let's say, comment
"#initial_token") and let Cassandra handle node/tokens distribution
combined to a Murmur3Partitioner and you're all set. Not sure what you're
doing with IPs, still don't get it but it is either useless or even bad
from what I understand.

Finally (as you find out) you can't auto_bootstrap a seed node indeed.

Hope this help.

C*heers,

Alain


2015-09-07 18:11 GMT+02:00 ICHIBA Sara <ichi.sara@gmail.com>:

> I think I know where my problem is coming from. I took a look at the log
> of cassandra on each node and I saw something related to bootstrap. it says
> that the node is a seed so there will be no bootstraping. Actually I made a
> mistake. in the cassandra.yaml file each node have two ips as seeds. the ip
> of the machine itself and the ip of the real seed server. Once i removed
> the local IP the problem seems to be fixed.
>
> 2015-09-07 18:01 GMT+02:00 ICHIBA Sara <ichi.sara@gmail.com>:
>
>> Thank you all for your answers.
>>
>> @Alain:
>> Can you detail actions performed,
>> >>like how you load data
>> >>>i have a haproxy in front of my cassandra database, so i'm sure that
>> my application queries each time a different coordinator
>>
>> >>what scaleup / scaledown are and precise if you let it decommission
>> fully (streams finished, node removed from nodetool status)
>> >>> i'm using openstack platform to autoscale cassandra cluster.
>> Actually, in openstack, the combination of ceilometer + heat allow to users
>> to automate the deployment of their applications and supervise their
>> resources. they can order the scale up (adding of new nodes automatically)
>> when resources(cpu, ram,...) are needed or scaledown (remove unecessary VMs
>> automatically).
>> so with heat i can spawn automatically a cluster of 2 cassandra VMs
>> (create the cluster and configure each cassandra server with a template).
>> My cluster can go from 2 nodes to 6 based on the workloads. when their is a
>> scaledown action, heat automatically execute a script on my node and
>> decommission it before removing it.
>>
>> >>Also, I am not sure of the meaning of this --> " i'm affecting to each
>> of my node a different token based on there ip address (the token is
>> A+B+C+D and the ip is A.B.C.D)".
>>
>> look at this:
>> [root@demo-server-seed-wgevseugyjd7 ~]# nodetool status bng;
>> Datacenter: DC1
>> ===============
>> Status=Up/Down
>> |/ State=Normal/Leaving/Joining/Moving
>> --  Address     Load       Tokens  Owns (effective)  Host
>> ID                               Rack
>> UN  40.0.0.149  789.03 KB  189     100.0%
>> bd0b2616-18d9-4bc2-a80b-eebd67474712  RAC1
>> UN  40.0.0.168  300.38 KB  208     100.0%
>> ebd9732b-ebfc-4a6c-b354-d7df860b57b0  RAC1
>>
>> the node with address 40.0.0.149 have the token 189=40+0+0+149
>> and the node with address 40.0.0.168 have the token 208=40+0+0+168
>>
>> this way i'm sure that each node in my cluster will have a different
>> token. I don't know what will happen if all the node have the same token??
>>
>> >>Aren't you using RandomPartitioner or Murmur3Partitioner
>>
>> i'm using the default one which is
>> partitioner: org.apache.cassandra.dht.Murmur3Partitioner
>>
>>
>> in order to configure cassandra on each node i'm using this script
>>
>>       inputs:
>>       - name: IP
>>       - name: SEED
>>       config: |
>>         #!/bin/bash -v
>>         cat << EOF >> /etc/resolv.conf
>>         nameserver 8.8.8.8
>>         nameserver 192.168.5.1
>>         EOF
>>
>>         DEFAULT=${DEFAULT:-/etc/cassandra/default.conf}
>>         CONFIG=/etc/cassandra/conf
>>         IFS="." read a b c d <<< $IP
>>         s="$[a[0]+b[0]+c[0]+d[0]]"
>>         sed -i -e "s/^cluster_name.*/cluster_name: 'Cassandra cluster for
>> freeradius'/" $CONFIG/cassandra.yaml
>>         sed -i -e "s/^num_tokens.*/num_tokens: $s/" $CONFIG/cassandra.yaml
>>         sed -i -e "s/^listen_address.*/listen_address: $IP/"
>> $CONFIG/cassandra.yaml
>>         sed -i -e "s/^rpc_address.*/rpc_address: 0.0.0.0/"
>> $CONFIG/cassandra.yaml
>>         sed -i -e "s/^broadcast_address.*/broadcast_address: $IP/"
>> $CONFIG/cassandra.yaml
>>         sed -i -e "s/broadcast_rpc_address.*/broadcast_rpc_address: $IP/"
>> $CONFIG/cassandra.yaml
>>         sed -i -e
>> "s/^commitlog_segment_size_in_mb.*/commitlog_segment_size_in_mb: 32/"
>> $CONFIG/cassandra.yaml
>>         sed -i -e "s/# JVM_OPTS=\"\$JVM_OPTS
>> -Djava.rmi.server.hostname=<public name>\"/JVM_OPTS=\"\$JVM_OPTS
>> -Djava.rmi.server.hostname=$IP\"/" $CONFIG/cassandra-env.sh
>>         sed -i -e "s/- seeds.*/- seeds: \"$SEED\"/" $CONFIG/cassandra.yaml
>>
>>         sed -i -e "s/^endpoint_snitch.*/endpoint_snitch:
>> GossipingPropertyFileSnitch/" $CONFIG/cassandra.yaml
>>         echo MAX_HEAP_SIZE="4G" >>  $CONFIG/cassandra-env.sh
>>         echo HEAP_NEWSIZE="800M" >> $CONFIG/cassandra-env.sh
>>         service cassandra stop
>>         rm -rf /var/lib/cassandra/data/system/*
>>         service cassandra start
>>
>>
>>
>> 2015-09-07 16:30 GMT+02:00 Ryan Svihla <rs@foundev.pro>:
>>
>>> If that's what tracing is telling you then it's fine and just a product
>>> of data distribution (note your token count isn't identical anyway).
>>>
>>> If you're doing cl one queries directly against particular nodes and
>>> getting different results it sounds like dropped mutations, streaming
>>> errors and or timeouts. Does running repair or reading at CL level all give
>>> you an accurate total record count?
>>>
>>> nodetool tpstats should help post bootstrap identify dropped mutations
>>> but you also want to monitor logs for any errors (in general this is always
>>> good advice for any system).. There could be a myriad or problems with
>>> bootstrapping new nodes, usually this is related to under provisioning.
>>>
>>> On Mon, Sep 7, 2015 at 8:19 AM Alain RODRIGUEZ <arodrime@gmail.com>
>>> wrote:
>>>
>>>> Hi Sara,
>>>>
>>>> Can you detail actions performed, like how you load data, what scaleup
>>>> / scaledown are and precise if you let it decommission fully (streams
>>>> finished, node removed from nodetool status) etc ?
>>>>
>>>> This would help us to help you :).
>>>>
>>>> Also, what happens if you query using "CONSISTENCY LOCAL_QUORUM;" (or
>>>> ALL) before your select ? If not using cqlsh, set the Consistency Level of
>>>> your client to LOCAL_QUORUM or ALL and try to select again.
>>>>
>>>> Also, I am not sure of the meaning of this --> " i'm affecting to each
>>>> of my node a different token based on there ip address (the token is
>>>> A+B+C+D and the ip is A.B.C.D)". Aren't you using RandomPartitioner or
>>>> Murmur3Partitioner ?
>>>>
>>>> C*heers,
>>>>
>>>> Alain
>>>>
>>>>
>>>>
>>>> 2015-09-07 12:01 GMT+02:00 Edouard COLE <Edouard.COLE@rgsystem.com>:
>>>>
>>>>> Please, don't mail me directly
>>>>>
>>>>> I read your answer, but I cannot help anymore
>>>>>
>>>>> And answering with "Sorry, I can't help" is pointless :)
>>>>>
>>>>> Wait for the community to answer
>>>>>
>>>>> De : ICHIBA Sara [mailto:ichi.sara@gmail.com]
>>>>> Envoyé : Monday, September 07, 2015 11:34 AM
>>>>> À : user@cassandra.apache.org
>>>>> Objet : Re: cassandra scalability
>>>>>
>>>>> when there's a scaledown action, i make sure to decommission the node
>>>>> before. but still, I don't understand why I'm having this behaviour.
is it
>>>>> normal. what do you do normally to remove a node? is it related to tokens?
>>>>> i'm affecting to each of my node a different token based on there ip
>>>>> address (the token is A+B+C+D and the ip is A.B.C.D)
>>>>>
>>>>> 2015-09-07 11:28 GMT+02:00 ICHIBA Sara <ichi.sara@gmail.com>:
>>>>> at the biginning it looks like this:
>>>>>
>>>>> [root@demo-server-seed-k6g62qr57nok ~]# nodetool status
>>>>> Datacenter: DC1
>>>>> ===============
>>>>> Status=Up/Down
>>>>> |/ State=Normal/Leaving/Joining/Moving
>>>>> --  Address     Load       Tokens  Owns    Host
>>>>> ID                               Rack
>>>>> UN  40.0.0.208  128.73 KB  248     ?
>>>>> 6e7788f9-56bf-4314-a23a-3bf1642d0606  RAC1
>>>>> UN  40.0.0.209  114.59 KB  249     ?
>>>>> 84f6f0be-6633-4c36-b341-b968ff91a58f  RAC1
>>>>> UN  40.0.0.205  129.53 KB  245     ?
>>>>> aa233dc2-a8ae-4c00-af74-0a119825237f  RAC1
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> [root@demo-server-seed-k6g62qr57nok ~]# nodetool status
>>>>> service_dictionary
>>>>> Datacenter: DC1
>>>>> ===============
>>>>> Status=Up/Down
>>>>> |/ State=Normal/Leaving/Joining/Moving
>>>>> --  Address     Load       Tokens  Owns (effective)  Host
>>>>> ID                               Rack
>>>>> UN  40.0.0.208  128.73 KB  248     68.8%
>>>>> 6e7788f9-56bf-4314-a23a-3bf1642d0606  RAC1
>>>>> UN  40.0.0.209  114.59 KB  249     67.8%
>>>>> 84f6f0be-6633-4c36-b341-b968ff91a58f  RAC1
>>>>> UN  40.0.0.205  129.53 KB  245     63.5%
>>>>> aa233dc2-a8ae-4c00-af74-0a119825237f  RAC1
>>>>>
>>>>> the result of the query select * from service_dictionary.table1; gave
>>>>> me
>>>>>  70 rows from 40.0.0.205
>>>>> 64 from 40.0.0.209
>>>>> 54 from 40.0.0.208
>>>>>
>>>>> 2015-09-07 11:13 GMT+02:00 Edouard COLE <Edouard.COLE@rgsystem.com>:
>>>>> Could you provide the result of :
>>>>> - nodetool status
>>>>> - nodetool status YOURKEYSPACE
>>>>>
>>>>>
>>>>>
>>>> --
>>> Regards,
>>>
>>> Ryan Svihla
>>
>>
>>
>

Mime
View raw message