cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stefan Miklosovic <stefan.mikloso...@instaclustr.com>
Subject Re: Issue while updating a record in 3 node cassandra cluster deployed using kubernetes
Date Wed, 10 Apr 2019 00:39:08 GMT
>> I have a 3 node cassandra cluster with Replication factor as 2 and read-write consistency
set to QUORUM.

I am not sure what you want to achieve with this. If you have three
nodes and RF 2, for each write there will be two replicas, right ...
If one of your replicas is down out of two in total, you will never
reach quorum as one node is down and one is up and that is not quorum
if half of your nodes is up. If one of your nodes fails and the record
is on that one and some other, your query fails too so your cluster is
not protected against any failed nodes.

On Tue, 9 Apr 2019 at 23:10, Mahesh Daksha <dakshme@gmail.com> wrote:
>
> Hello All,
>
> I have a 3 node cassandra cluster with Replication factor as 2 and read-write consistency
set to QUORUM. We are using Spring data cassandra. All infrastructure is deployed using kubernetes.
>
> Now in normal use case many records gets inserted to cassandra table. Then we try to
modify/update one of the record using save method of repo, like below:
>
> ChunkMeta tmpRec = chunkMetaRepository.save(chunkMeta);
>
> After execution of above statement we never see any exception or error. But still this
update state goes silent/fail intermittently. That is at times the record in the db gets updated
successfully where as other time it fails. Also in the above query when we print tmpRec it
contains the updated and correct value every time. Still in the db these updated values doesn't
get reflected.
>
> We check the the cassandra transport TRACE logs on all nodes and found the our queries
are getting logged there and are being executed also with out any error or exception.
>
> Now another weird observation is this all thing works erfectly fine if I am using single
cassandra node (in kubernetes) or if we deploy above infra using ansible (even works for 3
nodes for Ansible).
>
> It looks some issue is specifically with the kubernetes 3 node deployment of cassandra.
Primarily looks like replication among nodes causing this.
>
> Please suggest.
>
>
>
> I have a 3 node cassandra cluster with Replication factor as 2 and read-write consistency
set to QUORUM. We are using Spring data cassandra. All infrastructure is deployed using kubernetes.
>
> Now in normal use case many records gets inserted to cassandra table. Then we try to
modify/update one of the record using save method of repo, like below:
>
> ChunkMeta tmpRec = chunkMetaRepository.save(chunkMeta);
>
> After execution of above statement we never see any exception or error. But still this
update fail intermittently. That is when we check the record in the db sometime it gets updated
successfully where as other time it fails. Also in the above query when we print tmpRec it
contains the updated and correct value. Still in the db these updated values doesnt get reflected.
>
> We check the the cassandra transport TRACE logs on all nodes and found the our queries
are getting logged there and are being executed also.
>
> Now another weird observation is this all thing works if I am using single cassandra
node (in kubernetes) or if we deploy above infra using ansible (even works for 3 nodes for
Ansible).
>
> It looks some issue is specifically with the kubernetes 3 node deployment of cassandra.
Primarily looks like replication among nodes causing this.
>
> Please suggest.
>
> Below are the contents of  my cassandra Docker file:
>
> FROM ubuntu:16.04
>
> RUN apt-get update && apt-get install -y python sudo lsof vim dnsutils net-tools
&& apt-get clean && \
>     addgroup testuser && useradd -g testuser testuser && usermod --password
testuser testuser;
>
> RUN mkdir -p /opt/test && \
>     mkdir -p /opt/test/data;
>
> ADD jre8.tar.gz /opt/test/
> ADD apache-cassandra-3.11.0-bin.tar.gz /opt/test/
>
> RUN chmod 755 -R /opt/test/jre && \
>     ln -s /opt/test/jre/bin/java /usr/bin/java && \
>     mv /opt/test/apache-cassandra* /opt/test/cassandra;
>
> RUN mkdir -p /opt/test/cassandra/logs;
>
> ENV JAVA_HOME /opt/test/jre
> RUN export JAVA_HOME
>
> COPY version.txt /opt/test/cassandra/version.txt
>
> WORKDIR /opt/test/cassandra/bin/
>
> RUN mkdir -p /opt/test/data/saved_caches && \
>     mkdir -p /opt/test/data/commitlog && \
>     mkdir -p /opt/test/data/hints && \
>     chown -R testuser:testuser /opt/test/data && \
>     chown -R testuser:testuser /opt/test;
>
> USER testuser
>
> CMD cp /etc/cassandra/cassandra.yml ../conf/conf.yml && perl -p -e 's/\$\{([^}]+)\}/defined
$ENV{$1} ? $ENV{$1} : $&/eg; s/\$\{([^}]+)\}//eg' ../conf/conf.yml > ../conf/cassandra.yaml
&& rm ../conf/conf.yml && ./cassandra -f
>
> Please note conf.yml is basically cassandra.yml file having properties related to cassandra.
>
>
> Thanks,
>
> Mahesh Daksha

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org


Mime
View raw message