cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mahesh Daksha <daks...@gmail.com>
Subject Re: [EXTERNAL] Issue while updating a record in 3 node cassandra cluster deployed using kubernetes
Date Tue, 09 Apr 2019 14:48:10 GMT
Thank you Sean for your response. We are also suspecting the same and
analyzing/troubleshooting it around queries associated timestamp.

Thanks,
Mahesh Daksha


On Tue, Apr 9, 2019 at 7:08 PM Durity, Sean R <SEAN_R_DURITY@homedepot.com>
wrote:

> My first suspicion would be to look at the server times in the cluster. It
> looks like other cases where a write occurs (with no errors) but the data
> is not retrieved as expected. If the write occurs with an earlier timestamp
> than the existing data, this is the behavior you would see. The write would
> occur, but it would not be the latest data to be retrieved. The write looks
> like it fails silently, but it actually does exactly what it is designed to
> do.
>
>
>
> Sean Durity
>
>
>
> *From:* Mahesh Daksha <dakshme@gmail.com>
> *Sent:* Tuesday, April 09, 2019 9:10 AM
> *To:* user@cassandra.apache.org
> *Subject:* [EXTERNAL] Issue while updating a record in 3 node cassandra
> cluster deployed using kubernetes
>
>
>
> Hello All,
>
> I have a 3 node cassandra cluster with Replication factor as 2 and
> read-write consistency set to QUORUM. We are using Spring data cassandra.
> All infrastructure is deployed using kubernetes.
>
> Now in normal use case many records gets inserted to cassandra table. Then
> we try to modify/update one of the record using save method of repo, like
> below:
>
> ChunkMeta *tmpRec* = chunkMetaRepository.*save*(chunkMeta);
>
> After execution of above statement we never see any exception or error.
> But still this update state goes silent/fail intermittently. That is at
> times the record in the db gets updated successfully where as other time it
> fails. Also in the above query when we print *tmpRec* it contains the
> updated and correct value every time. Still in the db these updated values
> doesn't get reflected.
>
> We check the the cassandra transport TRACE logs on all nodes and found the
> our queries are getting logged there and are being executed also with out
> any error or exception.
>
> Now another weird observation is this all thing works erfectly fine if I
> am using single cassandra node (in kubernetes) or if we deploy above infra
> using ansible (even works for 3 nodes for Ansible).
>
> It looks some issue is specifically with the kubernetes 3 node deployment
> of cassandra. Primarily looks like replication among nodes causing this.
>
> Please suggest.
>
>
>
>
> I have a 3 node cassandra cluster with Replication factor as 2 and
> read-write consistency set to QUORUM. We are using Spring data cassandra.
> All infrastructure is deployed using kubernetes.
>
> Now in normal use case many records gets inserted to cassandra table. Then
> we try to modify/update one of the record using save method of repo, like
> below:
>
> ChunkMeta tmpRec = chunkMetaRepository.*save*(chunkMeta);
>
> After execution of above statement we never see any exception or error.
> But still this update fail intermittently. That is when we check the record
> in the db sometime it gets updated successfully where as other time it
> fails. Also in the above query when we print *tmpRec* it contains the
> updated and correct value. Still in the db these updated values doesnt get
> reflected.
>
> We check the the cassandra transport TRACE logs on all nodes and found the
> our queries are getting logged there and are being executed also.
>
> Now another weird observation is this all thing works if I am using single
> cassandra node (in kubernetes) or if we deploy above infra using ansible
> (even works for 3 nodes for Ansible).
>
> It looks some issue is specifically with the kubernetes 3 node deployment
> of cassandra. Primarily looks like replication among nodes causing this.
>
> Please suggest.
>
> Below are the contents of  my cassandra Docker file:
>
> FROM ubuntu:16.04
>
>
>
> RUN apt-get update && apt-get install -y python sudo lsof vim dnsutils net-tools
&& apt-get clean && \
>
>     addgroup testuser && useradd -g testuser testuser && usermod --password
testuser testuser;
>
>
>
> RUN mkdir -p /opt/test && \
>
>     mkdir -p /opt/test/data;
>
>
>
> ADD jre8.tar.gz /opt/test/
>
> ADD apache-cassandra-3.11.0-bin.tar.gz /opt/test/
>
>
>
> RUN chmod 755 -R /opt/test/jre && \
>
>     ln -s /opt/test/jre/bin/java /usr/bin/java && \
>
>     mv /opt/test/apache-cassandra* /opt/test/cassandra;
>
>
>
> RUN mkdir -p /opt/test/cassandra/logs;
>
>
>
> ENV JAVA_HOME /opt/test/jre
>
> RUN export JAVA_HOME
>
>
>
> COPY version.txt /opt/test/cassandra/version.txt
>
>
>
> WORKDIR /opt/test/cassandra/bin/
>
>
>
> RUN mkdir -p /opt/test/data/saved_caches && \
>
>     mkdir -p /opt/test/data/commitlog && \
>
>     mkdir -p /opt/test/data/hints && \
>
>     chown -R testuser:testuser /opt/test/data && \
>
>     chown -R testuser:testuser /opt/test;
>
>
>
> USER testuser
>
>
>
> CMD cp /etc/cassandra/cassandra.yml ../conf/conf.yml && perl -p -e 's/\$\{([^}]+)\}/defined
$ENV{$1} ? $ENV{$1} : $&/eg; s/\$\{([^}]+)\}//eg' ../conf/conf.yml > ../conf/cassandra.yaml
&& rm ../conf/conf.yml && ./cassandra -f
>
> Please note conf.yml is basically cassandra.yml file having properties
> related to cassandra.
>
>
>
> Thanks,
>
> Mahesh Daksha
>
> ------------------------------
>
> The information in this Internet Email is confidential and may be legally
> privileged. It is intended solely for the addressee. Access to this Email
> by anyone else is unauthorized. If you are not the intended recipient, any
> disclosure, copying, distribution or any action taken or omitted to be
> taken in reliance on it, is prohibited and may be unlawful. When addressed
> to our clients any opinions or advice contained in this Email are subject
> to the terms and conditions expressed in any applicable governing The Home
> Depot terms of business or client engagement letter. The Home Depot
> disclaims all responsibility and liability for the accuracy and content of
> this attachment and for any damages or losses arising from any
> inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other
> items of a destructive nature, which may be contained in this attachment
> and shall not be liable for direct, indirect, consequential or special
> damages in connection with this e-mail message or its attachment.
>

Mime
View raw message