Mailing-List: contact dev-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Content-Type: text/plain;
	charset=us-ascii
Mime-Version: 1.0 (1.0)
Subject: Re: Dropped Mutation and Read messages.
From: Oskar Kjellin <oskar.kjellin@gmail.com>
In-Reply-To: <4ED290F6-D936-4081-81BF-7BDE0DEAF8CF@internalcircle.com>
Date: Thu, 11 May 2017 19:57:10 +0200
Cc: varun saluja <saluja50@gmail.com>,
 "dev@cassandra.apache.org" <dev@cassandra.apache.org>
Content-Transfer-Encoding: quoted-printable
Message-Id: <A3DD1C3A-3871-46B0-ADCA-0D716DC542F6@gmail.com>
References: <pony-cbf26b79e85a638b66a4e458fab73015af03a2e9-84bfe5827bdc9d02eeb6ddb0f5f4ef5682bf7942@dev.cassandra.apache.org> <C165A7F1-3EC6-46EB-8509-2111594224D2@gmail.com> <CAGt4B3CgaBTZoJ8e9tmFsfm+xNf7QZkjtC3+VV5XB4uSmVeaKA@mail.gmail.com> <192AAD1E-1FB7-4964-82DC-176FCA0CDE70@gmail.com> <CAGt4B3D4L2vo7ucLVErPXGvd-tb8dB-Vr-xz9qF51kkR40bj_A@mail.gmail.com> <CAGt4B3BE_RVsiVxDYiFxzyxpYBRNJ2jqjjuZ7OtCeU4aeZwOYg@mail.gmail.com> <93FAEC36-35F0-4BE5-A89F-66C872AC7957@gmail.com> <4ED290F6-D936-4081-81BF-7BDE0DEAF8CF@internalcircle.com>
To: Michael Kjellman <mkjellman@internalcircle.com>
archived-at: Thu, 11 May 2017 17:57:18 -0000

Indeed, sorry. Subscribed to both so missed which one this was.=20

Sent from my iPhone

> On 11 May 2017, at 19:56, Michael Kjellman <mkjellman@internalcircle.com> w=
rote:
>=20
> This discussion should be on the C* user mailing list. Thanks!
>=20
> best,
> kjellman
>=20
>> On May 11, 2017, at 10:53 AM, Oskar Kjellin <oskar.kjellin@gmail.com> wro=
te:
>>=20
>> That seems way too low. Depending on what type of disk you have it should=
 be closer to 1-200MB.
>> That's probably causing your problems. It would still take a while for yo=
u to compact all your data tho=20
>>=20
>> Sent from my iPhone
>>=20
>>> On 11 May 2017, at 19:50, varun saluja <saluja50@gmail.com> wrote:
>>>=20
>>> nodetool getcompactionthrougput
>>>=20
>>> ./nodetool getcompactionthroughput
>>> Current compaction throughput: 16 MB/s
>>>=20
>>> Regards,
>>> Varun Saluja
>>>=20
>>>> On 11 May 2017 at 23:18, varun saluja <saluja50@gmail.com> wrote:
>>>> Hi,
>>>>=20
>>>> PFB results for same. Numbers are scary here.
>>>>=20
>>>> [root@WA-CASSDB2 bin]# ./nodetool compactionstats
>>>> pending tasks: 137
>>>>  compaction type         keyspace                 table    completed   =
       total    unit   progress
>>>>       Compaction           system                 hints   5762711108   8=
37522028005   bytes      0.69%
>>>>       Compaction   walletkeyspace   user_txn_history_v2    101477894   =
  4722068388   bytes      2.15%
>>>>       Compaction   walletkeyspace   user_txn_history_v2   1511866634   7=
53221762663   bytes      0.20%
>>>>       Compaction   walletkeyspace   user_txn_history_v2   3664734135   =
 18605501268   bytes     19.70%
>>>> Active compaction remaining time :  26h32m28s
>>>>=20
>>>>=20
>>>>=20
>>>>> On 11 May 2017 at 23:15, Oskar Kjellin <oskar.kjellin@gmail.com> wrote=
:
>>>>> What does nodetool compactionstats show?
>>>>>=20
>>>>> I meant compaction throttling. nodetool getcompactionthrougput
>>>>>=20
>>>>>=20
>>>>>> On 11 May 2017, at 19:41, varun saluja <saluja50@gmail.com> wrote:
>>>>>>=20
>>>>>> Hi Oskar,
>>>>>>=20
>>>>>> Thanks for response.
>>>>>>=20
>>>>>> Yes, could see lot of threads for compaction. Actually we are loading=
 around 400GB data  per node on 3 node cassandra cluster.
>>>>>> Throttling was set to write around 7k TPS per node. Job ran fine for 2=
 days and then, we start getting Mutation drops  , longer GC and very high l=
oad on system.
>>>>>>=20
>>>>>> System log reports:
>>>>>> Enqueuing flush of compactions_in_progress: 1156 (0%) on-heap, 1132 (=
0%) off-heap
>>>>>>=20
>>>>>> The job was stopped 12 hours back. But, still these failures can be s=
een. Can you Please let me know how shall i proceed further. If possible, Pl=
ease suggest some parameters for high write intensive jobs.
>>>>>>=20
>>>>>>=20
>>>>>> Regards,
>>>>>> Varun Saluja
>>>>>>=20
>>>>>>=20
>>>>>>> On 11 May 2017 at 23:01, Oskar Kjellin <oskar.kjellin@gmail.com> wro=
te:
>>>>>>> Do you have a lot of compactions going on? It sounds like you might'=
ve built up a huge backlog. Is your throttling configured properly?
>>>>>>>=20
>>>>>>>> On 11 May 2017, at 18:50, varun saluja <saluja50@gmail.com> wrote:
>>>>>>>>=20
>>>>>>>> Hi Experts,
>>>>>>>>=20
>>>>>>>> Seeking your help on a production issue.  We were running high writ=
e intensive job on our 3 node cassandra cluster V 2.1.7.
>>>>>>>>=20
>>>>>>>> TPS on nodes were high. Job ran for more than 2 days and thereafter=
, loadavg on 1 of the node increased to very high number like loadavg : 29.
>>>>>>>>=20
>>>>>>>> System log reports:
>>>>>>>>=20
>>>>>>>> INFO  [ScheduledTasks:1] 2017-05-11 22:11:04,466 MessagingService.j=
ava:888 - 839 MUTATION messages dropped in last 5000ms
>>>>>>>> INFO  [ScheduledTasks:1] 2017-05-11 22:11:04,466 MessagingService.j=
ava:888 - 2 READ messages dropped in last 5000ms
>>>>>>>> INFO  [ScheduledTasks:1] 2017-05-11 22:11:04,466 MessagingService.j=
ava:888 - 1 REQUEST_RESPONSE messages dropped in last 5000ms
>>>>>>>>=20
>>>>>>>> The job was stopped due to heavy load. But sill after 12 hours , we=
 can see mutation drops messages and sudden increase on avgload
>>>>>>>>=20
>>>>>>>> Are these hintedhandoff mutations? Can we stop these.
>>>>>>>> Strangely this behaviour is seen only on 2 nodes. Node 1 does not s=
how any load or any such activity.
>>>>>>>>=20
>>>>>>>> Due to heavy load and GC , there are intermittent gossip failures a=
mong node. Can you someone Please help.
>>>>>>>>=20
>>>>>>>> PS: Load job was stopped on cluster. Everything ran fine for few ho=
urs and and Later issue started again like mutation messages drops.
>>>>>>>>=20
>>>>>>>> Thanks and Regards,
>>>>>>>> Varun Saluja
>>>>>>>>=20
>>>>>>>> -------------------------------------------------------------------=
--
>>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
>>>>>>>> For additional commands, e-mail: dev-help@cassandra.apache.org
>>>>>>>>=20
>>>>>>=20
>>>>=20
>>>=20
>=20

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
For additional commands, e-mail: dev-help@cassandra.apache.org