From user-return-61166-archive-asf-public=cust-asf.ponee.io@cassandra.apache.org Thu May 24 23:12:31 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 81CDF180636 for ; Thu, 24 May 2018 23:12:30 +0200 (CEST) Received: (qmail 69024 invoked by uid 500); 24 May 2018 21:12:28 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 69014 invoked by uid 99); 24 May 2018 21:12:28 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 24 May 2018 21:12:28 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 3A7F4C049D for ; Thu, 24 May 2018 21:12:28 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.97 X-Spam-Level: * X-Spam-Status: No, score=1.97 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, T_DKIMWL_WL_MED=-0.01] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=aegisco-com.20150623.gappssmtp.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id v8xrlR-H8Ozc for ; Thu, 24 May 2018 21:12:26 +0000 (UTC) Received: from mail-wm0-f43.google.com (mail-wm0-f43.google.com [74.125.82.43]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 264835F3DF for ; Thu, 24 May 2018 21:12:26 +0000 (UTC) Received: by mail-wm0-f43.google.com with SMTP id 18-v6so3287092wml.2 for ; Thu, 24 May 2018 14:12:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=aegisco-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=UOHuuP/kxg1p+zCBIcKEZWhKOuhNNVi0QlFs7OkdjWY=; b=Gizd/Ag5dCSh4DAigscX2zQe5SiX331faNl48lKTheF7uKfgtygW1jTdfOLB2qzCSk YVg+22v84FI6Dnnp2vN1YqhCiGgx8iEdxtBVS11MnFH7RMJ4DFq4Af+kQwYZdSlmSAN6 9og4DIUCU7LrOEgsHVg1W6z7sir5uFv9qRlO1BZKbWawSbGxcmYF4mQjq09/PvxZLamp hpmVg8cgRASJmDp2IKypBKIfF9wZ+7Wjevl7tv9eGRJ/b0NedmSyrgKuwjxRkkLm6pdT TuIrUZAohqWbyK+Bmt3Vpdb1ckSbipNqJe2WxtGjQBsteFdkZI6yB8mofUs1tPmmwX6/ zt+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=UOHuuP/kxg1p+zCBIcKEZWhKOuhNNVi0QlFs7OkdjWY=; b=ityR+5VXMBM3dGQK7gje9c/fsHsKsFJrPwF3DzMPiBRQrMwTpqXheZpBy7WqV4JsPq 5M0O4K3qqNfHewYYjzOzPFtJbyJWdQa+ZWojJDzSU7TGpfJkoCyqBS+s9fHw7jaotvNs O6ccN5D/e8qg7qW7SP4Jn6Ks4HWLzbGKN7F7yO7C6QmFdRdUmvbkGDVqsCQ4DP+Q/CXt gLUF/fRw+lK33K/j/ya9hGSROnxtYhNCIjv4rc5uuyw7f/RGzq3LAlCoRQsNTnlWxrDd tMt4wcO8mRr9txlBzveQd4DJ6maWhQWefiXL5i3WNa8KVyZBSKTCV6TtL1pwyQdiWmCv plJw== X-Gm-Message-State: ALKqPwdDEy6buwQvBek3GgBHihFhksO0URcFo6NcLGyavWcYBZwSZMoW DNzXXnqOs/Ctsq/gk2hs91G/cZPGxhW9uL1eDg6ayQ== X-Google-Smtp-Source: AB8JxZp4QivYx/YNWtA7ZHqbGRc60QJcRSqX5VJf3++C51iFl5lHA+9ghiw6fsm+BdpAMLM124vJOsPsIG0OqeNoVN8= X-Received: by 2002:a50:9ecf:: with SMTP id a73-v6mr5498186edf.92.1527196338534; Thu, 24 May 2018 14:12:18 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:aa7:d696:0:0:0:0:0 with HTTP; Thu, 24 May 2018 14:12:17 -0700 (PDT) In-Reply-To: References: From: Dennis Lovely Date: Thu, 24 May 2018 14:12:17 -0700 Message-ID: Subject: Re: Question About Reaper To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary="00000000000042cbc6056cfa1b1c" --00000000000042cbc6056cfa1b1c Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable looks like you're connecting to a service listening on SSL but you don't have the CA used in your truststore On Thu, May 24, 2018 at 1:58 PM, Surbhi Gupta wrote: > Getting below error: > > Caused by: sun.security.validator.ValidatorException: PKIX path building > failed: sun.security.provider.certpath.SunCertPathBuilderException: > unable to find valid certification path to requested target > > at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397) > > at sun.security.validator.PKIXValidator.engineValidate( > PKIXValidator.java:302) > > at sun.security.validator.Validator.validate(Validator.java:260) > > at sun.security.ssl.X509TrustManagerImpl.validate( > X509TrustManagerImpl.java:324) > > at sun.security.ssl.X509TrustManagerImpl.checkTrusted( > X509TrustManagerImpl.java:281) > > at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted( > X509TrustManagerImpl.java:136) > > at sun.security.ssl.ClientHandshaker.serverCertificate( > ClientHandshaker.java:1501) > > ... 20 common frames omitted > > Any thought? > > On 24 May 2018 at 10:35, Surbhi Gupta wrote: > >> Another question, We use 9142 cqlsh port in one of the datacenter and on >> other datacenter we use 9042 port. >> How should we configure this ? >> >> On 24 May 2018 at 10:22, Surbhi Gupta wrote: >> >>> What is the impact of >>> PARALLEL - all replicas at the same time ? >>> Will it make repair faster,? >>> Do we expect more CPU , Load and memory usage in case if we use Paralle= l >>> , compare to other settings ? >>> >>> >>> >>> On 21 May 2018 at 22:55, Alexander Dejanovski >>> wrote: >>> >>>> You won't be able to have less segments than vnodes, so just use 256 >>>> segments per node, use parallel as repair parallelism, and set intensi= ty to >>>> 1. >>>> >>>> You apparently have more than 3TB per node, and that kind of density i= s >>>> always challenging when it comes to run "fast" repairs. >>>> >>>> Cheers, >>>> >>>> Le mar. 22 mai 2018 =C3=A0 07:28, Surbhi Gupta a >>>> =C3=A9crit : >>>> >>>>> We are on Dse 4.8.15 and it is cassandra 2.1. >>>>> What are the best configuration to use for reaper for 144 nodes with >>>>> 256 vnodes and it shows around 532TB data when we start opscenter rep= airs. >>>>> >>>>> We need to finish repair soon. >>>>> >>>>> On Mon, May 21, 2018 at 10:53 AM Alexander Dejanovski < >>>>> alex@thelastpickle.com> wrote: >>>>> >>>>>> Hi Subri, >>>>>> >>>>>> Reaper might indeed be your best chance to reduce the overhead of >>>>>> vnodes there. >>>>>> The latest betas include a new feature that will group vnodes sharin= g >>>>>> the same replicas in the same segment. This will allow to have less >>>>>> segments than vnodes, and is available with Cassandra 2.2 and onward= s (the >>>>>> improvement is especially beneficial with Cassandra 3.0+ as such tok= en >>>>>> ranges will be repaired in a single session). >>>>>> >>>>>> We have a gitter that you can join if you want to ask questions. >>>>>> >>>>>> Cheers, >>>>>> >>>>>> Le lun. 21 mai 2018 =C3=A0 15:29, Surbhi Gupta >>>>>> a =C3=A9crit : >>>>>> >>>>>>> Thanks Abdul >>>>>>> >>>>>>> On Mon, May 21, 2018 at 6:28 AM Abdul Patel >>>>>>> wrote: >>>>>>> >>>>>>>> We have a paramater in reaper yaml file called >>>>>>>> repairManagerSchrdulingIntervalSeconds default is 10 seconds , i >>>>>>>> tested with 8,6,5 seconds and found 5 seconds optimal for my envir= onment >>>>>>>> ..you go down further but it will have cascading effects in cpu an= d memory >>>>>>>> consumption. >>>>>>>> So test well. >>>>>>>> >>>>>>>> >>>>>>>> On Monday, May 21, 2018, Surbhi Gupta >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Thanks a lot for your inputs, >>>>>>>>> Abdul, how did u tune reaper? >>>>>>>>> >>>>>>>>> On Sun, May 20, 2018 at 10:10 AM Jonathan Haddad < >>>>>>>>> jon@jonhaddad.com> wrote: >>>>>>>>> >>>>>>>>>> FWIW the largest deployment I know about is a single reaper >>>>>>>>>> instance managing 50 clusters and over 2000 nodes. >>>>>>>>>> >>>>>>>>>> There might be bigger, but I either don=E2=80=99t know about it = or can=E2=80=99t >>>>>>>>>> remember. >>>>>>>>>> >>>>>>>>>> On Sun, May 20, 2018 at 10:04 AM Abdul Patel >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> I recently tested reaper and it actually helped us alot. Even >>>>>>>>>>> with our small footprint 18 node reaper takes close to 6 hrs.>>>>>>>>>> 13 hrs ,i was able to tune it 50%>. But it really depends on nu= mber nodes. >>>>>>>>>>> For example if you have 4 nodes then it runs on 4*256 = =3D1024 >>>>>>>>>>> segements , so for your env. Ut will be 256*144 close to 36k se= gements. >>>>>>>>>>> Better test on poc box how much time it takes and then proceed >>>>>>>>>>> further ..i have tested so far in 1 dc only , we can actually h= ave seperate >>>>>>>>>>> reaper instance handling seperate dc but havent tested it yet. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Sunday, May 20, 2018, Surbhi Gupta >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi, >>>>>>>>>>>> >>>>>>>>>>>> We have a cluster with 144 nodes( 3 datacenter) with 256 Vnode= s >>>>>>>>>>>> . >>>>>>>>>>>> When we tried to start repairs from opscenter then it showed >>>>>>>>>>>> 1.9Million ranges to repair . >>>>>>>>>>>> And even after doing compaction and strekamthroughput to 0 , >>>>>>>>>>>> opscenter is not able to help us much to finish repair in 9 da= ys timeframe . >>>>>>>>>>>> >>>>>>>>>>>> What is your thought on Reaper ? >>>>>>>>>>>> Do you think , Reaper might be able to help us in this scenari= o >>>>>>>>>>>> ? >>>>>>>>>>>> >>>>>>>>>>>> Thanks >>>>>>>>>>>> Surbhi >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>> Jon Haddad >>>>>>>>>> http://www.rustyrazorblade.com >>>>>>>>>> twitter: rustyrazorblade >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>> >>>>>>> -- >>>>>> ----------------- >>>>>> Alexander Dejanovski >>>>>> France >>>>>> @alexanderdeja >>>>>> >>>>>> Consultant >>>>>> Apache Cassandra Consulting >>>>>> http://www.thelastpickle.com >>>>>> >>>>>> >>>>>> -- >>>> ----------------- >>>> Alexander Dejanovski >>>> France >>>> @alexanderdeja >>>> >>>> Consultant >>>> Apache Cassandra Consulting >>>> http://www.thelastpickle.com >>>> >>> >>> >> > --00000000000042cbc6056cfa1b1c Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
looks like you're connecting to a service listening on= SSL but you don't have the CA used in your truststore

On Thu, May 24, 2018 at 1:5= 8 PM, Surbhi Gupta <surbhi.gupta01@gmail.com> wrote:<= br>
Getting below error:

Caused by: sun.security.validator.ValidatorExce= ption: PKIX path building failed: sun.security.provider.certpath.= SunCertPathBuilderException: unable to find valid certification path to req= uested target

at sun.security.validator.PKIXValidator.d= oBuild(PKIXValidator.java:397)

at sun.security.validator.PKIXValidator.e= ngineValidate(PKIXValidator.java:302)

at sun.security.validator.Validator.valid= ate(Validator.java:260)

at sun.security.ssl.X509TrustManagerImpl.= validate(X509TrustManagerImpl.java:324)

at sun.security.ssl.X509TrustManagerImpl.= checkTrusted(X509TrustManagerImpl.java:281)

at sun.security.ssl.X509TrustManagerImpl.= checkServerTrusted(X509TrustManagerImpl.java:136)

at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1501)

... 20 common frames omitted

<= div>
<= /div>
Any th= ought?

On 24 May 2018 at 10:35, Su= rbhi Gupta <surbhi.gupta01@gmail.com> wrote:
Another question, We use 9142 cq= lsh port in one of the datacenter and on other datacenter we use 9042 port.=
How should we configure this ?=C2=A0

On 24 May 2018 at 10= :22, Surbhi Gupta <surbhi.gupta01@gmail.com> wrote:
What is the impact of=C2=A0
PARALLEL= =C2=A0- all replicas at the same time ?
Will it make repair faster,?
Do we expect more CPU , Load and memory usage in cas= e if we use Parallel , compare to other settings ?



On 21 May 2018 at 22:55, Alexander De= janovski <alex@thelastpickle.com> wrote:
You won't be able to have less segments than vnodes, so just use = 256 segments per node, use parallel as repair parallelism, and set intensit= y to 1.

You apparently have more than 3TB per node, and = that kind of density is always challenging when it comes to run "fast&= quot; repairs.=C2=A0

Cheers,

<= div dir=3D"ltr">Le mar. 22 mai 2018 =C3=A0 07:28, Surbhi Gupta <surbhi.gupta01@gmail= .com> a =C3=A9crit=C2=A0:
We are= on Dse 4.8.15 and it is cassandra 2.1.
What are the best c= onfiguration to use for reaper for 144 nodes with 256 vnodes and it shows a= round 532TB data when we start opscenter repairs.

= We need to finish repair soon.

On Mon, May 21, 2018 at 10:53 AM Alexander Dejanovski <alex@thelastpickle.com>= wrote:
Hi Subri,

Reaper might= indeed be your best chance to reduce the overhead of vnodes there.
The latest betas include a new feature that will group vnodes sharing th= e same replicas in the same segment. This will allow to have less segments = than vnodes, and is available with Cassandra 2.2 and onwards (the improveme= nt is especially beneficial with Cassandra 3.0+ as such token ranges will b= e repaired in a single session).

We have a gitter = that you can join if you want to ask questions.

Ch= eers,

Le lun. 21 mai 2018 =C3= =A0 15:29, Surbhi Gupta <surbhi.gupta01@gmail.com> a =C3=A9crit=C2=A0:
Thanks Abdul

On Mon, May 21, 2018 at 6:28 AM Abdul Patel <abd786.ap@gmail.com&g= t; wrote:
We have a paramater in reaper yaml file= called repairManagerSchrdulingIntervalSeconds default is 10 seconds= =C2=A0 =C2=A0, i tested with 8,6,5 seconds and found 5 seconds optimal for = my environment ..you go down further but it will have cascading effects in = cpu and memory consumption.
So test well.


On Monday, = May 21, 2018, Surbhi Gupta <surbhi.gupta01@gmail.com> wrote:
Thanks a lot for your inputs,
Abdul, how did u tune reape= r?

On Sun, May 20, 2018 at 10= :10 AM Jonathan Haddad <jon@jonhaddad.com> wrote:
FWIW= the largest deployment I know about is a single reaper instance managing 5= 0 clusters and over 2000 nodes.=C2=A0

There might = be bigger, but I either don=E2=80=99t know about it or can=E2=80=99t rememb= er.=C2=A0

On Sun, May 20, 201= 8 at 10:04 AM Abdul Patel <abd786.ap@gmail.com> wrote:
= Hi,

I recently tested reaper and it actually helped us a= lot. Even with our small footprint 18 node reaper takes close to 6 hrs.<= intially took 13 hrs ,i was able to tune it 50%>. But it really depends = on number nodes. For example if you have 4 nodes then it runs on 4*256<v= nodes> =3D1024 segements , so for your env. Ut will be 256*144 close to = 36k segements.
Better test on poc box how much time it takes and = then proceed further ..i have tested so far in 1 dc only , we can actually = have seperate reaper instance handling seperate dc but havent tested it yet= .
--






-- <= br>
-----------------=
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting
<= a href=3D"http://www.thelastpickle.com/" target=3D"_blank">http://www.thela= stpickle.com


--
-----= ------------
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting




--00000000000042cbc6056cfa1b1c--