Return-Path: X-Original-To: apmail-helix-user-archive@minotaur.apache.org Delivered-To: apmail-helix-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 7155117C41 for ; Fri, 13 Mar 2015 00:24:55 +0000 (UTC) Received: (qmail 19255 invoked by uid 500); 13 Mar 2015 00:24:55 -0000 Delivered-To: apmail-helix-user-archive@helix.apache.org Received: (qmail 19214 invoked by uid 500); 13 Mar 2015 00:24:55 -0000 Mailing-List: contact user-help@helix.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@helix.apache.org Delivered-To: mailing list user@helix.apache.org Received: (qmail 19204 invoked by uid 99); 13 Mar 2015 00:24:55 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Mar 2015 00:24:55 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of varun@pinterest.com designates 209.85.223.178 as permitted sender) Received: from [209.85.223.178] (HELO mail-ie0-f178.google.com) (209.85.223.178) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Mar 2015 00:24:24 +0000 Received: by iecvj10 with SMTP id vj10so67409511iec.0 for ; Thu, 12 Mar 2015 17:24:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pinterest.com; s=google; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=7ccKfoOkiP5pbEc8hKighQKHTMxtbqCUnsbyAvdI3f0=; b=fkNjJDmuJTGGwt/IIx3RBZPQs9k18PPE783OZ+mkMKZ/4+GZWsC3jF7Y1ZpZgnl/r6 xGaHp4g4AvanPrg5tXJaTjT0iAbvway08LVOFqSpeiZtHW8HbG4X9QY6L7hEhVm5wZA6 HCtjL5zqTEsf/oc+ZYeAZNWr4Wfm1lfqiGGGA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=7ccKfoOkiP5pbEc8hKighQKHTMxtbqCUnsbyAvdI3f0=; b=ZYPAu/HNPcw4ELfXcjtMYhShdLKnJ2YvFiGfd+7res0TQWbO/pSBurrssP8OSxWlBJ EEMBT0WFt2lKb+AVWH2AAZQAAQMUykBuPG2lxdgYM4dr7wLKfc+mNl4CniSJrv4EWzDe c00WLn33IJY8GBam4XCYYAfCzBat7UeJsJ8iRSaS8CYK890W886qBQWsk1C/eKEBINDw tbLR4g5TrIoPb+gfWxJiceHUgxOormIl6HwRuRIE2uHnvrgNEnFpBJr+qOETzbJubd0U XLE81NCWJFX3+JV3pV0N7OurlLtWeEUuyR88/bjSM+B2P5kSiiqxRrEEUIbkKY2zEOVG B7HQ== X-Gm-Message-State: ALoCoQmHGqeV3Z2tG31QJU/ZNnXfNxleTJ/qwYCqUztWPWhEZ4kciYRyBC8PWpsYI1jqHckqD6PM MIME-Version: 1.0 X-Received: by 10.107.32.73 with SMTP id g70mr40977330iog.55.1426206262278; Thu, 12 Mar 2015 17:24:22 -0700 (PDT) Received: by 10.107.53.7 with HTTP; Thu, 12 Mar 2015 17:24:22 -0700 (PDT) In-Reply-To: References: <23CA11DC8830BA44A37C6B44B14D013CB9D23AD7@ESV4-MB02.linkedin.biz> Date: Thu, 12 Mar 2015 17:24:22 -0700 Message-ID: Subject: Re: Cutting a release 0.6.5 tonight From: Varun Sharma To: user@helix.apache.org Content-Type: multipart/alternative; boundary=001a11404078a3d26d05112085f8 X-Virus-Checked: Checked by ClamAV on apache.org --001a11404078a3d26d05112085f8 Content-Type: text/plain; charset=UTF-8 I tested it out using a resource with 8192 partitions and 8 buckets and it worked out just fine - (disable and drop). On Thu, Mar 12, 2015 at 4:08 PM, Varun Sharma wrote: > I just deployed this to a server node and when the current states node is > moved over - the bucket nodes are no longer present - the node has executed > dropped transitions for all the nodes. Is it expected to not contain any > buckets ? > > On Thu, Mar 12, 2015 at 1:33 PM, Zhen Zhang wrote: > >> Should be no problem. Controller and participant are compatible. >> >> ------------------------------ >> *From:* Varun Sharma [varun@pinterest.com] >> *Sent:* Thursday, March 12, 2015 1:29 PM >> *To:* user@helix.apache.org >> >> *Subject:* Re: Cutting a release 0.6.5 tonight >> >> Just to confirm - it wont be a problem if the controller is running >> 0.6.5 and participant runs 0.6.4 for a while due to this ? >> >> On Thu, Mar 12, 2015 at 1:25 PM, kishore g wrote: >> >>> Yes, I will think of a way to make it backwards compatible. >>> >>> Thanks >>> Kishore G >>> On Mar 12, 2015 1:21 PM, "Varun Sharma" wrote: >>> >>>> Kishore, I will be testing it out today. Btw, did you also look at the >>>> issue with change of Abstract method signature createNewStateModel in >>>> StateModelFactory ? >>>> >>>> On Thu, Mar 12, 2015 at 9:58 AM, kishore g wrote: >>>> >>>>> Jason and I have checked in the changes needed. I am not sure if >>>>> TestSchedulerMsg is fixed. >>>>> >>>>> Last call for any other changes needed. >>>>> >>>>> Varun, it will be great if you can test the bug fixes for bucketing >>>>> before we cut the release. >>>>> >>>>> thanks, >>>>> Kishore G >>>>> >>>>> On Wed, Mar 11, 2015 at 5:42 PM, kishore g >>>>> wrote: >>>>> >>>>>> Thanks Lei, I fixed the first two. >>>>>> >>>>>> Here is what caused the failure. I added the code to copy all >>>>>> simple fields from IS to EV when we update the ExternalView in controller. >>>>>> This is to enable compression in ExternalView if its set in IS. I could >>>>>> have copied only enableCompression variable but I thought its good to have >>>>>> partition number/replica etc in ExternalView as well. Let me know if you >>>>>> foresee any problem in this. Only thing I could think of is IdealState is >>>>>> deleted in which case I copy the simplefields from existing externalview. >>>>>> >>>>>> I dont understand why testschedulermsg test cases are failing. >>>>>> Uncommenting my code does not help either. >>>>>> >>>>>> >>>>>> On Wed, Mar 11, 2015 at 4:06 PM, Kanak Biscuitwala < >>>>>> kanak.b@hotmail.com> wrote: >>>>>> >>>>>>> The first two are concerning. I don't think the scheduler message >>>>>>> test failures should block releases. >>>>>>> >>>>>>> ---------------------------------------- >>>>>>> > From: lxia@linkedin.com.INVALID >>>>>>> > To: dev@helix.apache.org >>>>>>> > CC: user@helix.apache.org >>>>>>> > Subject: RE: Cutting a release 0.6.5 tonight >>>>>>> > Date: Wed, 11 Mar 2015 21:25:15 +0000 >>>>>>> > >>>>>>> > The test failed are: >>>>>>> > >>>>>>> > >>>>>>> org.apache.helix.integration.TestExternalViewUpdates.testExternalViewUpdates >>>>>>> > >>>>>>> org.apache.helix.integration.TestEnableCompression.testEnableCompressionResource >>>>>>> > org.apache.helix.integration.TestSchedulerMessage.testSchedulerMsg3 >>>>>>> > org.apache.helix.integration.TestSchedulerMessage.testSchedulerMsg4 >>>>>>> > >>>>>>> org.apache.helix.integration.TestSchedulerMessage.testSchedulerMsgContraints >>>>>>> > >>>>>>> org.apache.helix.integration.TestSchedulerMessage.testSchedulerMsgUsingQueue >>>>>>> > >>>>>>> > I got these from my local build too. (mvn clean install package on >>>>>>> helix-0.6.x) >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > Thanks >>>>>>> > Lei >>>>>>> > >>>>>>> > -- >>>>>>> > >>>>>>> > Lei Xia >>>>>>> > Software Engineer >>>>>>> > Data Infrastructure/Distributed Data Systems/Nuage >>>>>>> > LinkedIn >>>>>>> > >>>>>>> > lxia@linkedin.com >>>>>>> > www.linkedin.com/in/lxia1 >>>>>>> > >>>>>>> > ________________________________________ >>>>>>> > From: kishore g [g.kishore@gmail.com] >>>>>>> > Sent: Wednesday, March 11, 2015 2:03 PM >>>>>>> > To: dev@helix.apache.org >>>>>>> > Cc: user@helix.apache.org >>>>>>> > Subject: Re: Cutting a release 0.6.5 tonight >>>>>>> > >>>>>>> > Hi Lei, >>>>>>> > >>>>>>> > Can you point to the failures? >>>>>>> > >>>>>>> > thanks, >>>>>>> > Kishore G >>>>>>> > >>>>>>> > On Wed, Mar 11, 2015 at 1:19 PM, Lei Xia >>>>>>> wrote: >>>>>>> > >>>>>>> >> Hi, Kishore >>>>>>> >> >>>>>>> >> I saw there are regression test failures from last two recent >>>>>>> commits >>>>>>> >> on 0.6.x branch, running from both local box and Linkedin's >>>>>>> hudson jobs. >>>>>>> >> Are we going to fix them before the release? >>>>>>> >> >>>>>>> >> >>>>>>> >> Thanks >>>>>>> >> Lei >>>>>>> >> >>>>>>> >> -- >>>>>>> >> >>>>>>> >> Lei Xia >>>>>>> >> Software Engineer >>>>>>> >> Data Infrastructure/Distributed Data Systems/Nuage >>>>>>> >> LinkedIn >>>>>>> >> >>>>>>> >> lxia@linkedin.com >>>>>>> >> www.linkedin.com/in/lxia1 >>>>>>> >> >>>>>>> >> ________________________________________ >>>>>>> >> From: kishore g [g.kishore@gmail.com] >>>>>>> >> Sent: Wednesday, March 11, 2015 12:04 PM >>>>>>> >> To: user@helix.apache.org >>>>>>> >> Cc: dev@helix.apache.org >>>>>>> >> Subject: Re: Cutting a release 0.6.5 tonight >>>>>>> >> >>>>>>> >> Will write a test case for migration and document the steps >>>>>>> >> On Mar 11, 2015 11:37 AM, "Varun Sharma" >>>>>>> wrote: >>>>>>> >> >>>>>>> >>> What would be the migration path from non-compressed buckets to >>>>>>> >> compressed >>>>>>> >>> non bucket resources ? It seems even the CURRENTSTATES are being >>>>>>> bucketed >>>>>>> >>> in this case, I thought that was not expected with bucketing. >>>>>>> Does the >>>>>>> >>> controller read these current states appropriately ? To migrate, >>>>>>> it seems >>>>>>> >>> that we would need to also rewrite the CURRENT STATES ? >>>>>>> >>> >>>>>>> >>> On Wed, Mar 11, 2015 at 10:02 AM, kishore g >>>>>>> wrote: >>>>>>> >>> >>>>>>> >>>> Hi, >>>>>>> >>>> >>>>>>> >>>> I will work with Jason to cut a 0.6.5 release tonight. >>>>>>> >>>> >>>>>>> >>>> The new thing I added is to enableCompression while storing >>>>>>> data in >>>>>>> >>>> Zookeeper, this allows us to go up to 100k partitions per >>>>>>> resource >>>>>>> >> without >>>>>>> >>>> having to use bucketing feature. We also fixed few bugs with >>>>>>> bucketed >>>>>>> >>>> resource just in case some one needs it. >>>>>>> >>>> >>>>>>> >>>> The property store api needs some changes, I plan to get it in >>>>>>> today. >>>>>>> >>>> >>>>>>> >>>> Let me know if you need any other changes to be included. Are >>>>>>> there any >>>>>>> >>>> changes that went into 0.7.x branch that we need to merge it >>>>>>> back in to >>>>>>> >>>> 0.6.x ? >>>>>>> >>>> >>>>>>> >>>> thanks, >>>>>>> >>>> Kishore G >>>>>>> >>>> >>>>>>> >>>> >>>>>>> >>>> >>>>>>> >>>> >>>>>>> >>>> >>>>>>> >>>> >>>>>>> >>> >>>>>>> >> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>> >>>> >> > --001a11404078a3d26d05112085f8 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
I tested it out using a resource with 8192 partitions and = 8 buckets and it worked out just fine - (disable and drop).

On Thu, Mar 12, 2015 at 4:0= 8 PM, Varun Sharma <varun@pinterest.com> wrote:
I just deployed this to a server n= ode and when the current states node is moved over - the bucket nodes are n= o longer present - the node has executed dropped transitions for all the no= des. Is it expected to not contain any buckets ?

On Thu, Mar 12, 2015 at 1:33 PM, Zhen Zhang <zzhang@linkedin.com&g= t; wrote:
Should be no problem. Controller and participant are compatible.

Fro= m: Varun Sharma [varun@pinterest.com]
Sent: Thursday, March 12, 2015 1:29 PM
To: user@= helix.apache.org

Subject: Re: Cutting a release 0.6.5 tonight

Just to confirm - it wont be a problem if the controller i= s running 0.6.5 and participant runs 0.6.4 for a while due to this ?

On Thu, Mar 12, 2015 at 1:25 PM, kishore g <g.kishore@gmai= l.com> wrote:

Yes, I will think of a way to make it backwards compatible.<= /p>

Thanks
Kishore G

On Mar 12, 2015 1:21 PM, "Varun Sharma"= ; <varun@pinter= est.com> wrote:
Kishore, I will be testing it out today. Btw, did you also= look at the issue with change of Abstract method signature createNewStateM= odel in StateModelFactory ?

On Thu, Mar 12, 2015 at 9:58 AM, kishore g <g.kishore@gmai= l.com> wrote:
Jason and I have checked in the changes needed. I am not s= ure if TestSchedulerMsg is fixed.=C2=A0

Last call for any other changes needed.=C2=A0

Varun, it will be great if you can test the bug fixes for bucketing be= fore we cut the release.

thanks,
Kishore G

On Wed, Mar 11, 2015 at 5:42 PM, kishore g <g.kishore@gmai= l.com> wrote:
Thanks Lei, I fixed the first two.=C2=A0

Here is what caused the failure. I added the code to copy all simple f= ields from IS to EV when we update the ExternalView in controller. This is = to enable compression in ExternalView if its set in IS. I could have copied= only enableCompression variable but I thought its good to have partition number/replica etc in ExternalVie= w as well. Let me know if you foresee any problem in this. Only thing I cou= ld think of is IdealState is deleted in which case I copy the simplefields = from existing externalview.

I dont understand why testschedulermsg test cases are failing. Uncomme= nting my code does not help either.


On Wed, Mar 11, 2015 at 4:06 PM, Kanak Biscuitwa= la <kanak.b@hotmai= l.com> wrote:
The first two are concerning. I don't think the scheduler message test = failures should block releases.

----------------------------------------
> From: lxia@linkedin.com.INVALID
> To: dev@heli= x.apache.org
> CC: user@he= lix.apache.org
> Subject: RE: Cutting a release 0.6.5 tonight
> Date: Wed, 11 Mar 2015 21:25:15 +0000
>
> The test failed are:
>
> org.apache.helix.integration.TestExternalViewUpdates.testExternalViewU= pdates
> org.apache.helix.integration.TestEnableCompression.testEnableCompressi= onResource
> org.apache.helix.integration.TestSchedulerMessage.testSchedulerMsg3 > org.apache.helix.integration.TestSchedulerMessage.testSchedulerMsg4 > org.apache.helix.integration.TestSchedulerMessage.testSchedulerMsgCont= raints
> org.apache.helix.integration.TestSchedulerMessage.testSchedulerMsgUsin= gQueue
>
> I got these from my local build too. (mvn clean install package on hel= ix-0.6.x)
>
>
>
> Thanks
> Lei
>
> --
>
> Lei Xia
> Software Engineer
> Data Infrastructure/Distributed Data Systems/Nuage
> LinkedIn
>
> lxia@linkedin.c= om
> www.lin= kedin.com/in/lxia1
>
> ________________________________________
> From: kishore g [g.kishore@gmail.com]
> Sent: Wednesday, March 11, 2015 2:03 PM
> To: dev@heli= x.apache.org
> Cc: user@he= lix.apache.org
> Subject: Re: Cutting a release 0.6.5 tonight
>
> Hi Lei,
>
> Can you point to the failures?
>
> thanks,
> Kishore G
>
> On Wed, Mar 11, 2015 at 1:19 PM, Lei Xia <lxia@linkedin.com.invalid= > wrote:
>
>> Hi, Kishore
>>
>> I saw there are regression test failures from last two recent comm= its
>> on 0.6.x branch, running from both local box and Linkedin's hu= dson jobs.
>> Are we going to fix them before the release?
>>
>>
>> Thanks
>> Lei
>>
>> --
>>
>> Lei Xia
>> Software Engineer
>> Data Infrastructure/Distributed Data Systems/Nuage
>> LinkedIn
>>
>> lxia@linked= in.com
>> www= .linkedin.com/in/lxia1
>>
>> ________________________________________
>> From: kishore g [g.kishore@gmail.com]
>> Sent: Wednesday, March 11, 2015 12:04 PM
>> To: use= r@helix.apache.org
>> Cc: dev@= helix.apache.org
>> Subject: Re: Cutting a release 0.6.5 tonight
>>
>> Will write a test case for migration and document the steps
>> On Mar 11, 2015 11:37 AM, "Varun Sharma" <varun@pinterest.com> w= rote:
>>
>>> What would be the migration path from non-compressed buckets t= o
>> compressed
>>> non bucket resources ? It seems even the CURRENTSTATES are bei= ng bucketed
>>> in this case, I thought that was not expected with bucketing. = Does the
>>> controller read these current states appropriately ? To migrat= e, it seems
>>> that we would need to also rewrite the CURRENT STATES ?
>>>
>>> On Wed, Mar 11, 2015 at 10:02 AM, kishore g <g.kishore@gmail.com> wrot= e:
>>>
>>>> Hi,
>>>>
>>>> I will work with Jason to cut a 0.6.5 release tonight.
>>>>
>>>> The new thing I added is to enableCompression while storin= g data in
>>>> Zookeeper, this allows us to go up to 100k partitions per = resource
>> without
>>>> having to use bucketing feature. We also fixed few bugs wi= th bucketed
>>>> resource just in case some one needs it.
>>>>
>>>> The property store api needs some changes, I plan to get i= t in today.
>>>>
>>>> Let me know if you need any other changes to be included. = Are there any
>>>> changes that went into 0.7.x branch that we need to merge = it back in to
>>>> 0.6.x ?
>>>>
>>>> thanks,
>>>> Kishore G
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 <= /div>






--001a11404078a3d26d05112085f8--