Mailing-List: contact dev-help@aurora.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@aurora.incubator.apache.org
Received-SPF: pass (athena.apache.org: domain of josh@foursquare.com
 designates 209.85.220.177 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAD99eHOeE-QuvLxsZJXLxvdi4gs0kW260hNg54ipx87RNiuDYg@mail.gmail.com>
References: 
 <CAD99eHOgt58es4YkUk55y2Aq8=r9_H-OqAEFNLxWenwBMjEJFw@mail.gmail.com>
 <CAFTdr0B=Q6bWS0FqBetXVKDhgm0Q=9=Jiy3RH-HA99iCoBNuGw@mail.gmail.com>
 <CABkJX4L2e4zxxyESeDDKjjgiA3rJynS4kSa4s_dOKd5q45am2A@mail.gmail.com>
 <CAD99eHNRn1G50QxZgAvBhFdnkyDc2OMu==4PYu_Dhg4rkKfK2Q@mail.gmail.com>
 <CAAATh-ayMcoXuZvq0-t7TrL1sFyYHKK_fuhk4Leir3ez-3sPgg@mail.gmail.com>
 <CABkJX4KFWZPANrUj7HtLr8Ktu4EDfSmPu5WVaT_sb_oY+VmCgA@mail.gmail.com>
 <CABkJX4J6eHP3RU=grBhspQ2K9Bvy1+A=fZqtuzOWpzSKCtpAgw@mail.gmail.com>
 <CAD99eHOeE-QuvLxsZJXLxvdi4gs0kW260hNg54ipx87RNiuDYg@mail.gmail.com>
From: Josh Adams <josh@foursquare.com>
Date: Wed, 16 Jul 2014 12:21:24 -0700
Message-ID: 
 <CABkJX4+7OfE-RWXXkEYrpes1rb-=wHZ+yG=DwASe-UZ2hpY2yg@mail.gmail.com>
Subject: Re: Task Constraints
To: Kevin Burg <kburg@foursquare.com>
Cc: Kevin Sweeney <kevints@apache.org>,
 Aurora <dev@aurora.incubator.apache.org>,
	Brian Wickman <wickman@apache.org>, Leo Kim <leo@foursquare.com>
Content-Type: multipart/alternative; boundary=20cf3071cfc6fb326e04fe546e1a

--20cf3071cfc6fb326e04fe546e1a
Content-Type: text/plain; charset=UTF-8

+Leo Kim who is looking at the compiler error with us.


On Wed, Jul 16, 2014 at 8:25 AM, Kevin Burg <kburg@foursquare.com> wrote:

> The idea with the fix is to read the slave's attributes right off the
> offer rather than going into 'AttributeStore' and keying on the slave's
> name. The slave's resources are read off the offer in this way, so I don't
> see why it can't be done with attributes as well.
>
> Someone who understands all the places where SchedulingFilter.filter is
> used might be able to fix this better than I can.
>
>
> On Wed, Jul 16, 2014 at 6:40 AM, Josh Adams <josh@foursquare.com> wrote:
>
>> Hi there,
>>
>> Given that we would need to disrupt running jobs to add constraints in
>> the future we are blocking on
>> https://issues.apache.org/jira/browse/AURORA-582 before we can push any
>> of our services on to Aurora in production.
>>
>> Kevin Burg attempted to resolve the related bug
>> https://issues.apache.org/jira/browse/AURORA-328 by making some changes
>> here:
>> https://github.com/foursquare/incubator-aurora/commit/b1962fad3fe9ef76954fa107abed25d78b809331
>> but we seem to be getting a type mismatch when compiling the code.
>>
>> Any help and/or info on the bugfix progress would be much appreciated.
>> Aside from AURORA-582 we are ready to roll (pun intended!)
>>
>> Best,
>> Josh
>>
>>
>> On Mon, Jul 14, 2014 at 11:42 AM, Josh Adams <josh@foursquare.com> wrote:
>>
>>> Ah, makes sense. We'll try that. Thanks for clarifying this Kevin.
>>>
>>> Josh
>>>
>>>
>>> On Mon, Jul 14, 2014 at 11:30 AM, Kevin Sweeney <kevints@apache.org>
>>> wrote:
>>>
>>>> Slaves persist their attributes (including attributes) across restarts
>>>> due to slave recovery (that's what allows you to upgrade mesos in-place
>>>> without killing the tasks they're managing). Unfortunately to change
>>>> attributes you need to remove persisted slave metadata (the "meta"
>>>> directory). This will kill all of a slave's underlying tasks but the newly
>>>> registered slave should have the correct attributes.
>>>>
>>>>
>>>> On Mon, Jul 14, 2014 at 11:26 AM, Kevin Burg <kburg@foursquare.com>
>>>> wrote:
>>>>
>>>>> I've confirmed by looking at that endpoint that new attributes are not
>>>>> being picked up and modified attributes are retaining their old values.
>>>>> This is after restarting both the slaves and the scheduler process.
>>>>>
>>>>>
>>>>> On Mon, Jul 14, 2014 at 11:09 AM, Josh Adams <josh@foursquare.com>
>>>>> wrote:
>>>>>
>>>>> > Thanks Brian. Kevin should have some followup questions shortly.
>>>>> >
>>>>> > Josh
>>>>> >
>>>>> >
>>>>> > On Mon, Jul 14, 2014 at 10:37 AM, Brian Wickman <wickman@apache.org>
>>>>> > wrote:
>>>>> >
>>>>> >> host/rack should not be treated specially.
>>>>> >>
>>>>> >> If you go to the "/slaves" endpoint on the scheduler UI, what does
>>>>> it
>>>>> >> report as attributes being exported by your slaves?  You might want
>>>>> to
>>>>> >> validate there that the "staging" attribute got picked up properly.
>>>>>  If
>>>>> >> it's not getting picked up (e.g. the attributes are getting cached
>>>>> >> incorrectly by the scheduler?) then you should file an issue.
>>>>> >>
>>>>> >>
>>>>> >> On Fri, Jul 11, 2014 at 5:24 PM, Kevin Burg <kburg@foursquare.com>
>>>>> wrote:
>>>>> >>
>>>>> >>> Hi,
>>>>> >>>
>>>>> >>> I'm having trouble getting the task constraint resolver worker with
>>>>> >>> attributes other than 'host' and 'rack.' Are arbitrary attribute
>>>>> keys in
>>>>> >>> the mesos slaves supported currently?
>>>>> >>>
>>>>> >>> Here is the setup.
>>>>> >>>
>>>>> >>> The slaves are configured to run with
>>>>> >>> `--attributes=host:<host>;rack:<rack>;staging:true`
>>>>> >>>
>>>>> >>> (I've also tried this with staging:1, and staging:foo)
>>>>> >>>
>>>>> >>> The constraint generated from the .aurora config looks like the
>>>>> following
>>>>> >>> Constraint(name:staging, constraint:<TaskConstraint
>>>>> >>> value:ValueConstraint(negated:false, values:[true])>)
>>>>> >>>
>>>>> >>> The schedule request then gets vetoed with the following veto
>>>>> object:
>>>>> >>> Veto{reason=Constraint not satisfied: staging, score=1000,
>>>>> >>> valueMismatch=true}]
>>>>> >>>
>>>>> >>> The constraints generated for 'host' and 'rack' look identical
>>>>> except for
>>>>> >>> the different name of course. I've even tried bouncing every mesos
>>>>> and
>>>>> >>> aurora process on the machine to see if maybe stale attributes
>>>>> were being
>>>>> >>> assigned to the slaves. All the offers being made to the master
>>>>> look
>>>>> >>> correct though, which leads me to believe that the constraint
>>>>> solver just
>>>>> >>> doesn't work for arbitrary attributes.
>>>>> >>>
>>>>> >>> We would appreciate any help you can offer.
>>>>> >>>
>>>>> >>> Thanks,
>>>>> >>> Kevin
>>>>> >>>
>>>>> >>
>>>>> >>
>>>>> >
>>>>> >
>>>>> > --
>>>>> > ===============
>>>>> > josh adams
>>>>> > production engineer
>>>>> > foursquare
>>>>> >
>>>>> > (gv) 415-830-4106
>>>>> > ===============
>>>>> > foursquare.com/jobs
>>>>> >
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> ===============
>>> josh adams
>>> production engineer
>>> foursquare
>>>
>>> (gv) 415-830-4106
>>> ===============
>>> foursquare.com/jobs
>>>
>>
>>
>>
>> --
>> ===============
>> josh adams
>> production engineer
>> foursquare
>>
>> (gv) 415-830-4106
>> ===============
>> foursquare.com/jobs
>>
>
>


-- 
===============
josh adams
production engineer
foursquare

(gv) 415-830-4106
===============
foursquare.com/jobs

--20cf3071cfc6fb326e04fe546e1a--