Return-Path: X-Original-To: apmail-mesos-user-archive@www.apache.org Delivered-To: apmail-mesos-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 1166219A9C for ; Fri, 15 Apr 2016 03:37:42 +0000 (UTC) Received: (qmail 87141 invoked by uid 500); 15 Apr 2016 03:37:41 -0000 Delivered-To: apmail-mesos-user-archive@mesos.apache.org Received: (qmail 87074 invoked by uid 500); 15 Apr 2016 03:37:41 -0000 Mailing-List: contact user-help@mesos.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@mesos.apache.org Delivered-To: mailing list user@mesos.apache.org Received: (qmail 87064 invoked by uid 99); 15 Apr 2016 03:37:41 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 15 Apr 2016 03:37:41 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 0DF3CC0D64 for ; Fri, 15 Apr 2016 03:37:41 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.194 X-Spam-Level: *** X-Spam-Status: No, score=3.194 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, NORMAL_HTTP_TO_IP=0.001, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, URI_HEX=1.313, WEIRD_PORT=0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx2-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id znObIFLPUv_D for ; Fri, 15 Apr 2016 03:37:39 +0000 (UTC) Received: from mail-yw0-f177.google.com (mail-yw0-f177.google.com [209.85.161.177]) by mx2-lw-us.apache.org (ASF Mail Server at mx2-lw-us.apache.org) with ESMTPS id DE9595F1E9 for ; Fri, 15 Apr 2016 03:37:38 +0000 (UTC) Received: by mail-yw0-f177.google.com with SMTP id i84so124743498ywc.2 for ; Thu, 14 Apr 2016 20:37:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to; bh=g4zBLB9n4vdZk4rA1Ne8JTjcSkw5X5/PGVKqCcsr4H4=; b=AhEhkRg8vnHAr3Z3wzUv+2XvR7jD0AEq1f73dLqnSfOx3HtziO3OTZu3TP9FtW9f7W mwXd9m9XpHjDSwx7+DpQ9XUlgwyGxzoUbmJ6Ua+Nsch5PXO0epbZdrKx+kvmGs7OVd7f NqrM8Z78PVa9gq9nKh0QYCnomFwp85R/A0nDSPlfNeCcl8AWtXJY6Wfj9KoBMdDxdzhS cGabm15dwKGsuXK/hF8uHdD3CE2wMm8cm+B67RQ8OoBGeWulfDt5PnXnz1E0dgcf4Tqr ruYePHwOjMLeM6MhymLPYhYdC/y9BfNn3cj5JRmCBt9ZB5nKYh6/p+qZOb6R51mHw5B1 3jfw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to; bh=g4zBLB9n4vdZk4rA1Ne8JTjcSkw5X5/PGVKqCcsr4H4=; b=iIzy8cR6TDKFCMoq+jreBbZ2KrBCQLq8Fg9ZuUYDT8uiz4U9yceLtArsRgwIO/3alZ JNXv2tKSaXSx3sLFa0AcAVtYdVYvvQKWBBAPZe//i2+1ozgAkAJ1cxD0sqUdGclpivEa cL5gKsh6y3LylI8JQm7rTl32nCs+jUanT3BFU7noZkV8BS8+ZX95j4tpE39M9uB8zdm5 wtKILzg5YQuRbUgBJ7239wsx7FQDsT3NHclF+zs26gkplF1P4ZcIJe5P/vKftYuK5Arr GhszK9lT6K5WmcNK0qm6bX+MInOfxsS01iyYMZF9gwen6sRFSS3OyhQ/fJLCDHMPTxoR NH+w== X-Gm-Message-State: AOPr4FWt4thL3hiyRAdeX+IgQWaFdyhsF+7shRSp+BF/ea3W/1c0hi98rQcaRhaCzUJwr2xIG/AncvszmhdnnA== MIME-Version: 1.0 X-Received: by 10.37.52.216 with SMTP id b207mr10584023yba.174.1460691458315; Thu, 14 Apr 2016 20:37:38 -0700 (PDT) Received: by 10.129.121.199 with HTTP; Thu, 14 Apr 2016 20:37:38 -0700 (PDT) In-Reply-To: References: <12iu7se7fo5i489opyh3vu6a6-0@mailer.nylas.com> Date: Fri, 15 Apr 2016 11:37:38 +0800 Message-ID: Subject: Re: decline_offer_duration for spark frameworks From: haosdent To: user Content-Type: multipart/alternative; boundary=001a1147bdfc7fe3cf05307dbb17 --001a1147bdfc7fe3cf05307dbb17 Content-Type: text/plain; charset=UTF-8 >I could not find any details on how set decline offer timeout in the spark framework. Seems spark framework don't support this while marathon support decline offer duration. Have you ever try set different role and reservation resources for marathon? Related documents are [reservation.md]( https://github.com/apache/mesos/blob/master/docs/reservation.md) and [ weights.md](https://github.com/apache/mesos/blob/master/docs/weights.md) On Fri, Apr 15, 2016 at 11:24 AM, David Greenberg wrote: > Hi Rodrick, > You should check out Cook (github.com/twosigma/cook). It acts as a server > for Spark drivers, except that it can manage preemption and fair-sharing > between those drivers, so that you don't encounter starvation. If you'd > like to discuss it, I'd be happy to hop on a call. > > Best, > David > > On Thu, Apr 14, 2016 at 11:09 PM haosdent wrote: > >> Hi, @Rodrick Which spark mode are you running? coarse-grained >> or fine-grained? >> >> On Fri, Apr 15, 2016 at 10:05 AM, Rodrick Brown >> wrote: >> >>> I have hundreds of small spark jobs running on my Mesos cluster >>> causing starvation to other frameworks like Marathon on my cluster. >>> >>> Is their a way to prevent these frameworks from getting offers so often? >>> >>> Apr 15 02:00:12 prod-mesos-m-3.$SERVER.com mesos-master[10259]: I0415 >>> 02:00:12.503734 10266 master.cpp:3641] Processing DECLINE call for offers: >>> [ 50ceafa4-f3c1-4738-a9eb-c5d3bf0ff742-O7112667 ] for >>> framework 50ceafa4-f3c1-4738-a9eb-c5d3bf0ff742-15936 (KafkaDirectConsumer[trades-topic]) >>> at scheduler-9e557d33-e4a4-44ce-9dbe-0a7ca7c4842d@172.1.121.183:34858. >>> >>> >>> >>> >>> >>> -- >>> >>> *Rodrick Brown* / Systems Engineer >>> >>> +1 917 445 6839 / rodrick@orchardplatform.com >>> >>> >>> *Orchard Platform* >>> >>> 101 5th Avenue, 4th Floor, New York, NY 10003 >>> >>> http://www.orchardplatform.com >>> >>> Orchard Blog | Marketplace >>> Lending Meetup >>> >>> *NOTICE TO RECIPIENTS*: This communication is confidential and intended >>> for the use of the addressee only. If you are not an intended recipient of >>> this communication, please delete it immediately and notify the sender >>> by return email. Unauthorized reading, dissemination, distribution or >>> copying of this communication is prohibited. This communication does not constitute >>> an offer to sell or a solicitation of an indication of interest to purchase >>> any loan, security or any other financial product or instrument, nor is it >>> an offer to sell or a solicitation of an indication of interest to purchase >>> any products or services to any persons who are prohibited from receiving >>> such information under applicable law. The contents of this communication >>> may not be accurate or complete and are subject to change without notice. >>> As such, Orchard App, Inc. (including its subsidiaries and affiliates, >>> "Orchard") makes no representation regarding the accuracy or >>> completeness of the information contained herein. The intended recipient is >>> advised to consult its own professional advisors, including those >>> specializing in legal, tax and accounting matters. Orchard does not >>> provide legal, tax or accounting advice. >>> >> >> >> >> -- >> Best Regards, >> Haosdent Huang >> > -- Best Regards, Haosdent Huang --001a1147bdfc7fe3cf05307dbb17 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
>I could not find any details on how set decline offer = timeout in the spark framework.=C2=A0
Seems spark framework don't s= upport this while marathon support decline offer duration.

Have you ever try set different role and reservation resources for= marathon? Related documents are [reserva= tion.md](https://github.com/apache/mesos/blob/master/docs/reservation.= md) and [weights.md](https://github.com/= apache/mesos/blob/master/docs/weights.md)

On Fri, Apr 15, 2016 at 11:24 AM, D= avid Greenberg <dsg123456789@gmail.com> wrote:
Hi Rodrick,
You should check= out Cook (gi= thub.com/twosigma/cook). It acts as a server for Spark drivers, except = that it can manage preemption and fair-sharing between those drivers, so th= at you don't encounter starvation. If you'd like to discuss it, I&#= 39;d be happy to hop on a call.

Best,
Da= vid

On Thu, Apr 14, 2016 at 11:09 PM haosdent <<= a href=3D"mailto:haosdent@gmail.com" target=3D"_blank">haosdent@gmail.com> wrote:
Hi, = @Rodrick Which spark mode are you running? =C2=A0coarse-grained or=C2=A0fin= e-grained?

On Fri, Apr 15, 2016 at 10:05 AM, Rodrick B= rown <rodrick@orchard-app.com> wrote:
I have hundreds of small spark jobs running on my=C2= =A0Mesos=C2=A0cluster causing=C2=A0starvation to other frameworks like Mara= thon on my cluster.=C2=A0

<= div>Is their a way to prevent these frameworks from getting offers so often= ?=C2=A0

Apr 15 02:00:12=C2=A0prod-mesos-m-3.$SERVER.com= =C2=A0mesos-master[10259]: I0415 02:00:12.503734 10266 master.cpp:3641] Pro= cessing DECLINE call for offers: [=C2=A050ceafa4-f3c1-4738-a9eb-c5d3bf0ff74= 2-O7112667=C2=A0] for framework=C2=A050ceafa4-f3c1-4738-a9eb-c5d3bf0ff742-1= 5936=C2=A0(KafkaDirectConsumer[trades-topic]) at=C2=A0scheduler-9e557d33-e4a4-44ce-9dbe-0a7ca7c4842d@172.1.121.183:34858<= /a>.=C2=A0





--= =C2=A0

Rodric= k Brown=C2=A0/=C2=A0Systems Engineer=C2=A0

+1 917 445 6839 /=C2=A0rodrick@orchardplatform.com=

Orchard Pla= tform=C2=A0

101= 5th Avenue, 4th Floor, New York, NY 10003=C2=A0

http://www.orchardplatform.com

Orchard Blog=C2=A0|=C2=A0Marketplace Lending M= eetup


NOTICE TO RECIPIENTS: This communication= is confidential and intended for the use of the addressee only. If you are= not an intended recipient of this communication, please delete it=C2=A0immediately and notify the sender by return email. Unauthorized r= eading, dissemination, distribution or copying of this communication is pro= hibited. This communication does not=C2=A0constitute an offer to sel= l or a solicitation of an indication of interest to purchase any loan, secu= rity or any other financial product or instrument, nor is it an offer to se= ll or a solicitation of an=C2=A0indication of interest to purchase any prod= ucts or services to any persons who are prohibited from receiving such info= rmation under applicable law. The contents of this communication may not be= =C2=A0accurate or complete and are subject to change without notice. As suc= h, Orchard App, Inc. (including its subsidiaries and affiliates, "Orch= ard") makes no representation regarding the=C2=A0accuracy or com= pleteness of the information contained herein. The intended recipient is ad= vised to consult its own professional advisors, including those specializin= g in legal, tax and accounting=C2=A0matters. Orchard does not provid= e legal, tax or accounting advice.



--
Be= st Regards,
Haosdent Huang



--
=
Best Regards,
Haosdent Huang
--001a1147bdfc7fe3cf05307dbb17--