hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sunil Govind <sunil.gov...@gmail.com>
Subject Re: how to add a shareable node label?
Date Wed, 12 Oct 2016 14:20:34 GMT
Hi Frank

Thanks for sharing more details. Let me try this combination and I might be
wrong, so pls correct me. And I think sharable node-label could help here.


Labels:

Node1-4 = default label

Node8-9 = "special" label


Queues:

"ProdQ" accessable-labels  is ""  (only default label)

"TestQ" accessable-labels  is ""  (only default label)


"LabeledQ" accessable-labels "special"


Capacity Per Queue:

"ProdQ"

capacity=50%

max-capacity=100%


"TestQ"

capacity=50%

max-capacity=50%


"LabeledQ"

special.capacity=100%

special.max-capacity=100%


Various Choices:

* Jobs in ProdQ is assured with 50% of default label resources and it can
go to 100% if there are no resource running in TestQ

* Jobs in TestQ can only get 50% of default label resources.

* If jobs in ProdQ or TestQ needed to make use of "special" label machines,
it is only possible when there are resource available in "special" label.
"special" is a non-exclusive label which can share its resource with
"default" label.

* Any job submitted in "LabeledQ"  is assured with 100%
of special resources and can use 100% if nothing is there. I think
preemption could be made optional here.

If Inter queue preemption is enabled, we can enforce a normalization faster
for default label, else apps might need to wait.  We could also try another
approach as I shared in an earlier mail. But it ensure some % of resources
for ProdQ and TestQ in LabeledQ which may not be suitable.


Thanks

Sunil

On Tue, Oct 11, 2016 at 10:38 PM Frank Luo <jluo@merkleinc.com> wrote:

>
>
>
>
>
>
>
>
> Hah, how so? I am confused as I was under impression that I needed sharing
> but not preemption.
>
>
>
> Let’s model this out.
>
>
>
> Assuming I got 4 “normal” machines node1-4, and two special node8 and
> node9 where JobA can be executed on.
>
>
>
> And I need two queues, ProdQ and TestQ equally sharing Node1-4, and a
> “LabeledQ” with node8/9.
>
>
>
> When ProdQ is full, it can overflow to TestQ and further to LabeledQ. If
> TestQ is full, the tasks stay in TestQ, or optionally overflow to LabeledQ
> (either way
> is fine as long as it doesn’t go to ProdQ). And when JobA is running, it
> can only go to LabelledQ. If something else is on LabelledQ, JobA waits.
>
>
>
> Do you mind to illustrate how to config the queues to achieve what I am
> looking for?
>
>
>
> Thank you Sunil.
>
>
>
> *From:* Sunil Govind [mailto:sunil.govind@gmail.com]
>
>
> *Sent:* Tuesday, October 11, 2016 11:44 AM
>
>
>
> *To:* Frank Luo <jluo@merkleinc.com>; user@hadoop.apache.org
>
> *Subject:* Re: how to add a shareable node label?
>
>
>
>
> Hi Frank
>
>
>
>
>
>
> Extremely sorry for the delay..
>
>
>
>
>
>
>
> Yes, you are correct. Sharing feature of node label is not needed in your
> case.
>
>
>
> Existing node labels and a queue model could solve the problem.
>
>
>
>
>
>
>
> Thanks
>
>
>
> Sunil
>
>
>
>
>
>
>
> On Fri, Oct 7, 2016 at 11:59 PM Frank Luo <jluo@merkleinc.com> wrote:
>
>
>
>
>
> That is correct, Sunil.
>
>
>
> Just to confirm,  the Node Labeling feature on 2.8 or 3.0 alpha won’t
> satisfy
> my need, right?
>
>
>
> *From:*
> Sunil Govind [mailto:sunil.govind@gmail.com]
>
>
> *Sent:* Friday, October 07, 2016 12:09 PM
>
>
>
>
>
>
>
> *To:* Frank Luo <jluo@merkleinc.com>;
> user@hadoop.apache.org
>
> *Subject:* Re: how to add a shareable node label?
>
>
>
>
>
>
>
>
> HI Frank
>
>
>
>
>
>
> In that case, preemption may not be needed. So over-utilizing resources of
> queueB will be running till it completes. Since queueA is under served,
> then any next free container could
> go to queueA which is for Job_A.
>
>
>
>
>
>
>
> Thanks
>
>
>
> Sunil
>
>
>
>
>
>
>
> On Fri, Oct 7, 2016 at 9:58 PM Frank Luo <jluo@merkleinc.com> wrote:
>
>
>
>
>
> Sunil,
>
>
>
> Your description pretty much matches my understanding. Except for “Job_A
> will have to run as per its schedule w/o any delay”. My situation is that
> Job_A can be delayed. As long as it runs in queueA, I am happy.
>
>
>
> Just as you said, processes normally running in queueB might not be
> preemptable.
> So if they overflow to queueA then got preempted, then that is not good.
>
>
>
> *From:*
> Sunil Govind [mailto:sunil.govind@gmail.com]
>
>
> *Sent:* Friday, October 07, 2016 10:50 AM
>
>
>
>
>
>
>
> *To:* Frank Luo <jluo@merkleinc.com>;
> user@hadoop.apache.org
>
>
>
>
>
> *Subject:*
> Re: how to add a shareable node label?
>
>
>
>
>
>
>
>
>
> HI Frank
>
>
>
>
>
>
> Thanks for the details.
>
>
>
>
>
>
>
> I am not quite sure if I understood you problem correctly. I think you are
> looking for a solution to ensure that Job_A will have to run as per its
> schedule w/o any delay. Meantime
> you also do not want to waste resources on those high end machine where
> Job_A is running.
>
>
>
>
>
>
>
> I think you still need node label exclusivity here since there is h/w
> dependency. But if you have 2 queues' which are shared to use "labelA"
> here, then always "Job_A" can be planned
> to run in that queue, say "queueA". Other jobs could be run in "queueB"
> here. So if you tune capacities and if preemption is enabled per queue
> level, overutilized resources used by "queueB" could be preempted for
> "Job_A".
>
>
>
>
>
>
>
> But if your sharable jobs are like some linux jobs which should not be
> preempted, then this may be only a half solution.
>
>
>
>
>
>
>
> Thanks
>
>
>
>
>
> Sunil
>
>
>
>
>
>
>
> On Fri, Oct 7, 2016 at 7:36 AM Frank Luo <jluo@merkleinc.com> wrote:
>
>
>
>
>
> Sunil,
>
>
>
> You confirmed my understanding. I got the understanding by reading the docs
> and haven’t really tried 2.8 or 3.0-alphal.
>
>
>
> My situation is that I am in a multi-tenant env, and  got several very
> powerful
> machines with expensive licenses to run a particular linux job, let’s say
> Job_A. But the job is executed infrequently, so I want to let other jobs to
> use the machines when Job_A is not running. In the meaning time, I am not
> powerful enough to force all other
> jobs to be preemptable. As matter of fact, I know they have Hadoop jobs
> inserting into sql-server, or just pure linux jobs that are not preemptable
> in nature. So preempt jobs is not an option for me.
>
>
>
> I hope it makes sense.
>
>
>
> Frank
>
>
>
> *From:*
> Sunil Govind [mailto:sunil.govind@gmail.com]
>
>
> *Sent:* Thursday, October 06, 2016 2:15 PM
>
>
>
>
>
>
>
> *To:* Frank Luo <jluo@merkleinc.com>;
> user@hadoop.apache.org
>
> *Subject:* Re: how to add a shareable node label?
>
>
>
>
>
>
>
>
> HI Frank
>
>
>
>
>
>
> Ideally those containers will be preempted if there are unsatisfied demand
> for "configured label".
>
>
>
>
>
>
>
> I could explain this:
>
>
>
> "labelA" has few empty resources.  All nodes under "default" label is
> used. Hence a new application which is submitted to "default" label has to
> wait. But if "labelA" is non-exclusive
> and there are some free resources, this new application can run on
> "labelA".
>
>
>
> However if there are some more new apps submitted to "labelA", and if
> there are no more resources available in "labelA", then it may preempt
> containers from the app which was sharing
> containers earlier.
>
>
>
>
>
>
>
> May be you could share some more information so tht it may become more
> clear. Also I suppose you are running this in hadoop 3 alpha1 release.
> please correct me if I m wrong.
>
>
>
>
>
>
>
> Thanks
>
>
>
> Sunil
>
>
>
>
>
>
>
> On Thu, Oct 6, 2016 at 9:44 PM Frank Luo <jluo@merkleinc.com> wrote:
>
>
>
>
>
> Thanks Sunil.
>
>
>
>
> Ø
> 3. If there is any future ask for those resources , we will preempt the
> non labeled apps and give them back to labeled apps.
>
>
>
> Unfortunately, I am still not able to use it, because of the preemptive
> behavior.
> The jobs that steals labelled resources are not preemptable, and I’d
> rather waiting instead of killing.
>
>
>
> *From:*
> Sunil Govind [mailto:sunil.govind@gmail.com]
>
>
> *Sent:* Thursday, October 06, 2016 1:59 AM
>
>
>
>
>
>
>
> *To:* Frank Luo <jluo@merkleinc.com>;
> user@hadoop.apache.org
>
>
>
>
>
> *Subject:*
> Re: how to add a shareable node label?
>
>
>
>
>
>
>
>
> Hi Frank
>
>
> I think as of today this is not possible. You could try and experience the
> "non-exlusive" feature of node-label which will officially come in 2.8
> soon. Or you can try it in "Hadoop
> 3 alpha1" release too if its fine to check. YARN-3214
> <https://issues.apache.org/jira/browse/YARN-3214> has
> the details for the nodelabel sharing concept.
>
>
>
>
>
>
>
> Thanks
>
>
>
> Sunil
>
>
>
>
>
>
>
> On Wed, Oct 5, 2016 at 8:14 PM Frank Luo <jluo@merkleinc.com> wrote:
>
>
>
>
>
> Sunil, thanks for responding.
>
>
>
> So is there any way to dedicate one kind of jobs to certain machines, then
> having
> those machines be shared if no dedicated job running?
>
>
>
> *From:*
> Sunil Govind [mailto:sunil.govind@gmail.com]
>
>
> *Sent:* Wednesday, October 05, 2016 12:50 AM
>
> *To:* Frank Luo <jluo@merkleinc.com>;
> user@hadoop.apache.org;
> user@yarn.apache.org
>
>
>
>
>
>
>
> *Subject:* Re: how to add a shareable node label?
>
>
>
>
>
>
>
>
> Hi Frank,
>
>
>
>
>
>
>
>
>
>
>
>
> As far as I checked, all labels are "exclusive" in 2.7. In upcoming 2.8
> release, we can get "non-exclusive" or sharable node labels.
>
>
>
>
>
>
>
> Thanks
>
>
>
> Sunil
>
>
>
>
>
>
>
>
>
>
>
> On Wed, Oct 5, 2016 at 8:40 AM Frank Luo <jluo@merkleinc.com> wrote:
>
>
>
>
>
> I am using Hadoop 2.7.3, when I run:
>
>
> $ yarn rmadmin -addToClusterNodeLabels "Label1(exclusive=false)"
>
>
>
> I got an error as:
>
>
> … addToClusterNodeLabels: java.io.IOException: label name should only
> contains {0-9, a-z, A-Z, -, _} and should not started with {-,_}
>
>
>
> If I just use “Label1”, it will work fine, but I want a shareable one.
>
>
>
>
> Anyone knows a better way to do it?
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> *Access the Q2 2016 Digital Marketing Report for a fresh set of trends and
> benchmarks in digital marketing*
> <http://www2.merkleinc.com/l/47252/2016-07-26/47gt7c>
>
>
> *Download our latest report titled “The Case for Change: Exploring the
> Myths of Customer-Centric Transformation”*
> <http://www2.merkleinc.com/l/47252/2016-08-04/4b9p7c>
>
> This email and any attachments transmitted with it are intended for use by
> the intended recipient(s) only. If you have received this email in error,
> please notify the sender
> immediately and then delete it. If you are not the intended recipient, you
> must not keep, use, disclose, copy or distribute this email without the
> author’s prior permission. We take precautions to minimize the risk of
> transmitting software viruses, but we
> advise you to perform your own virus checks on any attachment to this
> message. We cannot accept liability for any loss or damage caused by
> software viruses. The information contained in this communication may be
> confidential and may be subject to the attorney-client
> privilege.
>
>

Mime
View raw message