reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Julia (JIRA)" <>
Subject [jira] [Created] (REEF-1641) How to decide which one is maser revalautor
Date Thu, 13 Oct 2016 23:23:20 GMT
Julia created REEF-1641:

             Summary: How to decide which one is maser revalautor 
                 Key: REEF-1641
             Project: REEF
          Issue Type: Improvement
            Reporter: Julia

Currently in IMRU, we have different spec for master vs mapper elevators. We use EvaluatorBatchId
to distinguish which one is what we requested when receiving IAllocatedEvalauator. However,
EvaluatorBatchId  may not be supported in some environment such as HDI. So we need a way to
decide which evaluator is for master. 

There are multiple options:
a). Match the specification. The issue is when we ask for evaluators with specific cores and
memory, we may not get the exact same specification but some rounded off resources. 

b). Use the same spec for both master and mappers. Choose the first one for master if master
doesn't exist. The issue is if mappers are more memory heavy as compared to update func. Allocating
some extra memory for one update evaluator is ok. But in scenarios where its other way around,
giving all mappers extra memory can be a huge waste.

c). Request the first evaluator for master. After receiving it, record that one as master,
then request the mappers. The issue is what if master fails during WaitingForEvalautor phase?
A solution is assuming if master fails at any stage, fail the system. 

This message was sent by Atlassian JIRA

View raw message