hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Varun Saxena (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5792) adopt the id prefix for YARN, MR, and DS entities
Date Fri, 11 Nov 2016 09:14:59 GMT

    [ https://issues.apache.org/jira/browse/YARN-5792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15656583#comment-15656583

Varun Saxena commented on YARN-5792:

bq. At a high level, I'd like to discuss options. I see that you're mostly using the (inverted)
start time as the id prefix. Would it be better to use simply the id instead whenever possible?
I wanted to discuss the same as well. That is why I mixed both approaches in first patch.
I had used Id bit of job id to have id prefix for jobs and attempt created time for attempts
despite it being possible that attempt id can be used.
Anyways in latest patch for YARN and DS attempts I now use  the id bit of attempt id in addition
to id bit of job  id.

Coming to task IDs', the ID bit of task is cannot really be used because that's within the
scope of task type (map/reduce). Whether task is map or reduce can be stored in one bit and
then we can shift around id. Frankly this should be fine as one bit of long will be used as
sign bit and we do not expect ids' to be negative. But should we do it. Some mappers may be
launched after reducers are launched so with the approach mentioned above, we may not get
that order. So I would continue going with current approach.
Task attempts are anyways one level below tasks.

Talking about containers, id bit of container ID is a long. And containers are within scope
of an app attempt (id bit of attempt is int). Although number of attempts are never going
to be a very large and id bit of container ID not reach near the value of long (i.e. not as
many containers are likely to be launched) so we can still adjust both in 8 bytes by doing
some bit shifting. But then we will be making assumptions which IMO we should not do so its
safer to go with container start time. Thoughts ?

> adopt the id prefix for YARN, MR, and DS entities
> -------------------------------------------------
>                 Key: YARN-5792
>                 URL: https://issues.apache.org/jira/browse/YARN-5792
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>    Affects Versions: YARN-5355
>            Reporter: Sangjin Lee
>            Assignee: Varun Saxena
>         Attachments: YARN-5792-YARN-5355.01.patch, YARN-5792-YARN-5355.02.patch, YARN-5792-YARN-5355.03.patch
> We introduced the entity id prefix to support flexible entity sorting (YARN-5715). We
should adopt the id prefix for YARN entities, MR entities, and DS entities to take advantage
of the id prefix.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message