hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sangjin Lee (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5585) [Atsv2] Reader side changes for entity prefix and support for pagination via additional filters
Date Wed, 21 Dec 2016 21:56:58 GMT

    [ https://issues.apache.org/jira/browse/YARN-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15768283#comment-15768283
] 

Sangjin Lee commented on YARN-5585:
-----------------------------------

OK, went over the patch once just now. First off, I can also reproduce the test failure:
{noformat}
Running org.apache.hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage
Tests run: 26, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 35.248 sec <<< FAILURE!
- in org.apache.hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage
testUIDQueryWithAndWithoutFlowContextInfo(org.apache.hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage)
 Time elapsed: 0.453 sec  <<< FAILURE!
java.lang.AssertionError: null
     at org.junit.Assert.fail(Assert.java:86)
     at org.junit.Assert.assertTrue(Assert.java:41)
     at org.junit.Assert.assertTrue(Assert.java:52)
     at org.apache.hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage.testUIDQueryWithAndWithoutFlowContextInfo(TestTimelineReaderWebServicesHBaseStorage.java:886)
{noformat}

(TimelineReaderWebServices.java)
- l.317: super-nit: let's use the java style if: {{if (split != null)}}
- l.333-335: I don't think we should set the info from the fromId to entity id prefix and
entity id. The entity id prefix and the entity id should be used for a true single-entity
query context. It would be confusing to "reuse" them to indicate the fromId. I would prefer
an explicit fromId fields in the context so it's crystal clear what they are.

(GenericEntityReader.java)
- l.442-463: currently it's doing a column value filter; would it be better to use stop and
start rows?
- l.473-502: as mentioned above, let's be explicit about the fromId

(TimelineReaderContext.java)
- see above; I would prefer not to mix real entity id prefix for single-entity queries and
entity id prefix + entity id for fromId for multi-entity queries

Finally, I know it's no longer directly used, but I think {{TimelineEntity.compareTo()}} needs
updating. It does not use the entity id prefix at all, and it's using the creation time which
is not very consistent with what we're doing. Can we update that method as part of this JIRA?
Thanks!

> [Atsv2] Reader side changes for entity prefix and support for pagination via additional
filters
> -----------------------------------------------------------------------------------------------
>
>                 Key: YARN-5585
>                 URL: https://issues.apache.org/jira/browse/YARN-5585
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelinereader
>            Reporter: Rohith Sharma K S
>            Assignee: Rohith Sharma K S
>            Priority: Critical
>              Labels: yarn-5355-merge-blocker
>         Attachments: 0001-YARN-5585.patch, YARN-5585-YARN-5355.0001.patch, YARN-5585-YARN-5355.0002.patch,
YARN-5585-YARN-5355.0003.patch, YARN-5585-workaround.patch, YARN-5585.v0.patch
>
>
> TimelineReader REST API's provides lot of filters to retrieve the applications. Along
with those, it would be good to add new filter i.e fromId so that entities can be retrieved
after the fromId. 
> Current Behavior : Default limit is set to 100. If there are 1000 entities then REST
call gives first/last 100 entities. How to retrieve next set of 100 entities i.e 101 to 200
OR 900 to 801?
> Example : If applications are stored database, app-1 app-2 ... app-10.
> *getApps?limit=5* gives app-1 to app-5. But to retrieve next 5 apps, there is no way
to achieve this. 
> So proposal is to have fromId in the filter like *getApps?limit=5&&fromId=app-5*
which gives list of apps from app-6 to app-10. 
> Since ATS is targeting large number of entities storage, it is very common use case to
get next set of entities using fromId rather than querying all the entites. This is very useful
for pagination in web UI.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message