hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rohith Sharma K S (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5585) [Atsv2] Reader side changes for entity prefix and support for pagination via additional filters
Date Tue, 20 Dec 2016 07:01:58 GMT

    [ https://issues.apache.org/jira/browse/YARN-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15763460#comment-15763460

Rohith Sharma K S commented on YARN-5585:

bq. Another option could be that if more than one row is encountered for a single entity read,
we send some sort of error message indicating multiple idprefixes in backend which can alert
the user/application of some issue on the write side.
bq. the storage may throw exceptions or return errors when multiple prefixes are found for
the same entity.
It make sense to me. Indicating user with exception is better so that user would look into
their code and fix it. 

For Entities, I think we can support below filters
# fromId="idPrefix:entityId" where scan will start from this row key. 
# fromId="idPrefix"  where scan will start from this row key.
# But for supporting "*:entityId",  the backend it is 2 queries which need to make where in
first query is for finding prefix which need to do range scan, and secondly again range scan
after finding entityPrefixId. And IMO, if we support only entityId then user always use entityId
fromId. The entityIdPrefix will become unusable.

For Entity, filter can be only idPrefix.
# Filter name would be *idPrefix* as-is. If this filter is passed then use Get API else, range
scan with SingleColumnValueFilter for matching entityId. 

> [Atsv2] Reader side changes for entity prefix and support for pagination via additional
> -----------------------------------------------------------------------------------------------
>                 Key: YARN-5585
>                 URL: https://issues.apache.org/jira/browse/YARN-5585
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelinereader
>            Reporter: Rohith Sharma K S
>            Assignee: Rohith Sharma K S
>            Priority: Critical
>              Labels: yarn-5355-merge-blocker
>         Attachments: 0001-YARN-5585.patch, YARN-5585-YARN-5355.0001.patch, YARN-5585-YARN-5355.0002.patch,
YARN-5585-workaround.patch, YARN-5585.v0.patch
> TimelineReader REST API's provides lot of filters to retrieve the applications. Along
with those, it would be good to add new filter i.e fromId so that entities can be retrieved
after the fromId. 
> Current Behavior : Default limit is set to 100. If there are 1000 entities then REST
call gives first/last 100 entities. How to retrieve next set of 100 entities i.e 101 to 200
OR 900 to 801?
> Example : If applications are stored database, app-1 app-2 ... app-10.
> *getApps?limit=5* gives app-1 to app-5. But to retrieve next 5 apps, there is no way
to achieve this. 
> So proposal is to have fromId in the filter like *getApps?limit=5&&fromId=app-5*
which gives list of apps from app-6 to app-10. 
> Since ATS is targeting large number of entities storage, it is very common use case to
get next set of entities using fromId rather than querying all the entites. This is very useful
for pagination in web UI.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message