hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Varun Saxena (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (YARN-5585) [Atsv2] Add a new filter fromId in REST endpoints
Date Tue, 04 Oct 2016 16:06:21 GMT

    [ https://issues.apache.org/jira/browse/YARN-5585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15545818#comment-15545818
] 

Varun Saxena edited comment on YARN-5585 at 10/4/16 4:06 PM:
-------------------------------------------------------------

Thanks [~rohithsharma] for the patch. Few comments.

# Intention behind having ID_PREFIX in EntityColumn ? According to me, we need not store prefix
in the column. Is it because we want to read it back and send it to client ?
# No need of GenericEntityReader#calculateTheClosestNextRowKeyForPrefix. Scan#setRowPrefixFilter
will do it for you. We should call it the same way as was done previously.
# As entity ID prefix is a long, EntityRowKeyConverter#SEGMENT_SIZES should have new segment
as Bytes.SIZEOF_LONG. It is currently given as VARIABLE_SIZE. Same change in TestRowKeys.
# In EntityRowKeyConverter#encode, no need to invert entity id prefix. We will take prefix
as-is. Sender can publish the entity with inverted prefix if he wants contents in descending
order (say). We can probably add something to TimelineUtils to invert it, if required, which
then clients can use.
# In GenericEntityReader#parseEntity we should fetch id prefix from result set and setIdPrefix
in TimelineEntity to be returned back. This will be useful for clients when they want to set
fromPrefix (will be useful in Tez UI use case).
# Javadoc in TimelineReader should be changed. It currently says entities would be sorted
by created time which is no longer true.
{code}
   * @return A set of <cite>TimelineEntity</cite> instances of the given entity
   *    type in the given context scope which matches the given predicates
   *    ordered by created time, descending. Each entity will only contain the
   *    metadata(id, type and created time) plus the given fields to retrieve.
{code}
# We should also update documentation to reflect id prefix.





was (Author: varun_saxena):
Thanks [~rohithsharma] for the patch. Few comments.

# Intention behind having ID_PREFIX in EntityColumn ? According to me, we need not store prefix
in the column. Is it because we want to read it back and send it to client ?
# No need of GenericEntityReader#calculateTheClosestNextRowKeyForPrefix. Scan#setRowPrefixFilter
will do it for you. We should call it the same way as was done previously.
# As entity ID prefix is a long, EntityRowKeyConverter#SEGMENT_SIZES should have new segment
as Bytes.SIZEOF_LONG. Same change in TestRowKeys.
# In EntityRowKeyConverter#encode, no need to invert entity id prefix. We will take prefix
as-is. Sender can publish the entity with inverted prefix if he wants contents in descending
order (say). We can probably add something to TimelineUtils to invert it, if required, which
then clients can use.
# In GenericEntityReader#parseEntity we should fetch id prefix from result set and setIdPrefix
in TimelineEntity to be returned back. This will be useful for clients when they want to set
fromPrefix (will be useful in Tez UI use case).
# Javadoc in TimelineReader should be changed. It currently says entities would be sorted
by created time which is no longer true.
{code}
   * @return A set of <cite>TimelineEntity</cite> instances of the given entity
   *    type in the given context scope which matches the given predicates
   *    ordered by created time, descending. Each entity will only contain the
   *    metadata(id, type and created time) plus the given fields to retrieve.
{code}
# We should also update documentation to reflect id prefix.




> [Atsv2] Add a new filter fromId in REST endpoints
> -------------------------------------------------
>
>                 Key: YARN-5585
>                 URL: https://issues.apache.org/jira/browse/YARN-5585
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelinereader
>            Reporter: Rohith Sharma K S
>            Assignee: Rohith Sharma K S
>            Priority: Critical
>         Attachments: 0001-YARN-5585.patch, YARN-5585-workaround.patch, YARN-5585.v0.patch
>
>
> TimelineReader REST API's provides lot of filters to retrieve the applications. Along
with those, it would be good to add new filter i.e fromId so that entities can be retrieved
after the fromId. 
> Current Behavior : Default limit is set to 100. If there are 1000 entities then REST
call gives first/last 100 entities. How to retrieve next set of 100 entities i.e 101 to 200
OR 900 to 801?
> Example : If applications are stored database, app-1 app-2 ... app-10.
> *getApps?limit=5* gives app-1 to app-5. But to retrieve next 5 apps, there is no way
to achieve this. 
> So proposal is to have fromId in the filter like *getApps?limit=5&&fromId=app-5*
which gives list of apps from app-6 to app-10. 
> Since ATS is targeting large number of entities storage, it is very common use case to
get next set of entities using fromId rather than querying all the entites. This is very useful
for pagination in web UI.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message