hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Haibo Chen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5627) [Atsv2] Support streaming reader API to fetch entities
Date Tue, 13 Mar 2018 04:14:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-5627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16396520#comment-16396520
] 

Haibo Chen commented on YARN-5627:
----------------------------------

Yes, max N entities at a time is what MR needs. I was indeed able to retrieve only relevant
fields to reduce the data transfer. But filters are not applicable when all tasks or task
attempts are to be retrieved. The data size as a result will still be big for big jobs. Server
side pagination will reduce the one time data transfer to multiple requests that are only
made whenever necessary.

> [Atsv2] Support streaming reader API to fetch entities
> ------------------------------------------------------
>
>                 Key: YARN-5627
>                 URL: https://issues.apache.org/jira/browse/YARN-5627
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelinereader
>            Reporter: Rohith Sharma K S
>            Assignee: Rohith Sharma K S
>            Priority: Major
>
> There is no limit for size of TimelineEntitie object. It can be varied from KB's to MB.
While reading entities list, it would be an potential issue that TimeLineReder would go into
OOM situation based on the entity size and limit. 
> Proposal is to support an streaming API to read entities list. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message