hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tao Yang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6259) Support pagination and optimize data transfer with zero-copy approach for containerlogs REST API in NMWebServices
Date Thu, 02 Mar 2017 01:32:45 GMT

    [ https://issues.apache.org/jira/browse/YARN-6259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15891423#comment-15891423
] 

Tao Yang commented on YARN-6259:
--------------------------------

Hi, [~rohithsharma]. Thank you for looking into this issue.
{quote}
I am not sure about how use cases will be served
{quote}
One common use case is to request last part of log and easily skip to another part for detecting
problem, instead of loading the entire log, it perhaps can save a lot of time. We have an
outer system to track apps and show container logs inside, meanwhile most of logs are very
large, so that pagination function is needed and the newly added containerlogs-info REST API
is a part of it.

{quote}
Instead of adding new LogInfo file, there is ContainerLogInfo file which can be used for pageSize
and pageIndex.
{quote}
ContainerLogInfo seems not exist in branch-2.8, perhaps it's for higher version?

> Support pagination and optimize data transfer with zero-copy approach for containerlogs
REST API in NMWebServices
> -----------------------------------------------------------------------------------------------------------------
>
>                 Key: YARN-6259
>                 URL: https://issues.apache.org/jira/browse/YARN-6259
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: nodemanager
>    Affects Versions: 2.8.1
>            Reporter: Tao Yang
>            Assignee: Tao Yang
>         Attachments: YARN-6259.001.patch
>
>
> Currently containerlogs REST API in NMWebServices will read and send the entire content
of container logs. Most of container logs are large and it's useful to support pagination.
> * Add pagesize and pageindex parameters for containerlogs REST API
> {code}
> URL: http://<nm_address>/ws/v1/node/containerlogs/<container_id>/<file_name>
> QueryParams:
>   pagesize - max bytes of one page , default 1MB
>   pageindex - index of required page, default 0, can be nagative(set -1 will get the
last page content)
> {code}
> * Add containerlogs-info REST API since sometimes we need to know the totalSize/pageSize/pageCount
info of log 
> {code}
> URL: http://<nm_address>/ws/v1/node/containerlogs-info/<container_id>/<file_name>
> QueryParams:
>   pagesize - max bytes of one page , default 1MB
> Response example:
>   {"logInfo":{"totalSize":2497280,"pageSize":1048576,"pageCount":3}}
> {code}
> Moreover, the data transfer pipeline (disk --> read buffer --> NM buffer -->
socket buffer) can be optimized to pipeline(disk --> read buffer --> socket buffer)
with zero-copy approach.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message