hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tao Yang (JIRA)" <j...@apache.org>
Subject [jira] [Created] (YARN-6259) Support pagination and optimize data transfer with zero-copy approach for containerlogs REST API in NMWebServices
Date Wed, 01 Mar 2017 11:11:45 GMT
Tao Yang created YARN-6259:
------------------------------

             Summary: Support pagination and optimize data transfer with zero-copy approach
for containerlogs REST API in NMWebServices
                 Key: YARN-6259
                 URL: https://issues.apache.org/jira/browse/YARN-6259
             Project: Hadoop YARN
          Issue Type: Improvement
          Components: nodemanager
    Affects Versions: 2.8.1
            Reporter: Tao Yang
            Assignee: Tao Yang


Currently containerlogs REST API in NMWebServices will read and send the entire content of
container logs. Most of container logs are large and it's useful to support pagination.
* Add pagesize and pageindex parameters for containerlogs REST API
{code}
URL: http://<nm_address>/ws/v1/node/containerlogs/<container_id>/<file_name>
QueryParams:
  pagesize - max bytes of one page , default 1MB
  pageindex - index of required page, default 0, can be nagative(set -1 will get the last
page content)
{code}
* Add containerlogs-info REST API since sometimes we need to know the totalSize/pageSize/pageCount
info of log 
{code}
URL: http://<nm_address>/ws/v1/node/containerlogs-info/<container_id>/<file_name>
QueryParams:
  pagesize - max bytes of one page , default 1MB
Response example:
  {"logInfo":{"totalSize":2497280,"pageSize":1048576,"pageCount":3}}
{code}

Moreover, the data transfer pipeline (disk --> read buffer --> NM buffer --> socket
buffer) can be optimized to pipeline(disk --> read buffer --> socket buffer) with zero-copy
approach.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message