hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhijie Shen (JIRA)" <j...@apache.org>
Subject [jira] [Created] (YARN-2733) Aggregated log file may be corrupted if IOException happens at copying
Date Thu, 23 Oct 2014 05:25:34 GMT
Zhijie Shen created YARN-2733:
---------------------------------

             Summary: Aggregated log file may be corrupted if IOException happens at copying
                 Key: YARN-2733
                 URL: https://issues.apache.org/jira/browse/YARN-2733
             Project: Hadoop YARN
          Issue Type: Improvement
          Components: log-aggregation
            Reporter: Zhijie Shen


In LogValue.write:
{code}
          while ((len = in.read(buf)) != -1) {
            //If buffer contents within fileLength, write
            if (len < bytesLeft) {
              out.write(buf, 0, len);
              bytesLeft-=len;
            }
            //else only write contents within fileLength, then exit early
            else {
              out.write(buf, 0, (int)bytesLeft);
              break;
            }
          }
{code}

If IOException happens at any iteration, copying is going to be stopped, and exception message
will be appended to the end. It is very likely that the total copied content length is smaller
than the file length which has been written into the aggregated file in advance. Later on,
when the reader uses this length to parse the content, error is going to happen because of
reading more bytes than actually being written.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message