hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6875) New aggregated log file format for YARN log aggregation.
Date Thu, 27 Jul 2017 15:30:01 GMT

    [ https://issues.apache.org/jira/browse/YARN-6875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16103346#comment-16103346

Jason Lowe commented on YARN-6875:

Thanks for posting the doc!  I'm not a big fan of having a separate file, even temporarily,
because log aggregation can already be a large portion of the namenode's write load on large
clusters.  Having that separate file will increase the namenode write load significantly (approximately
2x per log aggregation cycle if I understand it correctly).

Note that the separate index file doesn't solve all the race conditions for the reader.  For
example, this sequence:
# Reader checks for an index file which is not there
# Writer begins append and creates index file and starts appending
# Reader seeks to the end of the log file but does _not_ find the metainfo structure because
the writer is in the process of appending more data

This could be mitigated by having the reader repeat the attempt to read process from the beginning
so it can rediscover the index file, but this requires that the reader is capable of recognizing
that it is _not_ looking at a proper metainfo block on that first attempt.  The document does
not cover this necessary rinse-repeat cycle required on the reader's part, nor how a reader
can reliably identify the case where it is not looking at a proper metainfo block because
it happened to try to read just as an append operation occurs.

I'm wondering if we can eliminate the need for the index file, and thus reduce the write load
on the namenode, by having the reader be able to discover the metainfo file even during an
append operation.  Similar to sync markers in SequenceFile, we could create a unique, UUID-like
sync marker that is written out before every metainfo block.  The reader would attempt to
find the metainfo block normally (i.e.: seek to the last 64 bits of the file, read the 64-bit
offset, then seek back that far to check for a metainfo block).  If it finds it then great,
the reader is ready to read whatever it is looking for.  If it does not find a proper metainfo
file then it can start scanning backwards through the file looking for a metainfo sync marker.
 This scan could be accomplished via a number of ways, such as sequentially scanning backwards
block at a time in fixed-size blocks or seeking much farther backwards in a larger chunk that
is scanned forward in fixed-sized chunks then repeating if the marker is not found.

Isn't this a lot slower for the reader when it has to scan for the marker?  Yep, it sure is.
 However I would argue this is probably a rare occurrence in practice for two reasons:
# Logs are often written and never read
# Appending is a relatively rare and short-lived operation during the lifespan of a log file

By having the writer create the index file, we're essentially optimizing for this rare read-during-append
case at the expense of making every writer more expensive.  Instead the sync marker approach
optimizes for the much more common writing case, putting the load on the reader side if it
happens to encounter a log file mid-append during a read operation.  I would argue that should
be a relatively rare occurrence, and thus I'd rather optimize for the more common case.

Another alternative to the index file is using xattrs to associate the last good metainfo
offset with the file.  However that still leads to approximately the same namenode write ops
as the separate index file and requires special support on the underlying filesystem.  I'm
not a fan of using xattrs myself, but I thought I'd mention it in the interest of covering
the potential solutions.

> New aggregated log file format for YARN log aggregation.
> --------------------------------------------------------
>                 Key: YARN-6875
>                 URL: https://issues.apache.org/jira/browse/YARN-6875
>             Project: Hadoop YARN
>          Issue Type: New Feature
>            Reporter: Xuan Gong
>            Assignee: Xuan Gong
>         Attachments: YARN-6875-NewLogAggregationFormat-design-doc.pdf
> T-file is the underlying log format for the aggregated logs in YARN. We have seen several
performance issues, especially for very large log files.
> We will introduce a new log format which have better performance for large log files.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message