Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Thu, 2 Feb 2017 23:35:51 +0000 (UTC)
From: "Andras Piros (JIRA)" <jira@apache.org>
To: yarn-issues@hadoop.apache.org
Message-ID: <JIRA.13038215.1485454102000.21328.1486078551678@Atlassian.JIRA>
In-Reply-To: <JIRA.13038215.1485454102000@Atlassian.JIRA>
References: <JIRA.13038215.1485454102000@Atlassian.JIRA> <JIRA.13038215.1485454102108@jira-lw-us.apache.org>
Subject: [jira] [Comment Edited] (YARN-6125) The application attempt's
 diagnostic message should have a maximum size
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Thu, 02 Feb 2017 23:36:03 -0000


    [ https://issues.apache.org/jira/browse/YARN-6125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15850732#comment-15850732 ] 

Andras Piros edited comment on YARN-6125 at 2/2/17 11:35 PM:
-------------------------------------------------------------

* {{ExpectedException.none()}} as factory method, and {{ExpectedException}} general usage description can be found [*here*|https://github.com/junit-team/junit4/wiki/exception-testing#expectedexception-rule]. Basically if you don't expect any {{Exception}} s to be thrown, you don't call {{expect()}}, {{expectMessage()}} or {{expectCause()}}. Yes, you can always test exceptional behavior w/ [*try-catch idiom*|https://github.com/junit-team/junit4/wiki/exception-testing#trycatch-idiom] instead of {{ExpectedException}}
* got rid of {{Lists.newLinkedList()}} call
* got rid of that {{final}} variable

Posting the patch addressing those issues but not yet the truncate behavior.

Talked to [~templedf] and [~yufeigu] offline about truncate characteristics, here is one possible solution:
# when a message is being appended, and it doesn't fit into the buffer, trim message body to buffer size (minus header prefixes) preserving its head and add a prefix header telling users it's been partially truncated
# when a message is being appended, and it fits into the buffer:
## past messages are deleted in FIFO order
## for all the ones deleted we put a common prefix header stating there have been messages left out
## for the last one truncated we trim the message body and add a prefix header stating part of its tail has been left out

Some example scenarios, considering four consecutive {{append()}} calls w/ different messages each:
# nothing truncated:
{noformat}
message1 this is a very long message that fits into the buffer
message2 this is a lot shorter than the previour one
message3 this is even shorter
message4 the shortest one
{noformat}
# only the very last message has been partially truncated:
{noformat}
message1 this is a very long message that fits into the buffer
message2 this is a lot shorter than the previour one
message3 this is even shorter
<MESSAGE TAIL TRUNCATED>message4 this has been one that did not fit totally into the buffer...
{noformat}
# only the very first message has been partially truncated:
{noformat}
<MESSAGE TAIL TRUNCATED>message1 this has been one that did not fit totally into the buffer...
message2 this is a very long message that fits into the buffer
message3 this is a lot shorter than the previour one
message4 the shortest one
{noformat}
# the first several ones have been deleted, the previous one has been truncated partially:
{noformat}
<PREVIOUS MESSAGES DELETED>
<MESSAGE TAIL TRUNCATED>message3 this has been one that did not fit totally into the buffer...
message4 the shortest one
{noformat}
# all the previous ones have been deleted, plus the last one has been truncated:
{noformat}
<PREVIOUS MESSAGES DELETED>
<MESSAGE TAIL TRUNCATED>message4 this has been one that did not fit totally into the buffer...
{noformat}

[~jlowe], [~kasha], [~vvasudev] what is your opinion about the proposal?


was (Author: andras.piros):
* {{ExpectedException.none()}} as factory method, and {{ExpectedException}} general usage description can be found [*here*|https://github.com/junit-team/junit4/wiki/exception-testing#expectedexception-rule]. Basically if you don't expect any {{Exception}} s to be thrown, you don't call {{expect()}}, {{expectMessage()}} or {{expectCause()}}. Yes, you can always test exceptional behavior w/ [*try-catch idiom*|https://github.com/junit-team/junit4/wiki/exception-testing#trycatch-idiom] instead of {{ExpectedException}}
* got rid of {{Lists.newLinkedList()}} call
* got rid of that {{final}} variable

Posting the patch addressing those issues but not yet the truncate behavior.

Talked to [~templedf] and [~yufeigu] offline about truncate characteristics, here is one possible solution:
# when a message is being appended, and it doesn't fit into the buffer, trim message body to buffer size (minus header prefixes) preserving its head and add a prefix header telling users it's been partially truncated
# when a message is being appended, and it fits into the buffer:
## past messages are truncated in FIFO order
## for all the ones truncated fully we put a common prefix header stating there have been messages left out
## for the last one truncated partially we trim the message body and add a prefix header stating it's been partially truncated

Some example scenarios:
# nothing truncated:
{noformat}
message1 this is a very long message that fits into the buffer
message2 this is a lot shorter than the previour one
message3 the shortest one
{noformat}
# only the very last message has been partially truncated:
{noformat}
message1 this is a very long message that fits into the buffer
message2 this is a lot shorter than the previour one
message3 the shortest one
<MESSAGE TAIL TRUNCATED>message4 this has been one that did not fit totally into the buffer...
{noformat}
# only the very first message has been partially truncated:
{noformat}
<MESSAGE TAIL TRUNCATED>message1 this has been one that did not fit totally into the buffer...
message2 this is a very long message that fits into the buffer
message3 this is a lot shorter than the previour one
message4 the shortest one
{noformat}
# the first several ones have been deleted, the previous one has been truncated partially:
{noformat}
<PREVIOUS MESSAGES DELETED>
<MESSAGE TAIL TRUNCATED>message3 this has been one that did not fit totally into the buffer...
message4 the shortest one
{noformat}
# all the previous ones have been deleted, plus the last one has been truncated:
{noformat}
<PREVIOUS MESSAGES DELETED>
<MESSAGE TAIL TRUNCATED>message4 this has been one that did not fit totally into the buffer...
{noformat}

[~jlowe], [~kasha], [~vvasudev] what is your opinion about the proposal?

> The application attempt's diagnostic message should have a maximum size
> -----------------------------------------------------------------------
>
>                 Key: YARN-6125
>                 URL: https://issues.apache.org/jira/browse/YARN-6125
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: resourcemanager
>    Affects Versions: 2.7.0
>            Reporter: Daniel Templeton
>            Assignee: Andras Piros
>            Priority: Critical
>             Fix For: 3.0.0-alpha3
>
>         Attachments: YARN-6125.000.patch, YARN-6125.001.patch, YARN-6125.002.patch
>
>
> We've found through experience that the diagnostic message can grow unbounded.  I've seen attempts that have diagnostic messages over 1MB.  Since the message is stored in the state store, it's a bad idea to allow the message to grow unbounded.  Instead, there should be a property that sets a maximum size on the message.
> I suspect that some of the ZK state store issues we've seen in the past were due to the size of the diagnostic messages and not to the size of the classpath, as is the current prevailing opinion.
> An open question is how best to prune the message once it grows too large.  Should we
> # truncate the tail,
> # truncate the head,
> # truncate the middle,
> # add another property to make the behavior selectable, or
> # none of the above?


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org