hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Templeton (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-6125) The application attempt's diagnostic message should have a maximum size
Date Thu, 02 Feb 2017 22:04:51 GMT

    [ https://issues.apache.org/jira/browse/YARN-6125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15850577#comment-15850577
] 

Daniel Templeton commented on YARN-6125:
----------------------------------------

bq. the pom.xml dependency reordering was necessary in order to get ExpectedException working.

What is the net effect of declaring that no exception is expected?  Is it any different from
not declaring anything?

bq. Lists.newLinkedList() comes from Guava, which I like.

The issue isn't Guava.  The issue is that it was created to replace {{new LinkedList<Integer>()}}
(in this case), but as of Java 7 there's the diamond operator.  If you read the javadoc for
{{newList()}}, it says not to use it with Java 7 or later.

bq. I like final stuff (I know I'm weird), please specify where I should make variables /
fields non-final

I was thinking specifically of {code}    final int inputLength = csq.length();{code}  {{inputLength}}
is only ever used as a parameter, so declaring it final is kinda overkill.

While you're working on the patch, another architectural concern has occurred to me.  Assume
I have a 64k limit.  I append a message that's 65,530 bytes long.  I then append a message
that's 10 bytes long.  The current implementation will drop the first message completely.
 Seems like it might be better to only drop the first line of the first message,  just enough
to trim it down.  A partial message may be better than no message.  Granted, it's probably
not hugely useful to only have the tail of a message, but from a user's perspective it's less
unexpected to get a partial message than to drop a large message entirely.  One could make
the same case for just cutting it down to the exact length instead of minding line boundaries...
 [~jlowe], [~kasha], [~vvasudev], [~yufeigu], any thoughts?

Oh, one more thing...  Can we add a header that shows the content was trimmed?  If we're not
just clipping to the buffer size, it would be useful to have a way to notify the user that
the message is not complete.  In most cases it will be, so we want to be clear when it isn't.

> The application attempt's diagnostic message should have a maximum size
> -----------------------------------------------------------------------
>
>                 Key: YARN-6125
>                 URL: https://issues.apache.org/jira/browse/YARN-6125
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: resourcemanager
>    Affects Versions: 2.7.0
>            Reporter: Daniel Templeton
>            Assignee: Andras Piros
>            Priority: Critical
>             Fix For: 3.0.0-alpha3
>
>         Attachments: YARN-6125.000.patch, YARN-6125.001.patch, YARN-6125.002.patch
>
>
> We've found through experience that the diagnostic message can grow unbounded.  I've
seen attempts that have diagnostic messages over 1MB.  Since the message is stored in the
state store, it's a bad idea to allow the message to grow unbounded.  Instead, there should
be a property that sets a maximum size on the message.
> I suspect that some of the ZK state store issues we've seen in the past were due to the
size of the diagnostic messages and not to the size of the classpath, as is the current prevailing
opinion.
> An open question is how best to prune the message once it grows too large.  Should we
> # truncate the tail,
> # truncate the head,
> # truncate the middle,
> # add another property to make the behavior selectable, or
> # none of the above?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message