hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joep Rottinghuis (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5269) Bubble exceptions and errors all the way up the calls, including to clients.
Date Fri, 10 Mar 2017 01:08:38 GMT

    [ https://issues.apache.org/jira/browse/YARN-5269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15904194#comment-15904194
] 

Joep Rottinghuis commented on YARN-5269:
----------------------------------------

After looking at the code I am indeed a little surprised that we're not properly doing this.
All the plumbing seems correct on the client side:
TimelineV2ClientImpl#putEntities vs TimelineV2ClientImpl#putEntitiesAsync correctly call TimelineEntityDispatcher#dispatchEntities(boolean
sync,... with the correct argument. This argument does seem to make it into the params, and
on the server side TimelineCollectorWebService#putEntities correctly pulls the async parameter
from the rest call. See line 156:
{code}
    boolean isAsync = async != null && async.trim().equalsIgnoreCase("true");
{code}
However, this is where the problem starts. It simply calls TimelineCollector#putEntities and
ignores the value of isAsync. It should instead have called TimelineCollector#putEntitiesAsync,
which is currently not implemented.
putEntities should call putEntitiesAsync and then after that call writer.flush()

The fact that we flush on close and we flush periodically should be more of a concern of avoiding
data loss; close in case sync is never called and the periodic flush to guard against having
data from slow writers get buffered for a long time and expose us to risk of loss in case
the collector crashes with data in its buffers. Size-based flush is a different concern to
avoid blowing up memory footprint.

The spooling behavior is also somewhat separate.

We have two separate methods on our API putEntities and putEntitiesAsync and they should have
different behavior beyond waiting for the request to be sent. I can file a separate bug from
this one dealing with exception handling to tackle the sync vs async nature. During the meeting
today I was thinking about the HBase writer that has a flush, which definitely blocks until
data is flushed to HBase (ignoring the spooling for the moment).

> Bubble exceptions and errors all the way up the calls, including to clients.
> ----------------------------------------------------------------------------
>
>                 Key: YARN-5269
>                 URL: https://issues.apache.org/jira/browse/YARN-5269
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>    Affects Versions: YARN-2928
>            Reporter: Joep Rottinghuis
>            Assignee: Haibo Chen
>              Labels: YARN-5355, yarn-5355-merge-blocker
>
> Currently we ignore (swallow) exception from the HBase side in many cases (reads and
writes).
> Also, on the client side, neither TimelineClient#putEntities (the v2 flavor) nor the
#putEntitiesAsync method return any value.
> For the second drop we may want to consider how we properly bubble up exceptions throughout
the write and reader call paths and if we want to return a response in putEntities and some
future kind of result for putEntitiesAsync.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message