tez-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kuhu Shukla (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (TEZ-3831) Reduce Unordered memory needed for storing empty completed events
Date Fri, 15 Sep 2017 19:40:00 GMT

    [ https://issues.apache.org/jira/browse/TEZ-3831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16168408#comment-16168408

Kuhu Shukla commented on TEZ-3831:

Thanks [~jeagles] for the additional patch. Using NullFetchedInput looks fine to me. The adjust
call will add this input and that would cover cases of all empty partitions and some empty
+1. Will commit this shortly.

> Reduce Unordered memory needed for storing empty completed events
> -----------------------------------------------------------------
>                 Key: TEZ-3831
>                 URL: https://issues.apache.org/jira/browse/TEZ-3831
>             Project: Apache Tez
>          Issue Type: Bug
>            Reporter: Jonathan Eagles
>            Assignee: Jonathan Eagles
>             Fix For: 0.9.1
>         Attachments: Screen Shot 2017-09-13 at 4.55.11 PM.png, TEZ-3831.001-addendum.patch,
> the completedInputs blocking queue is used to store inputs for the UnorderedKVReader
to consume. With Auto-reduce parallelism enabled and nearly all empty inputs, the reader can't
prune the empty events from the blocking queue fast enough to keep up. In my scenario, an
OOM occurred. 

This message was sent by Atlassian JIRA

View raw message