tez-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jeff Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (TEZ-1647) Issue with caching of events in VertexManager::onRootVertexInitialized
Date Thu, 09 Oct 2014 01:14:38 GMT

    [ https://issues.apache.org/jira/browse/TEZ-1647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14164533#comment-14164533
] 

Jeff Zhang commented on TEZ-1647:
---------------------------------

[~hitesh] 
bq.  it calls context.addEvents(i1) and context.addEvents(i2). When this happens, the events
generated for i1 never get sent to the VertexImpl to be routed to the tasks.

In this case, it still could get the events from i1, the only impact is that we the put events
of i1 as the value of key i2. 

{code}
  public List<TezEvent> onRootVertexInitialized(String inputName,
      InputDescriptor inputDescriptor, List<Event> events) {
    plugin.onRootVertexInitialized(inputName, inputDescriptor, events);   // will call context.addEvents(i1),
context.addEvents(i2)
    return cachedRootInputEventMap.get(inputName);   // get all the events from i1,i2
  }
{code}

BTW, the customer VM of hive you refer is org.apache.hadoop.hive.ql.exec.tez.CustomPartitionVertex,
right ?


> Issue with caching of events in VertexManager::onRootVertexInitialized 
> -----------------------------------------------------------------------
>
>                 Key: TEZ-1647
>                 URL: https://issues.apache.org/jira/browse/TEZ-1647
>             Project: Apache Tez
>          Issue Type: Bug
>            Reporter: Hitesh Shah
>            Assignee: Jeff Zhang
>
> Came across a use-case in Hive where the current functionality does not work. 
> Consider a vertex with 2 inputs ( i1, i2 ):
> The custom vertex manager in Hive generates events only when both i1 and i2 are initialized.

> Therefore vertexManagerPlugin::onRootInitialized(i1) does nothing but for vertexManagerPLugin::onRootIniitialized(i2)
, it calls context.addEvents(i1) and context.addEvents(i2). When this happens, the events
generated for i1 never get sent to the VertexImpl to be routed to the tasks. 
>   
> What this implies is that
>   
>   



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message