Mailing-List: contact dev-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hive.apache.org
Date: Tue, 5 Mar 2013 06:21:13 +0000 (UTC)
From: "Gopal V (JIRA)" <jira@apache.org>
To: hive-dev@hadoop.apache.org
Message-ID: <JIRA.12634336.1361930005354.381052.1362464473016@arcas>
In-Reply-To: <JIRA.12634336.1361930005354@arcas>
References: <JIRA.12634336.1361930005354@arcas>
Subject: [jira] [Commented] (HIVE-4078) Remove the serialize-deserialize
 pair in CommonJoinResolver
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HIVE-4078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13593107#comment-13593107 ] 

Gopal V commented on HIVE-4078:
-------------------------------

This is not a pair of operations as previously assumed, but does the deserialization repeatedly for each alias

{code}
 for (int i = 0; i < numAliases; i++) {
       ...
          // create map join task and set big table as i
          ObjectPair<MapRedTask, String> newTaskAlias = convertTaskToMapJoinTask(xml, i);
          MapRedTask newTask = newTaskAlias.getFirst();
          bigTableAlias = newTaskAlias.getSecond();
 }
{code}
                
> Remove the serialize-deserialize pair in CommonJoinResolver
> -----------------------------------------------------------
>
>                 Key: HIVE-4078
>                 URL: https://issues.apache.org/jira/browse/HIVE-4078
>             Project: Hive
>          Issue Type: Bug
>          Components: Query Processor
>            Reporter: Gopal V
>            Assignee: Gopal V
>         Attachments: HIVE-4078-20130227.patch, HIVE-4078.patch
>
>
> CommonJoinProcessor tries to clone a MapredWork while attempting a conversion to a map-join
> {code}
>   // deep copy a new mapred work from xml
>   InputStream in = new ByteArrayInputStream(xml.getBytes("UTF-8"));
>   MapredWork newWork = Utilities.deserializeMapRedWork(in, physicalContext.getConf());
> {code}
> which is a very heavy operation memory wise & cpu-wise.
> Instead of cloning via XMLEncoder, it is faster to use BeanUtils.cloneBean() which is following same data paths (get/set bean methods) instead.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira