hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thejas M Nair (JIRA)" <j...@apache.org>
Subject [jira] Updated: (PIG-1472) Optimize serialization/deserialization between Map and Reduce and between MR jobs
Date Wed, 07 Jul 2010 05:16:50 GMT

     [ https://issues.apache.org/jira/browse/PIG-1472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Thejas M Nair updated PIG-1472:

    Attachment: PIG-1472.patch

Summary of changes in the patch -
1. The default TupleFactory is now BinSedesTupleFactory.  It returns the tuple implementation
in BinSedesTuple class. This changes the serialization format between Map and Reduce.
2. The (de)serialization in BinSedesTuple and DefaultAbstractBag uses an implementation of
a new InterSedes interface, which is returned by InterSedesFactory.getInterSedesInstance()

3. A new load function InterStorage is used for serializing data between MR jobs . This load
function should not be used like a regular load/store function to store persistent data.
4. DefaultTupleFactory has been retained, so that any external udfs that were using it can
still compile. DefaultTupleFactory is a subclass of BinSedesTupleFactory that does not override
any of the functions.

I think the serialization format should not be tied to the Tuple. A load function can return
any tuple implementation, if we happen to call write function of that tuple, it will not be
possible to read it in the reduce side using the default tuple. I think the InterSedes/InterSedesFactory
classes should be used instead. With this patch, the InterSedes/InterSedesFactory classes
get used only when BinSedesTuple is the default tuple.

> Optimize serialization/deserialization between Map and Reduce and between MR jobs
> ---------------------------------------------------------------------------------
>                 Key: PIG-1472
>                 URL: https://issues.apache.org/jira/browse/PIG-1472
>             Project: Pig
>          Issue Type: Improvement
>    Affects Versions: 0.8.0
>            Reporter: Thejas M Nair
>            Assignee: Thejas M Nair
>             Fix For: 0.8.0
>         Attachments: PIG-1472.patch
> In certain types of pig queries most of the execution time is spent in serializing/deserializing
(sedes) records between Map and Reduce and between MR jobs. 
> For example, if PigMix queries are modified to specify types for all the fields in the
load statement schema, some of the queries (L2,L3,L9, L10 in pigmix v1) that have records
with bags and maps being transmitted across map or reduce boundaries run a lot longer (runtime
increase of few times has been seen.
> There are a few optimizations that have shown to improve the performance of sedes in
my tests -
> 1. Use smaller number of bytes to store length of the column . For example if a bytearray
is smaller than 255 bytes , a byte can be used to store the length instead of the integer
that is currently used.
> 2. Instead of custom code to do sedes on Strings, use DataOutput.writeUTF and DataInput.readUTF.
 This reduces the cost of serialization by more than 1/2. 
> Zebra and BinStorage are known to use DefaultTuple sedes functionality. The serialization
format that these loaders use cannot change, so after the optimization their format is going
to be different from the format used between M/R boundaries.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message