Mailing-List: contact dev-help@pig.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@pig.apache.org
Date: Wed, 2 Mar 2011 00:05:37 +0000 (UTC)
From: "Alan Gates (JIRA)" <jira@apache.org>
To: pig-dev@hadoop.apache.org
Message-ID: 
 <1795461796.6501.1299024337005.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: 
 <1662850998.6409.1299022117621.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] Updated: (PIG-1875) Keep tuples serialized to limit spilling
 and speed it when it happens
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


     [ https://issues.apache.org/jira/browse/PIG-1875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alan Gates updated PIG-1875:
----------------------------

    Attachment: mrtuple.patch

Here's a first pass at what MToRTuple might look like.  I've done some basic testing to assure this works, but nothing comprehensive.

In test runs where I serialized 100k tuples, wrote them to disk, and read them back I got the following results:

DefaultTuple:
time to write to disk:       81.93 sec
size on disk:                98M
time to read from disk:      12.62 sec
size in memory (after read): 238M

MToRTuple:
time to write to disk:       10.49 sec
size on disk:                58M
time to read from disk:      1.10 sec
size in memory (after read): 57M

So roughly 1/4 the memory consumption and ~10x speedup on disk reads and writes.


> Keep tuples serialized to limit spilling and speed it when it happens
> ---------------------------------------------------------------------
>
>                 Key: PIG-1875
>                 URL: https://issues.apache.org/jira/browse/PIG-1875
>             Project: Pig
>          Issue Type: Improvement
>          Components: impl
>            Reporter: Alan Gates
>            Priority: Minor
>         Attachments: mrtuple.patch
>
>
> Currently Pig reads records off of the reduce iterator and immediately deserializes them into Java objects.  This takes up much more memory than serialized versions, thus Pig spills sooner then if it stored them in serialized form.  Also, if it does have to spill, it has to serialize them again, and then again deserialize them after reading from the spill file.
> We should explore storing them in memory serialized when they are read off of the reduce iterator.

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira