pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shravan Matthur Narayanamurthy (JIRA)" <j...@apache.org>
Subject [jira] Updated: (PIG-162) Rework mapreduce submission and monitoring
Date Mon, 09 Jun 2008 21:06:46 GMT

     [ https://issues.apache.org/jira/browse/PIG-162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Shravan Matthur Narayanamurthy updated PIG-162:

    Attachment: changes.patch

Includes some changes that have accumulated over time. Most of them are fixes to get the end
to end tests working. Some others are bugs I found while running through the code and some
are changes to the name() methods of the operators so that we can debug better using the PlanPrinter.
I have also modified POUserFunc & some others which hold transient references to function
specs more efficient by removing the if(func==null) checks. I don't know why it did not strike
me then but the soln is simple. I have just overridden the readObject method of the serialization
framework and after calling a default read method, I instantiate the func from the spec. So
whenever other methods are called they can be sure that they will get a non null func handle.
To give an overview of the changes here is a list:
1) Byte Comparison
2) Launcher separation
3) Make basic types, Cast & Sort serializable
4) Combiner plus Combiner support during JobCreation - Though I don't have test cases for
this one, I am pretty confident that it gets launched. I was trying to get a hand crafted
case working. But the limitations of the generate and chained projects did not enable me to
do so. However, I saw through manually at least that the combiner gets called and if used
appropriately, it shoud not have any problems.
5) Changes to translator that I had done to get some of the test cases working

> Rework mapreduce submission and monitoring
> ------------------------------------------
>                 Key: PIG-162
>                 URL: https://issues.apache.org/jira/browse/PIG-162
>             Project: Pig
>          Issue Type: Sub-task
>         Environment: This bug tracks works to rework the submission and monitoring interface
to map reduce as described in  http://wiki.apache.org/pig/PigTypesFunctionalSpec
>            Reporter: Alan Gates
>            Assignee: Alan Gates
>         Attachments: changes.patch, changes1.patch, incr7.patch, incr8.patch, mapreduceJumbo.patch,
mapreduceJumboWithComInc.patch, modJumbo.patch, split.png, TEST-org.apache.pig.test.TestMapReduce.txt,
TEST-org.apache.pig.test.TestMRCompiler.txt, TEST-org.apache.pig.test.TestMRCompiler.txt,

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message