hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shravan Matthur Narayanamurthy (JIRA)" <j...@apache.org>
Subject [jira] Updated: (PIG-162) Rework mapreduce submission and monitoring
Date Thu, 15 May 2008 23:24:55 GMT

     [ https://issues.apache.org/jira/browse/PIG-162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Shravan Matthur Narayanamurthy updated PIG-162:
-----------------------------------------------

    Attachment: incr7.patch

Patch with the following changes:
MR version of Sort supported (has ugly hand built physical plans for the quantile job &
sort job)
Made changes to the PlanPrinter to also visit and print the nested expr plans
Made changes to the name() methods of some Physical Operators to display more information.
Changed POProject to include a StarSpec. Included a boolean variable start to denote if the
POProject represents a star spec. If so in the getNext(Tuple) method, passes the inp tuple
as is instead of projecting something.
Changed POLocalRearrange by removing the Physical plan containing generate. It now works on
just a List of expr plans building key and value tuples on its own. This curtails one's ability
to supply flatten inside the local rearrange but I am really not sure if/how it is supported
currently and whether it really makes sense.
Made changes to MRCompiler to support POSort, to compute the requested parallelism as the
max of req par of all the operators inside a Map-Reduce job. Also included are changes to
MRCompiler which create the set of UDFs that are associated with a job so that they can be
loaded into the job jar file. To achieve this, have introduced two visitor classes UDFFinder
& UDFFinderForExp
Converted POUserFunc and POUserComparisonFunc into ExpressionOps.
Changed the launcher thingy to have non static protected launch method. Also made the launcher's
implicit constructor protected so that direct launcher objects can't be created.
Have testcases to check changes to MRCompiler for Sort, requested parallelism & UDF computation.
Include shubham's test cases for BinCond and Negative in the build.xml which he probably forgot
to do.


> Rework mapreduce submission and monitoring
> ------------------------------------------
>
>                 Key: PIG-162
>                 URL: https://issues.apache.org/jira/browse/PIG-162
>             Project: Pig
>          Issue Type: Sub-task
>         Environment: This bug tracks works to rework the submission and monitoring interface
to map reduce as described in  http://wiki.apache.org/pig/PigTypesFunctionalSpec
>            Reporter: Alan Gates
>            Assignee: Alan Gates
>         Attachments: incr7.patch, mapreduceJumbo.patch, mapreduceJumboWithComInc.patch,
modJumbo.patch, split.png, TEST-org.apache.pig.test.TestMRCompiler.txt, TEST-org.apache.pig.test.TestMRCompiler.txt,
TEST-org.apache.pig.test.TestUnion.txt
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message