hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alan Gates (JIRA)" <j...@apache.org>
Subject [jira] Commented: (PIG-162) Rework mapreduce submission and monitoring
Date Mon, 16 Jun 2008 20:53:44 GMT

    [ https://issues.apache.org/jira/browse/PIG-162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12605405#action_12605405
] 

Alan Gates commented on PIG-162:
--------------------------------

There's no question that we'd like to fix UDFs so they can be overloaded.  However, that is
not something we can fit in the scope of the pipeline rework at this point.

Some quick and dirty tests suggest that doing sums on integers is about an order of magnitude
faster than doing it on floats.  Given that sum is a very common operation in data processing
and that in many cases its on integers, I think it's worth optimizing.

So I don't see this as a way to start importing a bunch of functions into the language.  Instead
I see it as a way to get a huge speed up on a very commonly used feature.

> Rework mapreduce submission and monitoring
> ------------------------------------------
>
>                 Key: PIG-162
>                 URL: https://issues.apache.org/jira/browse/PIG-162
>             Project: Pig
>          Issue Type: Sub-task
>         Environment: This bug tracks works to rework the submission and monitoring interface
to map reduce as described in  http://wiki.apache.org/pig/PigTypesFunctionalSpec
>            Reporter: Alan Gates
>            Assignee: Alan Gates
>         Attachments: changes.patch, changes1.patch, incr7.patch, incr8.patch, mapreduceJumbo.patch,
mapreduceJumboWithComInc.patch, modJumbo.patch, newChanges.patch, split.png, TEST-org.apache.pig.test.TestMapReduce.txt,
TEST-org.apache.pig.test.TestMRCompiler.txt, TEST-org.apache.pig.test.TestMRCompiler.txt,
TEST-org.apache.pig.test.TestUnion.txt
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message