hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thejas M Nair (JIRA)" <j...@apache.org>
Subject [jira] Created: (PIG-688) PERFORMANCE: Vectorize operators
Date Thu, 26 Feb 2009 22:43:02 GMT
PERFORMANCE: Vectorize operators
--------------------------------

                 Key: PIG-688
                 URL: https://issues.apache.org/jira/browse/PIG-688
             Project: Pig
          Issue Type: Improvement
            Reporter: Thejas M Nair


By Vectorization, I mean passing multiple (/vector of) records at a time between operators
(and potentially other functions like udfs)

Vectorization of pig operators can improve performance by 
1. improving locality and cache utilization
2. Reducing number of function calls. Many functions calls are likely to be dynamically resolved.
There may be some checks in each function that we might be able to do once for several recs.
3. Potentially benefit from cpu pipeline architecture. ( But I don't know how good java VM
is at that ..)

To do vectorization in map stage, we need to use MapRunner - see PIG-687.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message