pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Julien Le Dem (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PIG-3764) Compile physical operators to bytecode
Date Thu, 07 Jul 2016 05:22:11 GMT

    [ https://issues.apache.org/jira/browse/PIG-3764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15365633#comment-15365633

Julien Le Dem commented on PIG-3764:

[~rohini] This sounds good. It does not have to be totally inlined since the JIT will inline
method calls, you want to avoid virtual calls though. My prototype is still out there [1].
One thing it did not take into account is nulls. But I think this can be branch out separately
(evaluate ignoring the nulls and then evaluate the is null)
Generating asm directly can be unwieldy. That's why I had made Brennus [2] to factor out a
lot of the logic (different operations per type, different stack frame size per type, all
sorts of special cases) see proto. [1]

1: https://github.com/julienledem/pig/compare/trunk...compile_physical_plan
2: https://github.com/julienledem/brennus

> Compile physical operators to bytecode
> --------------------------------------
>                 Key: PIG-3764
>                 URL: https://issues.apache.org/jira/browse/PIG-3764
>             Project: Pig
>          Issue Type: Improvement
>          Components: impl
>            Reporter: Julien Le Dem
>              Labels: GSOC2014
> I started a prototype here:
> https://github.com/julienledem/pig/compare/trunk...compile_physical_plan
> The current physical plan is relatively inefficient at evaluating expressions.
> In the context of a better execution engine (Tez, Spark, ...), compiling expressions
to bytecode would be a significant speedup.
> This is a candidate project for Google summer of code 2014. More information about the
program can be found at https://cwiki.apache.org/confluence/display/PIG/GSoc2014

This message was sent by Atlassian JIRA

View raw message