pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dmitriy V. Ryaboy (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PIG-3419) Pluggable Execution Engine
Date Mon, 16 Sep 2013 19:20:53 GMT

    [ https://issues.apache.org/jira/browse/PIG-3419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13768649#comment-13768649
] 

Dmitriy V. Ryaboy commented on PIG-3419:
----------------------------------------

+1 to marking the interfaces as evolving.
                
> Pluggable Execution Engine 
> ---------------------------
>
>                 Key: PIG-3419
>                 URL: https://issues.apache.org/jira/browse/PIG-3419
>             Project: Pig
>          Issue Type: New Feature
>    Affects Versions: 0.12
>            Reporter: Achal Soni
>            Assignee: Achal Soni
>            Priority: Minor
>             Fix For: 0.12
>
>         Attachments: execengine.patch, mapreduce_execengine.patch, stats_scriptstate.patch,
test_failures.txt, test_suite.patch, updated-8-22-2013-exec-engine.patch, updated-8-23-2013-exec-engine.patch,
updated-8-27-2013-exec-engine.patch, updated-8-28-2013-exec-engine.patch, updated-8-29-2013-exec-engine.patch
>
>
> In an effort to adapt Pig to work using Apache Tez (https://issues.apache.org/jira/browse/TEZ),
I made some changes to allow for a cleaner ExecutionEngine abstraction than existed before.
The changes are not that major as Pig was already relatively abstracted out between the frontend
and backend. The changes in the attached commit are essentially the barebones changes -- I
tried to not change the structure of Pig's different components too much. I think it will
be interesting to see in the future how we can refactor more areas of Pig to really honor
this abstraction between the frontend and backend. 
> Some of the changes was to reinstate an ExecutionEngine interface to tie together the
front end and backend, and making the changes in Pig to delegate to the EE when necessary,
and creating an MRExecutionEngine that implements this interface. Other work included changing
ExecType to cycle through the ExecutionEngines on the classpath and select the appropriate
one (this is done using Java ServiceLoader, exactly how MapReduce does for choosing the framework
to use between local and distributed mode). Also I tried to make ScriptState, JobStats, and
PigStats as abstract as possible in its current state. I think in the future some work will
need to be done here to perhaps re-evaluate the usage of ScriptState and the responsibilities
of the different statistics classes. I haven't touched the PPNL, but I think more abstraction
is needed here, perhaps in a separate patch. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message