pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Achal Soni (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (PIG-3419) Pluggable Execution Engine
Date Thu, 29 Aug 2013 19:59:53 GMT

    [ https://issues.apache.org/jira/browse/PIG-3419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13753985#comment-13753985
] 

Achal Soni commented on PIG-3419:
---------------------------------

I agree with all that is said, but there is no need to rename HExecutionEngine back. It doesn't
semantically make sense and I don't think that anybody was directly interacting it outside
of the test cases?

Whatever changes to Ambrose and Lipstick should be communicated clearly also. I have noted
some issues with PPNL before with regard to abstraction -- namely, Pig provides the MROperPlan
to the listeners, which is not relevant in a differen execution engine. Julien suggested this
should be fixed in a follow up patch. This will most certainly affect Ambrose and Lipstick
so we should be cautious in that regard.
                
> Pluggable Execution Engine 
> ---------------------------
>
>                 Key: PIG-3419
>                 URL: https://issues.apache.org/jira/browse/PIG-3419
>             Project: Pig
>          Issue Type: New Feature
>    Affects Versions: 0.12
>            Reporter: Achal Soni
>            Assignee: Achal Soni
>            Priority: Minor
>         Attachments: execengine.patch, mapreduce_execengine.patch, stats_scriptstate.patch,
test_failures.txt, test_suite.patch, updated-8-22-2013-exec-engine.patch, updated-8-23-2013-exec-engine.patch,
updated-8-27-2013-exec-engine.patch, updated-8-28-2013-exec-engine.patch, updated-8-29-2013-exec-engine.patch
>
>
> In an effort to adapt Pig to work using Apache Tez (https://issues.apache.org/jira/browse/TEZ),
I made some changes to allow for a cleaner ExecutionEngine abstraction than existed before.
The changes are not that major as Pig was already relatively abstracted out between the frontend
and backend. The changes in the attached commit are essentially the barebones changes -- I
tried to not change the structure of Pig's different components too much. I think it will
be interesting to see in the future how we can refactor more areas of Pig to really honor
this abstraction between the frontend and backend. 
> Some of the changes was to reinstate an ExecutionEngine interface to tie together the
front end and backend, and making the changes in Pig to delegate to the EE when necessary,
and creating an MRExecutionEngine that implements this interface. Other work included changing
ExecType to cycle through the ExecutionEngines on the classpath and select the appropriate
one (this is done using Java ServiceLoader, exactly how MapReduce does for choosing the framework
to use between local and distributed mode). Also I tried to make ScriptState, JobStats, and
PigStats as abstract as possible in its current state. I think in the future some work will
need to be done here to perhaps re-evaluate the usage of ScriptState and the responsibilities
of the different statistics classes. I haven't touched the PPNL, but I think more abstraction
is needed here, perhaps in a separate patch. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message