hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stefan Groschupf (JIRA)" <j...@apache.org>
Subject [jira] Commented: (PIG-120) support hadoop map reduce in loal mode
Date Thu, 28 Feb 2008 18:29:51 GMT

    [ https://issues.apache.org/jira/browse/PIG-120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12573394#action_12573394
] 

Stefan Groschupf commented on PIG-120:
--------------------------------------

Alan, sorry I'm not sure if I can follow you. In general I see 3 kind of how we can run pig,
LocalExecutionEngine, HadoopExecutionEngine - map reduce using a hadoop cluster and HadoopExecutionEngine
- map reduce using hadoops localJobRunner.
This is very very helpful for debugging and profiling since the HadoopExecutionEngine is used
but all runs in the same jvm. 
This patch makes it possible by not using a port in case the name node and job tracker are
"local" and also not opening a remote proxy to the jobtracker in that case.

Makes that sense? 

> support hadoop map reduce in loal mode
> --------------------------------------
>
>                 Key: PIG-120
>                 URL: https://issues.apache.org/jira/browse/PIG-120
>             Project: Pig
>          Issue Type: Bug
>            Reporter: Stefan Groschupf
>         Attachments: PIG-120_v_1.patch
>
>
> Currently pig support mapreduce and local as execution modes. LocalExecutionEngine is
used for local and HExecutionEngine for map reduce. HExecutionEngine always expect that hadoop
runs as cluster with a name node and jobtracker listing on a port. 
> Though, hadoop can also run in a local mode (LocalJobRunner) this would give several
advantages. 
> First it would speed up the test suite significant. Second it would be possible to debug
map reduce plans easily.
> For example we was able to debug and reproduce PIG-110 with this method.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message