hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ankit Modi (JIRA)" <j...@apache.org>
Subject [jira] Commented: (PIG-1053) Consider moving to Hadoop for local mode
Date Tue, 17 Nov 2009 22:38:39 GMT

    [ https://issues.apache.org/jira/browse/PIG-1053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12779179#action_12779179
] 

Ankit Modi commented on PIG-1053:
---------------------------------

This patch has an issue with custom comparators ( OrderBy) in Local Mode ( Does not affect
MapReduce mode ).

Details:
Pig uses custom Comparators by setting OutputKeyComparator to the customComparator.class,
and passing the jar path to JVM while starting the task.
In this new local mode a new JVM is not started. So hadoop does not have the classpath of
customComparator and fails.

A solution for the above problem would be to pass jarpath of customComparator in the "classpath"
argument to JVM running pig.

eg. CustomComparatorUse.pig
register custom.jar

A = load 'file';B = order a by * using custompackage.customclass; -- Here hadoop
>> bails out giving ClassNotFoundException

> Consider moving to Hadoop for local mode
> ----------------------------------------
>
>                 Key: PIG-1053
>                 URL: https://issues.apache.org/jira/browse/PIG-1053
>             Project: Pig
>          Issue Type: Improvement
>            Reporter: Alan Gates
>            Assignee: Ankit Modi
>         Attachments: hadoopLocal.patch
>
>
> We need to consider moving Pig to use Hadoop's local mode instead of its own.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message