hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ankit Modi (JIRA)" <j...@apache.org>
Subject [jira] Commented: (PIG-1053) Consider moving to Hadoop for local mode
Date Tue, 17 Nov 2009 22:38:39 GMT

    [ https://issues.apache.org/jira/browse/PIG-1053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12779179#action_12779179

Ankit Modi commented on PIG-1053:

This patch has an issue with custom comparators ( OrderBy) in Local Mode ( Does not affect
MapReduce mode ).

Pig uses custom Comparators by setting OutputKeyComparator to the customComparator.class,
and passing the jar path to JVM while starting the task.
In this new local mode a new JVM is not started. So hadoop does not have the classpath of
customComparator and fails.

A solution for the above problem would be to pass jarpath of customComparator in the "classpath"
argument to JVM running pig.

eg. CustomComparatorUse.pig
register custom.jar

A = load 'file';B = order a by * using custompackage.customclass; -- Here hadoop
>> bails out giving ClassNotFoundException

> Consider moving to Hadoop for local mode
> ----------------------------------------
>                 Key: PIG-1053
>                 URL: https://issues.apache.org/jira/browse/PIG-1053
>             Project: Pig
>          Issue Type: Improvement
>            Reporter: Alan Gates
>            Assignee: Ankit Modi
>         Attachments: hadoopLocal.patch
> We need to consider moving Pig to use Hadoop's local mode instead of its own.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message