pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ankit Modi (JIRA)" <j...@apache.org>
Subject [jira] Commented: (PIG-1053) Consider moving to Hadoop for local mode
Date Tue, 17 Nov 2009 22:46:39 GMT

    [ https://issues.apache.org/jira/browse/PIG-1053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12779181#action_12779181

Ankit Modi commented on PIG-1053:

This patch has an issue with custom comparators ( OrderBy) in Local Mode ( Does not affect
MapReduce mode ).

Pig uses custom Comparators by setting OutputKeyComparator to the customComparator.class,
and passing the jar path to JVM while starting the task.
In this new local mode a new JVM is not started. So hadoop does not have the classpath of
customComparator and fails.

A solution for the above problem would be to pass jarpath of customComparator in the "classpath"
argument to JVM running pig.

register custom.jar
A = load 'file';
B = order A by * using custompackage.customclass; --- Here hadoop bails out giving ClassNotFoundException
store B into 'file2'

JVM Command
{{java -cp pig.jar org.pig.apache.Main -x local CustomComparatorUse.pig # This does not work}}

Use this instead
{{java -cp pig.jar:{color:red}custom.jar{color} org.pig.apache.Main -x local CustomComparatorUse.pig}}

> Consider moving to Hadoop for local mode
> ----------------------------------------
>                 Key: PIG-1053
>                 URL: https://issues.apache.org/jira/browse/PIG-1053
>             Project: Pig
>          Issue Type: Improvement
>            Reporter: Alan Gates
>            Assignee: Ankit Modi
>         Attachments: hadoopLocal.patch
> We need to consider moving Pig to use Hadoop's local mode instead of its own.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message