hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Craig Macdonald (JIRA)" <j...@apache.org>
Subject [jira] Commented: (PIG-18) Making Pig work with HOD 0.4 and hadoop 0.16
Date Sun, 16 Mar 2008 15:02:24 GMT

    [ https://issues.apache.org/jira/browse/PIG-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12579199#action_12579199

Craig Macdonald commented on PIG-18:


Apologies on the HOD confusion. I think you are privy to more information than me!

For SSH tunneling, i understand your Hadoop cluster is behind a SSH gateway. Is it also SOCKS

An easier option would be: 
Run HOD locally, use ssh wrappers to run the Torque tools, and tell Java to use the SOCKS
server for all socket connections, i.e. the Hadoop connections.

For the SSH tunneling of the Torque commands, create this executable script in your path:
ssh sshgateway $REMOTEPBSPATH/$0 "$@"

and symlink qsub and qstat to it.

See http://java.sun.com/javase/6/docs/technotes/guides/net/proxies.html for how to configure
Java to use SOCKS.
A simple SOCKS server is http://ss5.sourceforge.net

Overall, this would keep the Pig codebase more lightweight.


> Making Pig work with HOD 0.4 and hadoop 0.16
> --------------------------------------------
>                 Key: PIG-18
>                 URL: https://issues.apache.org/jira/browse/PIG-18
>             Project: Pig
>          Issue Type: Bug
>          Components: impl
>            Reporter: Olga Natkovich
>            Assignee: Olga Natkovich
>            Priority: Critical
>         Attachments: hadoop16.jar, PIG-18.patch, PIG-18.patch-v2craigm-07032008, PIG-18.patch-v3craigm-10032008
> Pig needs to work with the soon to be released hadoop on demand (HOD) version.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message