pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Craig Macdonald <cra...@dcs.gla.ac.uk>
Subject Re: How to HOD?
Date Wed, 28 Nov 2007 20:04:02 GMT
Hi Ben,

Ok, Doh moment from me. Thanks for the script, but the hint was enough 
to remind me that there's a similar version already in SVN trunk.
I dont think there's no need for a separate issue, as upgrading to work 
on Hod4 is already an unresolved issue. Presumably an upgraded Pig  
would no longer required the expect script (though notably, I dont think 
hod 4 produces all the required output ;-)

Ta muchly


Benjamin Reed wrote:
> Ah yes, sorry about that. We had a problem with HOD not working well with 
> piped inputs and outputs, so we actually use an expect script to interface to 
> hod. (We should open an issue on this.)
> I'm attaching the script that we use.
> ben
> On Wednesday 28 November 2007 11:38:09 Craig Macdonald wrote:
>> Hi all,
>> I've been trying to setup Pig using Hadoop on Demand. Using some
>> hackery, my incantation now looks like
>> PATH=/users/tr.craigm/OF_tools/python/bin/:$PATH ROOT=$PWD
>> scripts/pig.pl -Dlog4j.level=debug -Dhod.server=local
>> -Dhod.expect.root=$PWD -Dhod.command=hod/bin/hod
>> -Dhod.expect.uselatest=hodrc/released -Dyinst.cluster=
>> -Dhadoop.root.logger=DEBUG,console  --cluster hodrc
>> (the name of my hodrc file is hodrc).
>> However, the HOD connection code in PigContext mystifies me. Does it
>> correspond to any released version of HOD?
>> It seems to connect to HOD, and parse the response.
>> PIG-18 (https://issues.apache.org/jira/browse/PIG-18) states that Pig
>> needs to be fixed to work with hod 4.
>> So I presume that Pig does not worth with the HOD version
>> hod-open-4.tar.gz  attached to
>> https://issues.apache.org/jira/browse/HADOOP-1301
>> However, it doesnt look like Pig works with the other version of Hod
>> attached to the same JIRA issue: hod.0.2.2.tar.gz
>> PigContent.java looks for output from HOD in the form of lines starting:
>> hdfsUI:
>> hdfs:
>> mapredUI:
>> mapred:
>> hadoopConf:
>> I cant find any source in either versions of HOD that resemble this.
>> Does anyone know if Pig will currently work with any currently openly
>> available version of HOD?
>> Thanks in advance
>> Craig

View raw message