Mailing-List: contact user-help@hama.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hama.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <345801A3A7546D488A0CE4001E4FFB287E7FA1@RHV-EXRDA-S11.corp.ebay.com>
References: 
 <CAGQgZQQbiby6s2cWdVHyPOpP9M_1PvEHnjoDuBuU1nvcXAw_0A@mail.gmail.com>
	<345801A3A7546D488A0CE4001E4FFB287E7FA1@RHV-EXRDA-S11.corp.ebay.com>
Date: Thu, 15 Nov 2012 11:04:35 +0900
Message-ID: 
 <CAGQgZQSzGjgeM_JrMEJYdzY3OUtfpRygRM7GJAJ-wGTa0jw1+A@mail.gmail.com>
Subject: Re: Load user library to Hama
From: "Edward J. Yoon" <edwardyoon@apache.org>
To: "Tu, Min" <mitu@paypal.com>
Cc: user@hama.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Can you provide me your tasklogs?

> I am writing my own VertexInputReader and need a third party library to
> parse the input.

If you added your jars to {$HAMA_HOME}/lib folder (all hosts), you can
check whether your jars are included as following command.

[oracle@bda01 hama-0.6.0]$ ps -ef | grep Groom

oracle   29928     1 23 17:25 ?        00:00:01
/usr/java/jdk1.6.0_29/bin/java -Xmx2000m
-Dhama.log.dir=3D/home/oracle/hama-0.6.0/bin/../logs
-Dhama.log.file=3Dhama-oracle-groom-bda01.oracle.com.log
-Dhama.home.dir=3D/home/oracle/hama-0.6.0/bin/.. -Dhama.id.str=3Doracle
-Dhama.root.logger=3DINFO,DRFA -Dhama.policy.file=3Dhama-policy.xml
-classpath /home/oracle/hama-0.6.0/bin/../conf:/usr/java/jdk1.6.0_29/lib/to=
ols.jar:/home/oracle/hama-0.6.0/bin/../hama-core-0.6.0.jar:/home/oracle/ham=
a-0.6.0/bin/../hama-examples-0.6.0.jar:/home/oracle/hama-0.6.0/bin/../hama-=
graph-0.6.0.jar:/home/oracle/hama-0.6.0/bin/../hama-ml-0.6.0.jar:/home/orac=
le/hama-0.6.0/bin/../lib/ant-1.7.1.jar:/home/oracle/hama-0.6.0/bin/../lib/a=
nt-launcher-1.7.1.jar:/home/oracle/hama-0.6.0/bin/../lib/avro-1.6.0.jar:/ho=
me/oracle/hama-0.6.0/bin/../lib/avro-ipc-1.6.0.jar:/home/oracle/hama-0.6.0/=
bin/../lib/commons-cli-1.2.jar:/home/oracle/hama-0.6.0/bin/../lib/commons-c=
onfiguration-1.7.jar:/home/oracle/hama-0.6.0/bin/../lib/commons-httpclient-=
3.0.1.jar:/home/oracle/hama-0.6.0/bin/../lib/commons-lang-2.6.jar:/home/ora=
cle/hama-0.6.0/bin/../lib/commons-logging-1.1.1.jar:/home/oracle/hama-0.6.0=
/bin/../lib/commons-math3-3.0.jar:/home/oracle/hama-0.6.0/bin/../lib/guava-=
10.0.1.jar:/home/oracle/hama-0.6.0/bin/../lib/guava-r09-jarjar.jar:/home/or=
acle/hama-0.6.0/bin/../lib/hadoop-core-0.20.2-cdh3u3b.jar:/home/oracle/hama=
-0.6.0/bin/../lib/hadoop-test-0.20.2-cdh3u3b.jar:/home/oracle/hama-0.6.0/bi=
n/../lib/jackson-core-asl-1.9.2.jar:/home/oracle/hama-0.6.0/bin/../lib/jack=
son-mapper-asl-1.9.2.jar:/home/oracle/hama-0.6.0/bin/../lib/jetty-6.1.14.ja=
r:/home/oracle/hama-0.6.0/bin/../lib/jetty-annotations-6.1.14.jar:/home/ora=
cle/hama-0.6.0/bin/../lib/jetty-util-6.1.14.jar:/home/oracle/hama-0.6.0/bin=
/../lib/jsp-2.1-6.1.14.jar:/home/oracle/hama-0.6.0/bin/../lib/jsp-api-2.1-6=
.1.14.jar:/home/oracle/hama-0.6.0/bin/../lib/junit-4.8.1.jar:/home/oracle/h=
ama-0.6.0/bin/../lib/log4j-1.2.16.jar:/home/oracle/hama-0.6.0/bin/../lib/ne=
tty-3.2.6.Final.jar:/home/oracle/hama-0.6.0/bin/../lib/servlet-api-6.0.32.j=
ar:/home/oracle/hama-0.6.0/bin/../lib/slf4j-api-1.5.8.jar:/home/oracle/hama=
-0.6.0/bin/../lib/slf4j-log4j12-1.5.8.jar:/home/oracle/hama-0.6.0/bin/../li=
b/snappy-java-1.0.4.1.jar:/home/oracle/hama-0.6.0/bin/../lib/zookeeper-3.3.=
3.jar
org.apache.hama.GroomServerRunner bda01-adm
oracle   30029 29694  0 17:25 pts/0    00:00:00 grep Groom


On Thu, Nov 15, 2012 at 10:18 AM, Tu, Min <mitu@paypal.com> wrote:
> Hi Edward and Suraj,
>
> Thank you very much for spending time answer my questions. Really
> appreciated.
>
> I have tried to put my library to $HAMA_HOME/lib folder in both BSP and
> Groom hosts.
>
> The BSP server pick it up fine but Groom server still starts without that
> package in the class path.
>
> Any thing I did wrong?
>
> Also Is there another way to ship the 3rd party jar without put those int=
o
> the $HAMA_HOME/lib? Like Distributed cache in hadoop?
>
> Here are the hama command and partial code :
>
> hama jar ./POC-0.0.1-SNAPSHOT.jar
> com.paypal.hadoop.platform.graphjob.SimpleRiskPropagation
> /user/mitu/riskData/DW_PAYMENT_SENT/preprocess/output/2nd
> /user/mitu/riskData/DW_PAYMENT_SENT/propagate/output/2nd
>
>
> --------------------- Main function in the SimpleRiskPropagation.java ---=
--
> public static void main(String... args) throws IOException,
>         InterruptedException, ClassNotFoundException {
>
>                 if (args.length < 2) {
>                         printUsage();
>                 }
>
>                 HamaConfiguration conf =3D new HamaConfiguration(new Conf=
iguration());
>                 GraphJob job =3D new GraphJob(conf, SimpleRiskPropagation=
.class);
>
>                 // set the defaults
>                 //job.setMaxIteration(30);
>                 job.setMaxIteration(3);
>                 job.setNumBspTask(2);
>
>                 if (args.length >=3D 4)
>                         job.setNumBspTask(Integer.parseInt(args[3]));
>                 if (args.length >=3D 3)
>                         job.setMaxIteration(Integer.parseInt(args[2]));
>
>                 job.setJobName("SimpleRiskPropagation");
>                 job.setInputPath(new Path(args[0]));
>                 job.setOutputPath(new Path(args[1]));
>
>                 job.setVertexClass(SimpleRiskPropagationVertex.class);
>                 job.setVertexIDClass(Text.class);
>                 job.setVertexValueClass(Pairs.class);
>                 job.setEdgeValueClass(SimpleTransaction.class);
>
>                 job.setInputKeyClass(LongWritable.class);
>                 job.setInputValueClass(Text.class);
>                 job.setInputFormat(TextInputFormat.class);
>
>
>                 /*
>                  * this reader class will be set in GraphJob conf with ke=
y:
> VERTEX_GRAPH_INPUT_READER
>                  * and used by class GraphJobRunner 's function loadVerti=
ces()
>                  */
>                 job.setVertexInputReaderClass(SimpleRiskPropagationVertex=
Reader.class);
>
>
>                 job.setPartitioner(HashPartitioner.class);
>                 job.setOutputFormat(TextOutputFormat.class);
>                 job.setOutputKeyClass(Text.class);
>                 job.setOutputValueClass(Text.class);
>
>                 long startTime =3D System.currentTimeMillis();
>                 if (job.waitForCompletion(true)) {
>                         System.out.println("Job Finished in "
>                                         + (System.currentTimeMillis() - s=
tartTime) / 1000.0 + " seconds");
>                 }
>         }
> --------------------- Main function in the SimpleRiskPropagation.java ---=
--
>
>
>
> On 11/13/12 6:13 AM, "Edward J. Yoon" <edwardyoon@apache.org> wrote:
>
>>Did you subscribe?
>>
>>---------- Forwarded message ----------
>>From: Edward J. Yoon <edwardyoon@apache.org>
>>Date: Tue, Nov 13, 2012 at 7:42 PM
>>Subject: Re: Load user library to Hama
>>To: user@hama.apache.org
>>
>>
>>Hi Tu, Min
>>
>>You can solve the problem by copying your 3rd party library to
>>{$HAMA}/lib folder.
>>
>>On Tue, Nov 13, 2012 at 7:32 PM, Suraj Menon <menonsuraj5@gmail.com>
>>wrote:
>>> Hi,
>>>
>>> Can you give us the command you are trying. Also, you can try the
>>> distributed cache feature of Hadoop for the same.
>>>
>>> Thanks,
>>> Suraj
>>>
>>>
>>> On Mon, Nov 12, 2012 at 9:03 PM, Tu, Min <mitu@paypal.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> I am new to Hama and got a question for using it.
>>>>
>>>> I am writing my own VertexInputReader and need a third party library t=
o
>>>> parse the input. How should I ship the library jar to the GroomServer
>>>>in
>>>> Hama. I have tried to use the =C2=ADlibjars option but did not work: g=
ot
>>>> ClassNotFoundException in the VertexInputReader part.
>>>>
>>>>
>>>> Thanks a lot
>>>>
>>>> Min
>>>>
>>
>>
>>
>>--
>>Best Regards, Edward J. Yoon
>>@eddieyoon
>>
>>
>>--
>>Best Regards, Edward J. Yoon
>>@eddieyoon
>


--=20
Best Regards, Edward J. Yoon
@eddieyoon