hama-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Edward J. Yoon" <edwardy...@apache.org>
Subject Re: Load user library to Hama
Date Thu, 15 Nov 2012 02:04:35 GMT
Can you provide me your tasklogs?

> I am writing my own VertexInputReader and need a third party library to
> parse the input.

If you added your jars to {$HAMA_HOME}/lib folder (all hosts), you can
check whether your jars are included as following command.

[oracle@bda01 hama-0.6.0]$ ps -ef | grep Groom

oracle   29928     1 23 17:25 ?        00:00:01
/usr/java/jdk1.6.0_29/bin/java -Xmx2000m
-Dhama.log.dir=/home/oracle/hama-0.6.0/bin/../logs
-Dhama.log.file=hama-oracle-groom-bda01.oracle.com.log
-Dhama.home.dir=/home/oracle/hama-0.6.0/bin/.. -Dhama.id.str=oracle
-Dhama.root.logger=INFO,DRFA -Dhama.policy.file=hama-policy.xml
-classpath /home/oracle/hama-0.6.0/bin/../conf:/usr/java/jdk1.6.0_29/lib/tools.jar:/home/oracle/hama-0.6.0/bin/../hama-core-0.6.0.jar:/home/oracle/hama-0.6.0/bin/../hama-examples-0.6.0.jar:/home/oracle/hama-0.6.0/bin/../hama-graph-0.6.0.jar:/home/oracle/hama-0.6.0/bin/../hama-ml-0.6.0.jar:/home/oracle/hama-0.6.0/bin/../lib/ant-1.7.1.jar:/home/oracle/hama-0.6.0/bin/../lib/ant-launcher-1.7.1.jar:/home/oracle/hama-0.6.0/bin/../lib/avro-1.6.0.jar:/home/oracle/hama-0.6.0/bin/../lib/avro-ipc-1.6.0.jar:/home/oracle/hama-0.6.0/bin/../lib/commons-cli-1.2.jar:/home/oracle/hama-0.6.0/bin/../lib/commons-configuration-1.7.jar:/home/oracle/hama-0.6.0/bin/../lib/commons-httpclient-3.0.1.jar:/home/oracle/hama-0.6.0/bin/../lib/commons-lang-2.6.jar:/home/oracle/hama-0.6.0/bin/../lib/commons-logging-1.1.1.jar:/home/oracle/hama-0.6.0/bin/../lib/commons-math3-3.0.jar:/home/oracle/hama-0.6.0/bin/../lib/guava-10.0.1.jar:/home/oracle/hama-0.6.0/bin/../lib/guava-r09-jarjar.jar:/home/oracle/hama-0.6.0/bin/../lib/hadoop-core-0.20.2-cdh3u3b.jar:/home/oracle/hama-0.6.0/bin/../lib/hadoop-test-0.20.2-cdh3u3b.jar:/home/oracle/hama-0.6.0/bin/../lib/jackson-core-asl-1.9.2.jar:/home/oracle/hama-0.6.0/bin/../lib/jackson-mapper-asl-1.9.2.jar:/home/oracle/hama-0.6.0/bin/../lib/jetty-6.1.14.jar:/home/oracle/hama-0.6.0/bin/../lib/jetty-annotations-6.1.14.jar:/home/oracle/hama-0.6.0/bin/../lib/jetty-util-6.1.14.jar:/home/oracle/hama-0.6.0/bin/../lib/jsp-2.1-6.1.14.jar:/home/oracle/hama-0.6.0/bin/../lib/jsp-api-2.1-6.1.14.jar:/home/oracle/hama-0.6.0/bin/../lib/junit-4.8.1.jar:/home/oracle/hama-0.6.0/bin/../lib/log4j-1.2.16.jar:/home/oracle/hama-0.6.0/bin/../lib/netty-3.2.6.Final.jar:/home/oracle/hama-0.6.0/bin/../lib/servlet-api-6.0.32.jar:/home/oracle/hama-0.6.0/bin/../lib/slf4j-api-1.5.8.jar:/home/oracle/hama-0.6.0/bin/../lib/slf4j-log4j12-1.5.8.jar:/home/oracle/hama-0.6.0/bin/../lib/snappy-java-1.0.4.1.jar:/home/oracle/hama-0.6.0/bin/../lib/zookeeper-3.3.3.jar
org.apache.hama.GroomServerRunner bda01-adm
oracle   30029 29694  0 17:25 pts/0    00:00:00 grep Groom


On Thu, Nov 15, 2012 at 10:18 AM, Tu, Min <mitu@paypal.com> wrote:
> Hi Edward and Suraj,
>
> Thank you very much for spending time answer my questions. Really
> appreciated.
>
> I have tried to put my library to $HAMA_HOME/lib folder in both BSP and
> Groom hosts.
>
> The BSP server pick it up fine but Groom server still starts without that
> package in the class path.
>
> Any thing I did wrong?
>
> Also Is there another way to ship the 3rd party jar without put those into
> the $HAMA_HOME/lib? Like Distributed cache in hadoop?
>
> Here are the hama command and partial code :
>
> hama jar ./POC-0.0.1-SNAPSHOT.jar
> com.paypal.hadoop.platform.graphjob.SimpleRiskPropagation
> /user/mitu/riskData/DW_PAYMENT_SENT/preprocess/output/2nd
> /user/mitu/riskData/DW_PAYMENT_SENT/propagate/output/2nd
>
>
> --------------------- Main function in the SimpleRiskPropagation.java -----
> public static void main(String... args) throws IOException,
>         InterruptedException, ClassNotFoundException {
>
>                 if (args.length < 2) {
>                         printUsage();
>                 }
>
>                 HamaConfiguration conf = new HamaConfiguration(new Configuration());
>                 GraphJob job = new GraphJob(conf, SimpleRiskPropagation.class);
>
>                 // set the defaults
>                 //job.setMaxIteration(30);
>                 job.setMaxIteration(3);
>                 job.setNumBspTask(2);
>
>                 if (args.length >= 4)
>                         job.setNumBspTask(Integer.parseInt(args[3]));
>                 if (args.length >= 3)
>                         job.setMaxIteration(Integer.parseInt(args[2]));
>
>                 job.setJobName("SimpleRiskPropagation");
>                 job.setInputPath(new Path(args[0]));
>                 job.setOutputPath(new Path(args[1]));
>
>                 job.setVertexClass(SimpleRiskPropagationVertex.class);
>                 job.setVertexIDClass(Text.class);
>                 job.setVertexValueClass(Pairs.class);
>                 job.setEdgeValueClass(SimpleTransaction.class);
>
>                 job.setInputKeyClass(LongWritable.class);
>                 job.setInputValueClass(Text.class);
>                 job.setInputFormat(TextInputFormat.class);
>
>
>                 /*
>                  * this reader class will be set in GraphJob conf with key:
> VERTEX_GRAPH_INPUT_READER
>                  * and used by class GraphJobRunner 's function loadVertices()
>                  */
>                 job.setVertexInputReaderClass(SimpleRiskPropagationVertexReader.class);
>
>
>                 job.setPartitioner(HashPartitioner.class);
>                 job.setOutputFormat(TextOutputFormat.class);
>                 job.setOutputKeyClass(Text.class);
>                 job.setOutputValueClass(Text.class);
>
>                 long startTime = System.currentTimeMillis();
>                 if (job.waitForCompletion(true)) {
>                         System.out.println("Job Finished in "
>                                         + (System.currentTimeMillis() - startTime) /
1000.0 + " seconds");
>                 }
>         }
> --------------------- Main function in the SimpleRiskPropagation.java -----
>
>
>
> On 11/13/12 6:13 AM, "Edward J. Yoon" <edwardyoon@apache.org> wrote:
>
>>Did you subscribe?
>>
>>---------- Forwarded message ----------
>>From: Edward J. Yoon <edwardyoon@apache.org>
>>Date: Tue, Nov 13, 2012 at 7:42 PM
>>Subject: Re: Load user library to Hama
>>To: user@hama.apache.org
>>
>>
>>Hi Tu, Min
>>
>>You can solve the problem by copying your 3rd party library to
>>{$HAMA}/lib folder.
>>
>>On Tue, Nov 13, 2012 at 7:32 PM, Suraj Menon <menonsuraj5@gmail.com>
>>wrote:
>>> Hi,
>>>
>>> Can you give us the command you are trying. Also, you can try the
>>> distributed cache feature of Hadoop for the same.
>>>
>>> Thanks,
>>> Suraj
>>>
>>>
>>> On Mon, Nov 12, 2012 at 9:03 PM, Tu, Min <mitu@paypal.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> I am new to Hama and got a question for using it.
>>>>
>>>> I am writing my own VertexInputReader and need a third party library to
>>>> parse the input. How should I ship the library jar to the GroomServer
>>>>in
>>>> Hama. I have tried to use the ­libjars option but did not work: got
>>>> ClassNotFoundException in the VertexInputReader part.
>>>>
>>>>
>>>> Thanks a lot
>>>>
>>>> Min
>>>>
>>
>>
>>
>>--
>>Best Regards, Edward J. Yoon
>>@eddieyoon
>>
>>
>>--
>>Best Regards, Edward J. Yoon
>>@eddieyoon
>



-- 
Best Regards, Edward J. Yoon
@eddieyoon

Mime
View raw message