hama-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Suraj Menon <surajsme...@apache.org>
Subject Re: Load user library to Hama
Date Wed, 12 Dec 2012 15:16:02 GMT
Hi, I think you should be able to handle this with 0.6 release using
Distributed cache. Did the new release work for you?

-Suraj

On Wed, Nov 14, 2012 at 9:04 PM, Edward J. Yoon <edwardyoon@apache.org>wrote:

> Can you provide me your tasklogs?
>
> > I am writing my own VertexInputReader and need a third party library to
> > parse the input.
>
> If you added your jars to {$HAMA_HOME}/lib folder (all hosts), you can
> check whether your jars are included as following command.
>
> [oracle@bda01 hama-0.6.0]$ ps -ef | grep Groom
>
> oracle   29928     1 23 17:25 ?        00:00:01
> /usr/java/jdk1.6.0_29/bin/java -Xmx2000m
> -Dhama.log.dir=/home/oracle/hama-0.6.0/bin/../logs
> -Dhama.log.file=hama-oracle-groom-bda01.oracle.com.log
> -Dhama.home.dir=/home/oracle/hama-0.6.0/bin/.. -Dhama.id.str=oracle
> -Dhama.root.logger=INFO,DRFA -Dhama.policy.file=hama-policy.xml
> -classpath
> /home/oracle/hama-0.6.0/bin/../conf:/usr/java/jdk1.6.0_29/lib/tools.jar:/home/oracle/hama-0.6.0/bin/../hama-core-0.6.0.jar:/home/oracle/hama-0.6.0/bin/../hama-examples-0.6.0.jar:/home/oracle/hama-0.6.0/bin/../hama-graph-0.6.0.jar:/home/oracle/hama-0.6.0/bin/../hama-ml-0.6.0.jar:/home/oracle/hama-0.6.0/bin/../lib/ant-1.7.1.jar:/home/oracle/hama-0.6.0/bin/../lib/ant-launcher-1.7.1.jar:/home/oracle/hama-0.6.0/bin/../lib/avro-1.6.0.jar:/home/oracle/hama-0.6.0/bin/../lib/avro-ipc-1.6.0.jar:/home/oracle/hama-0.6.0/bin/../lib/commons-cli-1.2.jar:/home/oracle/hama-0.6.0/bin/../lib/commons-configuration-1.7.jar:/home/oracle/hama-0.6.0/bin/../lib/commons-httpclient-3.0.1.jar:/home/oracle/hama-0.6.0/bin/../lib/commons-lang-2.6.jar:/home/oracle/hama-0.6.0/bin/../lib/commons-logging-1.1.1.jar:/home/oracle/hama-0.6.0/bin/../lib/commons-math3-3.0.jar:/home/oracle/hama-0.6.0/bin/../lib/guava-10.0.1.jar:/home/oracle/hama-0.6.0/bin/../lib/guava-r09-jarjar.jar:/home/oracle/hama-0.6.0/bin/../lib/hadoop-core-0.20.2-cdh3u3b.jar:/home/oracle/hama-0.6.0/bin/../lib/hadoop-test-0.20.2-cdh3u3b.jar:/home/oracle/hama-0.6.0/bin/../lib/jackson-core-asl-1.9.2.jar:/home/oracle/hama-0.6.0/bin/../lib/jackson-mapper-asl-1.9.2.jar:/home/oracle/hama-0.6.0/bin/../lib/jetty-6.1.14.jar:/home/oracle/hama-0.6.0/bin/../lib/jetty-annotations-6.1.14.jar:/home/oracle/hama-0.6.0/bin/../lib/jetty-util-6.1.14.jar:/home/oracle/hama-0.6.0/bin/../lib/jsp-2.1-6.1.14.jar:/home/oracle/hama-0.6.0/bin/../lib/jsp-api-2.1-6.1.14.jar:/home/oracle/hama-0.6.0/bin/../lib/junit-4.8.1.jar:/home/oracle/hama-0.6.0/bin/../lib/log4j-1.2.16.jar:/home/oracle/hama-0.6.0/bin/../lib/netty-3.2.6.Final.jar:/home/oracle/hama-0.6.0/bin/../lib/servlet-api-6.0.32.jar:/home/oracle/hama-0.6.0/bin/../lib/slf4j-api-1.5.8.jar:/home/oracle/hama-0.6.0/bin/../lib/slf4j-log4j12-1.5.8.jar:/home/oracle/hama-0.6.0/bin/../lib/snappy-java-1.0.4.1.jar:/home/oracle/hama-0.6.0/bin/../lib/zookeeper-3.3.3.jar
> org.apache.hama.GroomServerRunner bda01-adm
> oracle   30029 29694  0 17:25 pts/0    00:00:00 grep Groom
>
>
> On Thu, Nov 15, 2012 at 10:18 AM, Tu, Min <mitu@paypal.com> wrote:
> > Hi Edward and Suraj,
> >
> > Thank you very much for spending time answer my questions. Really
> > appreciated.
> >
> > I have tried to put my library to $HAMA_HOME/lib folder in both BSP and
> > Groom hosts.
> >
> > The BSP server pick it up fine but Groom server still starts without that
> > package in the class path.
> >
> > Any thing I did wrong?
> >
> > Also Is there another way to ship the 3rd party jar without put those
> into
> > the $HAMA_HOME/lib? Like Distributed cache in hadoop?
> >
> > Here are the hama command and partial code :
> >
> > hama jar ./POC-0.0.1-SNAPSHOT.jar
> > com.paypal.hadoop.platform.graphjob.SimpleRiskPropagation
> > /user/mitu/riskData/DW_PAYMENT_SENT/preprocess/output/2nd
> > /user/mitu/riskData/DW_PAYMENT_SENT/propagate/output/2nd
> >
> >
> > --------------------- Main function in the SimpleRiskPropagation.java
> -----
> > public static void main(String... args) throws IOException,
> >         InterruptedException, ClassNotFoundException {
> >
> >                 if (args.length < 2) {
> >                         printUsage();
> >                 }
> >
> >                 HamaConfiguration conf = new HamaConfiguration(new
> Configuration());
> >                 GraphJob job = new GraphJob(conf,
> SimpleRiskPropagation.class);
> >
> >                 // set the defaults
> >                 //job.setMaxIteration(30);
> >                 job.setMaxIteration(3);
> >                 job.setNumBspTask(2);
> >
> >                 if (args.length >= 4)
> >                         job.setNumBspTask(Integer.parseInt(args[3]));
> >                 if (args.length >= 3)
> >                         job.setMaxIteration(Integer.parseInt(args[2]));
> >
> >                 job.setJobName("SimpleRiskPropagation");
> >                 job.setInputPath(new Path(args[0]));
> >                 job.setOutputPath(new Path(args[1]));
> >
> >                 job.setVertexClass(SimpleRiskPropagationVertex.class);
> >                 job.setVertexIDClass(Text.class);
> >                 job.setVertexValueClass(Pairs.class);
> >                 job.setEdgeValueClass(SimpleTransaction.class);
> >
> >                 job.setInputKeyClass(LongWritable.class);
> >                 job.setInputValueClass(Text.class);
> >                 job.setInputFormat(TextInputFormat.class);
> >
> >
> >                 /*
> >                  * this reader class will be set in GraphJob conf with
> key:
> > VERTEX_GRAPH_INPUT_READER
> >                  * and used by class GraphJobRunner 's function
> loadVertices()
> >                  */
> >
> job.setVertexInputReaderClass(SimpleRiskPropagationVertexReader.class);
> >
> >
> >                 job.setPartitioner(HashPartitioner.class);
> >                 job.setOutputFormat(TextOutputFormat.class);
> >                 job.setOutputKeyClass(Text.class);
> >                 job.setOutputValueClass(Text.class);
> >
> >                 long startTime = System.currentTimeMillis();
> >                 if (job.waitForCompletion(true)) {
> >                         System.out.println("Job Finished in "
> >                                         + (System.currentTimeMillis() -
> startTime) / 1000.0 + " seconds");
> >                 }
> >         }
> > --------------------- Main function in the SimpleRiskPropagation.java
> -----
> >
> >
> >
> > On 11/13/12 6:13 AM, "Edward J. Yoon" <edwardyoon@apache.org> wrote:
> >
> >>Did you subscribe?
> >>
> >>---------- Forwarded message ----------
> >>From: Edward J. Yoon <edwardyoon@apache.org>
> >>Date: Tue, Nov 13, 2012 at 7:42 PM
> >>Subject: Re: Load user library to Hama
> >>To: user@hama.apache.org
> >>
> >>
> >>Hi Tu, Min
> >>
> >>You can solve the problem by copying your 3rd party library to
> >>{$HAMA}/lib folder.
> >>
> >>On Tue, Nov 13, 2012 at 7:32 PM, Suraj Menon <menonsuraj5@gmail.com>
> >>wrote:
> >>> Hi,
> >>>
> >>> Can you give us the command you are trying. Also, you can try the
> >>> distributed cache feature of Hadoop for the same.
> >>>
> >>> Thanks,
> >>> Suraj
> >>>
> >>>
> >>> On Mon, Nov 12, 2012 at 9:03 PM, Tu, Min <mitu@paypal.com> wrote:
> >>>
> >>>> Hi,
> >>>>
> >>>> I am new to Hama and got a question for using it.
> >>>>
> >>>> I am writing my own VertexInputReader and need a third party library
> to
> >>>> parse the input. How should I ship the library jar to the GroomServer
> >>>>in
> >>>> Hama. I have tried to use the ­libjars option but did not work: got
> >>>> ClassNotFoundException in the VertexInputReader part.
> >>>>
> >>>>
> >>>> Thanks a lot
> >>>>
> >>>> Min
> >>>>
> >>
> >>
> >>
> >>--
> >>Best Regards, Edward J. Yoon
> >>@eddieyoon
> >>
> >>
> >>--
> >>Best Regards, Edward J. Yoon
> >>@eddieyoon
> >
>
>
>
> --
> Best Regards, Edward J. Yoon
> @eddieyoon
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message