hama-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stratos Dimopoulos <stratos.dimopou...@gmail.com>
Subject Re: Using hama with Mesos 0.20 and CDH5.1.2
Date Sat, 08 Nov 2014 09:51:53 GMT
Hi Jeff,

Just wanted to let you know that your patch worked on my setup. Since
lately I am doing a lot of installations I was keeping some notes and now I
polished them and published a blog post (
https://strat0sphere.wordpress.com/2014/11/08/apache-hama-on-mesos/) on
which I am including some errors I encountered on the process as a
non-expert mesos/hama user and the configuration I did to make it work.
Also, if you think it would be useful to update the documentation (
http://wiki.apache.org/hama/GettingStartedMesos) on the Hama website,
mainly the hama-site.xml file, please let me know and I'll be happy doing
it as my first tiny contribution to the project.

I also noticed that all the links provided on the website are for the older
0.6.4 Hama version, which doesn't support Mesos. Not sure if this is
intentional or just a mistake.

Again, I really appreciated your help and thanks a ton for doing the fix in
such a timely manner!

cheers,
Stratos

On Mon, Oct 27, 2014 at 12:35 AM, Stratos Dimopoulos <
stratos.dimopoulos@gmail.com> wrote:

> Great thank you! Looking forward for the patch!
> On Oct 26, 2014 10:30 PM, "Jeff Fenchel" <jfenc91@gmail.com> wrote:
>
>> Stratos,
>>
>> Thanks for sending me the stack trace. I was able to replicate your
>> error. So, it definitely appears to be a bug in Hama. I will open up a
>> ticket and see if I can put together a patch for this sometime this
>> week. In the mean time, Hama without mesos seems to work fine with
>> cdh5.
>>
>> -Jeff
>>
>> On Fri, Oct 24, 2014 at 8:20 PM, Stratos Dimopoulos
>> <stratos.dimopoulos@gmail.com> wrote:
>> > And when you added the list's user email you had a typo which I copy
>> pasted
>> > and the email never reached the list - So I am just forwarding again to
>> the
>> > correct address this time! :-)
>> >
>> >
>> > On Thu, Oct 23, 2014 at 5:36 PM, Stratos Dimopoulos
>> > <stratos.dimopoulos@gmail.com> wrote:
>> >>
>> >> Yes Jeff correct - this is the way I compile. Here is the stack trace
>> >> close to the error on the executor - I don't think the log file on the
>> BSP
>> >> master would help you at all but let me know if you need to send you
>> >> anything else:
>> >>
>> >> I1022 17:25:31.828336 22440 exec.cpp:132] Version: 0.20.0
>> >> I1022 17:25:31.838382 22454 exec.cpp:206] Executor registered on slave
>> >> 20141019-011259-1257767434-5050-4023-1
>> >> 14/10/22 17:25:31 INFO bsp.MesosExecutor: Executor registered with the
>> >> slave
>> >> 14/10/22 17:25:31 INFO bsp.MesosExecutor: Launching task : Task_0
>> >> 14/10/22 17:25:32 INFO bsp.GroomServer: groom start
>> >> 14/10/22 17:25:32 INFO bsp.GroomServer:
>> >>
>> /mnt/mesos-work-dir/slaves/20141019-011259-1257767434-5050-4023-1/frameworks/20141019-011259-1257767434-5050-4023-0031/executors/executor_Task_0/runs/d69f21af-9180-4c0c-9263-82038ae81937/hama-0.7.0-SNAPSHOT/bsp/local
>> >> 14/10/22 17:25:32 WARN util.NativeCodeLoader: Unable to load
>> native-hadoop
>> >> library for your platform... using builtin-java classes where
>> applicable
>> >> 14/10/22 17:25:33 ERROR bsp.MesosExecutor: Caught exception, committing
>> >> suicide.
>> >> java.util.ServiceConfigurationError: org.apache.hadoop.fs.FileSystem:
>> >> Provider org.apache.hadoop.fs.LocalFileSystem not found
>> >>         at java.util.ServiceLoader.fail(ServiceLoader.java:231)
>> >>         at java.util.ServiceLoader.access$300(ServiceLoader.java:181)
>> >>         at
>> >> java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:365)
>> >>         at java.util.ServiceLoader$1.next(ServiceLoader.java:445)
>> >>         at
>> >> org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2364)
>> >>         at
>> >>
>> org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2375)
>> >>         at
>> >> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2392)
>> >>         at
>> org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:89)
>> >>         at
>> >> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2431)
>> >>         at
>> org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2413)
>> >>         at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:368)
>> >>         at
>> org.apache.hadoop.fs.FileSystem.getLocal(FileSystem.java:339)
>> >>         at
>> >> org.apache.hama.bsp.GroomServer.deleteLocalFiles(GroomServer.java:483)
>> >>         at
>> >> org.apache.hama.bsp.GroomServer.initialize(GroomServer.java:321)
>> >>         at org.apache.hama.bsp.GroomServer.run(GroomServer.java:860)
>> >>         at
>> org.apache.hama.bsp.MesosExecutor$1.run(MesosExecutor.java:92)
>> >>
>> >> thanks,
>> >> Stratos
>> >>
>> >> On Thu, Oct 23, 2014 at 2:51 PM, Jeff Fenchel <jfenc91@gmail.com>
>> wrote:
>> >>>
>> >>> Fyi... Sorry I forgot to add your email to list. Please add
>> >>> user@hama.apach.org to your reply.
>> >>>
>> >>> Thanks,
>> >>> Jeff
>> >>>
>> >>> ---------- Forwarded message ----------
>> >>> From: Jeff Fenchel <jfenc91@gmail.com>
>> >>> Date: Thursday, October 23, 2014
>> >>> Subject: Using hama with Mesos 0.20 and CDH5.1.2
>> >>> To: user@hama.apache.org
>> >>>
>> >>>
>> >>> Hey Stratos,
>> >>>
>> >>> Thanks for giving Hama on Mesos a try. I do not think it has ever been
>> >>> tested with hadoop 2, but I see no reason why it shouldn't work. So,
>> I am
>> >>>
>> >>> guessing you compiled with something like the following?
>> >>>
>> >>> mvn clean install -Phadoop2 -Dhadoop.version=2.3.0-cdh5.1.2
>> >>> -Dmesos.version=0.20.0 -DskipTests
>> >>>
>> >>> If so, can you please send us the complete stack trace? It does look
>> >>> like this might be a bug in hama.
>> >>>
>> >>> Thanks,
>> >>> -Jeff
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> On Thu, Oct 23, 2014 at 1:13 AM, Stratos Dimopoulos
>> >>> <stratos.dimopoulos@gmail.com> wrote:
>> >>> > Hi,
>> >>> >
>> >>> > I cloned Hama from Git and installed in a Eucalyptus cloud with
>> Mesos
>> >>> > 0.20
>> >>> > and CDH5.1.2 HDFS. It is also configured with zookeeper but my
>> >>> > zookeeper at
>> >>> > the moment just runs in one master. I followed the instructions
>> here:
>> >>> > http://wiki.apache.org/hama/GettingStartedMesos - that are for
an
>> older
>> >>> > version and additionally to that I added some more properties to
>> >>> > overcome
>> >>> > some runtime problems and until Mesos was able to recognize Hama
as
>> a
>> >>> > framework. Overall the properties I have set are the following:
>> >>> >
>> >>> > bsp.master.address
>> >>> > bsp.master.port
>> >>> > bsp.master.TaskWorkerManager.class
>> >>> > fs.default.name (I don't think it needs to be here but in an
>> effort to
>> >>> > fix
>> >>> > the issues I had)
>> >>> > hama.mesos.executor.uri
>> >>> > bsp.tasks.maximum.total
>> >>> > hama.mesos.master
>> >>> > bsp.child.java.opts
>> >>> > bsp.system.dir
>> >>> > bsp.local.dir
>> >>> > bsp.tmp.dir
>> >>> > bsp.disk.queue.dir
>> >>> > hama.zookeeper.quorum
>> >>> > hama.zookeeper.property.clientPort
>> >>> >
>> >>> > When I run the example the executor is running but the task fails.
>> The
>> >>> > error output on the executors log is: ERROR bsp.MesosExecutor:
>> Caught
>> >>> > exception, committing suicide.
>> >>> > java.util.ServiceConfigurationError:
>> org.apache.hadoop.fs.FileSystem:
>> >>> > Provider org.apache.hadoop.fs.LocalFileSystem not found
>> >>> >
>> >>> > I did some research on this error message but I cannot find
>> something
>> >>> > that
>> >>> > really helps. It seems to me as a configuration issue but I don't
>> know
>> >>> > how
>> >>> > to solve it.
>> >>> > Also note that I have verified that my HDFS works properly. I am
>> >>> > actually
>> >>> > already running Spark and Hadoop over Mesos without problems.
>> >>> >
>> >>> > I hope somebody can help!
>> >>> >
>> >>> > thanks,
>> >>> > Stratos
>> >>>
>> >>
>> >
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message