hama-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Edward J. Yoon" <edwardy...@apache.org>
Subject Re: Using hama with Mesos 0.20 and CDH5.1.2
Date Wed, 12 Nov 2014 09:06:53 GMT
Awesome! Please feel free to link to your blog post on
GettingStartedMesos wiki page, or update it. If you need edit
permission let us know your wiki ID.

On Sat, Nov 8, 2014 at 6:51 PM, Stratos Dimopoulos
<stratos.dimopoulos@gmail.com> wrote:
> Hi Jeff,
>
> Just wanted to let you know that your patch worked on my setup. Since
> lately I am doing a lot of installations I was keeping some notes and now I
> polished them and published a blog post (
> https://strat0sphere.wordpress.com/2014/11/08/apache-hama-on-mesos/) on
> which I am including some errors I encountered on the process as a
> non-expert mesos/hama user and the configuration I did to make it work.
> Also, if you think it would be useful to update the documentation (
> http://wiki.apache.org/hama/GettingStartedMesos) on the Hama website,
> mainly the hama-site.xml file, please let me know and I'll be happy doing
> it as my first tiny contribution to the project.
>
> I also noticed that all the links provided on the website are for the older
> 0.6.4 Hama version, which doesn't support Mesos. Not sure if this is
> intentional or just a mistake.
>
> Again, I really appreciated your help and thanks a ton for doing the fix in
> such a timely manner!
>
> cheers,
> Stratos
>
> On Mon, Oct 27, 2014 at 12:35 AM, Stratos Dimopoulos <
> stratos.dimopoulos@gmail.com> wrote:
>
>> Great thank you! Looking forward for the patch!
>> On Oct 26, 2014 10:30 PM, "Jeff Fenchel" <jfenc91@gmail.com> wrote:
>>
>>> Stratos,
>>>
>>> Thanks for sending me the stack trace. I was able to replicate your
>>> error. So, it definitely appears to be a bug in Hama. I will open up a
>>> ticket and see if I can put together a patch for this sometime this
>>> week. In the mean time, Hama without mesos seems to work fine with
>>> cdh5.
>>>
>>> -Jeff
>>>
>>> On Fri, Oct 24, 2014 at 8:20 PM, Stratos Dimopoulos
>>> <stratos.dimopoulos@gmail.com> wrote:
>>> > And when you added the list's user email you had a typo which I copy
>>> pasted
>>> > and the email never reached the list - So I am just forwarding again to
>>> the
>>> > correct address this time! :-)
>>> >
>>> >
>>> > On Thu, Oct 23, 2014 at 5:36 PM, Stratos Dimopoulos
>>> > <stratos.dimopoulos@gmail.com> wrote:
>>> >>
>>> >> Yes Jeff correct - this is the way I compile. Here is the stack trace
>>> >> close to the error on the executor - I don't think the log file on the
>>> BSP
>>> >> master would help you at all but let me know if you need to send you
>>> >> anything else:
>>> >>
>>> >> I1022 17:25:31.828336 22440 exec.cpp:132] Version: 0.20.0
>>> >> I1022 17:25:31.838382 22454 exec.cpp:206] Executor registered on slave
>>> >> 20141019-011259-1257767434-5050-4023-1
>>> >> 14/10/22 17:25:31 INFO bsp.MesosExecutor: Executor registered with the
>>> >> slave
>>> >> 14/10/22 17:25:31 INFO bsp.MesosExecutor: Launching task : Task_0
>>> >> 14/10/22 17:25:32 INFO bsp.GroomServer: groom start
>>> >> 14/10/22 17:25:32 INFO bsp.GroomServer:
>>> >>
>>> /mnt/mesos-work-dir/slaves/20141019-011259-1257767434-5050-4023-1/frameworks/20141019-011259-1257767434-5050-4023-0031/executors/executor_Task_0/runs/d69f21af-9180-4c0c-9263-82038ae81937/hama-0.7.0-SNAPSHOT/bsp/local
>>> >> 14/10/22 17:25:32 WARN util.NativeCodeLoader: Unable to load
>>> native-hadoop
>>> >> library for your platform... using builtin-java classes where
>>> applicable
>>> >> 14/10/22 17:25:33 ERROR bsp.MesosExecutor: Caught exception, committing
>>> >> suicide.
>>> >> java.util.ServiceConfigurationError: org.apache.hadoop.fs.FileSystem:
>>> >> Provider org.apache.hadoop.fs.LocalFileSystem not found
>>> >>         at java.util.ServiceLoader.fail(ServiceLoader.java:231)
>>> >>         at java.util.ServiceLoader.access$300(ServiceLoader.java:181)
>>> >>         at
>>> >> java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:365)
>>> >>         at java.util.ServiceLoader$1.next(ServiceLoader.java:445)
>>> >>         at
>>> >> org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2364)
>>> >>         at
>>> >>
>>> org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2375)
>>> >>         at
>>> >> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2392)
>>> >>         at
>>> org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:89)
>>> >>         at
>>> >> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2431)
>>> >>         at
>>> org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2413)
>>> >>         at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:368)
>>> >>         at
>>> org.apache.hadoop.fs.FileSystem.getLocal(FileSystem.java:339)
>>> >>         at
>>> >> org.apache.hama.bsp.GroomServer.deleteLocalFiles(GroomServer.java:483)
>>> >>         at
>>> >> org.apache.hama.bsp.GroomServer.initialize(GroomServer.java:321)
>>> >>         at org.apache.hama.bsp.GroomServer.run(GroomServer.java:860)
>>> >>         at
>>> org.apache.hama.bsp.MesosExecutor$1.run(MesosExecutor.java:92)
>>> >>
>>> >> thanks,
>>> >> Stratos
>>> >>
>>> >> On Thu, Oct 23, 2014 at 2:51 PM, Jeff Fenchel <jfenc91@gmail.com>
>>> wrote:
>>> >>>
>>> >>> Fyi... Sorry I forgot to add your email to list. Please add
>>> >>> user@hama.apach.org to your reply.
>>> >>>
>>> >>> Thanks,
>>> >>> Jeff
>>> >>>
>>> >>> ---------- Forwarded message ----------
>>> >>> From: Jeff Fenchel <jfenc91@gmail.com>
>>> >>> Date: Thursday, October 23, 2014
>>> >>> Subject: Using hama with Mesos 0.20 and CDH5.1.2
>>> >>> To: user@hama.apache.org
>>> >>>
>>> >>>
>>> >>> Hey Stratos,
>>> >>>
>>> >>> Thanks for giving Hama on Mesos a try. I do not think it has ever
been
>>> >>> tested with hadoop 2, but I see no reason why it shouldn't work.
So,
>>> I am
>>> >>>
>>> >>> guessing you compiled with something like the following?
>>> >>>
>>> >>> mvn clean install -Phadoop2 -Dhadoop.version=2.3.0-cdh5.1.2
>>> >>> -Dmesos.version=0.20.0 -DskipTests
>>> >>>
>>> >>> If so, can you please send us the complete stack trace? It does
look
>>> >>> like this might be a bug in hama.
>>> >>>
>>> >>> Thanks,
>>> >>> -Jeff
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> On Thu, Oct 23, 2014 at 1:13 AM, Stratos Dimopoulos
>>> >>> <stratos.dimopoulos@gmail.com> wrote:
>>> >>> > Hi,
>>> >>> >
>>> >>> > I cloned Hama from Git and installed in a Eucalyptus cloud
with
>>> Mesos
>>> >>> > 0.20
>>> >>> > and CDH5.1.2 HDFS. It is also configured with zookeeper but
my
>>> >>> > zookeeper at
>>> >>> > the moment just runs in one master. I followed the instructions
>>> here:
>>> >>> > http://wiki.apache.org/hama/GettingStartedMesos - that are
for an
>>> older
>>> >>> > version and additionally to that I added some more properties
to
>>> >>> > overcome
>>> >>> > some runtime problems and until Mesos was able to recognize
Hama as
>>> a
>>> >>> > framework. Overall the properties I have set are the following:
>>> >>> >
>>> >>> > bsp.master.address
>>> >>> > bsp.master.port
>>> >>> > bsp.master.TaskWorkerManager.class
>>> >>> > fs.default.name (I don't think it needs to be here but in an
>>> effort to
>>> >>> > fix
>>> >>> > the issues I had)
>>> >>> > hama.mesos.executor.uri
>>> >>> > bsp.tasks.maximum.total
>>> >>> > hama.mesos.master
>>> >>> > bsp.child.java.opts
>>> >>> > bsp.system.dir
>>> >>> > bsp.local.dir
>>> >>> > bsp.tmp.dir
>>> >>> > bsp.disk.queue.dir
>>> >>> > hama.zookeeper.quorum
>>> >>> > hama.zookeeper.property.clientPort
>>> >>> >
>>> >>> > When I run the example the executor is running but the task
fails.
>>> The
>>> >>> > error output on the executors log is: ERROR bsp.MesosExecutor:
>>> Caught
>>> >>> > exception, committing suicide.
>>> >>> > java.util.ServiceConfigurationError:
>>> org.apache.hadoop.fs.FileSystem:
>>> >>> > Provider org.apache.hadoop.fs.LocalFileSystem not found
>>> >>> >
>>> >>> > I did some research on this error message but I cannot find
>>> something
>>> >>> > that
>>> >>> > really helps. It seems to me as a configuration issue but I
don't
>>> know
>>> >>> > how
>>> >>> > to solve it.
>>> >>> > Also note that I have verified that my HDFS works properly.
I am
>>> >>> > actually
>>> >>> > already running Spark and Hadoop over Mesos without problems.
>>> >>> >
>>> >>> > I hope somebody can help!
>>> >>> >
>>> >>> > thanks,
>>> >>> > Stratos
>>> >>>
>>> >>
>>> >
>>>
>>



-- 
Best Regards, Edward J. Yoon
CEO at DataSayer Co., Ltd.

Mime
View raw message