hama-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Fenchel <jfen...@gmail.com>
Subject Re: Using hama with Mesos 0.20 and CDH5.1.2
Date Mon, 27 Oct 2014 05:30:04 GMT
Stratos,

Thanks for sending me the stack trace. I was able to replicate your
error. So, it definitely appears to be a bug in Hama. I will open up a
ticket and see if I can put together a patch for this sometime this
week. In the mean time, Hama without mesos seems to work fine with
cdh5.

-Jeff

On Fri, Oct 24, 2014 at 8:20 PM, Stratos Dimopoulos
<stratos.dimopoulos@gmail.com> wrote:
> And when you added the list's user email you had a typo which I copy pasted
> and the email never reached the list - So I am just forwarding again to the
> correct address this time! :-)
>
>
> On Thu, Oct 23, 2014 at 5:36 PM, Stratos Dimopoulos
> <stratos.dimopoulos@gmail.com> wrote:
>>
>> Yes Jeff correct - this is the way I compile. Here is the stack trace
>> close to the error on the executor - I don't think the log file on the BSP
>> master would help you at all but let me know if you need to send you
>> anything else:
>>
>> I1022 17:25:31.828336 22440 exec.cpp:132] Version: 0.20.0
>> I1022 17:25:31.838382 22454 exec.cpp:206] Executor registered on slave
>> 20141019-011259-1257767434-5050-4023-1
>> 14/10/22 17:25:31 INFO bsp.MesosExecutor: Executor registered with the
>> slave
>> 14/10/22 17:25:31 INFO bsp.MesosExecutor: Launching task : Task_0
>> 14/10/22 17:25:32 INFO bsp.GroomServer: groom start
>> 14/10/22 17:25:32 INFO bsp.GroomServer:
>> /mnt/mesos-work-dir/slaves/20141019-011259-1257767434-5050-4023-1/frameworks/20141019-011259-1257767434-5050-4023-0031/executors/executor_Task_0/runs/d69f21af-9180-4c0c-9263-82038ae81937/hama-0.7.0-SNAPSHOT/bsp/local
>> 14/10/22 17:25:32 WARN util.NativeCodeLoader: Unable to load native-hadoop
>> library for your platform... using builtin-java classes where applicable
>> 14/10/22 17:25:33 ERROR bsp.MesosExecutor: Caught exception, committing
>> suicide.
>> java.util.ServiceConfigurationError: org.apache.hadoop.fs.FileSystem:
>> Provider org.apache.hadoop.fs.LocalFileSystem not found
>>         at java.util.ServiceLoader.fail(ServiceLoader.java:231)
>>         at java.util.ServiceLoader.access$300(ServiceLoader.java:181)
>>         at
>> java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:365)
>>         at java.util.ServiceLoader$1.next(ServiceLoader.java:445)
>>         at
>> org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2364)
>>         at
>> org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2375)
>>         at
>> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2392)
>>         at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:89)
>>         at
>> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2431)
>>         at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2413)
>>         at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:368)
>>         at org.apache.hadoop.fs.FileSystem.getLocal(FileSystem.java:339)
>>         at
>> org.apache.hama.bsp.GroomServer.deleteLocalFiles(GroomServer.java:483)
>>         at
>> org.apache.hama.bsp.GroomServer.initialize(GroomServer.java:321)
>>         at org.apache.hama.bsp.GroomServer.run(GroomServer.java:860)
>>         at org.apache.hama.bsp.MesosExecutor$1.run(MesosExecutor.java:92)
>>
>> thanks,
>> Stratos
>>
>> On Thu, Oct 23, 2014 at 2:51 PM, Jeff Fenchel <jfenc91@gmail.com> wrote:
>>>
>>> Fyi... Sorry I forgot to add your email to list. Please add
>>> user@hama.apach.org to your reply.
>>>
>>> Thanks,
>>> Jeff
>>>
>>> ---------- Forwarded message ----------
>>> From: Jeff Fenchel <jfenc91@gmail.com>
>>> Date: Thursday, October 23, 2014
>>> Subject: Using hama with Mesos 0.20 and CDH5.1.2
>>> To: user@hama.apache.org
>>>
>>>
>>> Hey Stratos,
>>>
>>> Thanks for giving Hama on Mesos a try. I do not think it has ever been
>>> tested with hadoop 2, but I see no reason why it shouldn't work. So, I am
>>>
>>> guessing you compiled with something like the following?
>>>
>>> mvn clean install -Phadoop2 -Dhadoop.version=2.3.0-cdh5.1.2
>>> -Dmesos.version=0.20.0 -DskipTests
>>>
>>> If so, can you please send us the complete stack trace? It does look
>>> like this might be a bug in hama.
>>>
>>> Thanks,
>>> -Jeff
>>>
>>>
>>>
>>>
>>> On Thu, Oct 23, 2014 at 1:13 AM, Stratos Dimopoulos
>>> <stratos.dimopoulos@gmail.com> wrote:
>>> > Hi,
>>> >
>>> > I cloned Hama from Git and installed in a Eucalyptus cloud with Mesos
>>> > 0.20
>>> > and CDH5.1.2 HDFS. It is also configured with zookeeper but my
>>> > zookeeper at
>>> > the moment just runs in one master. I followed the instructions here:
>>> > http://wiki.apache.org/hama/GettingStartedMesos - that are for an older
>>> > version and additionally to that I added some more properties to
>>> > overcome
>>> > some runtime problems and until Mesos was able to recognize Hama as a
>>> > framework. Overall the properties I have set are the following:
>>> >
>>> > bsp.master.address
>>> > bsp.master.port
>>> > bsp.master.TaskWorkerManager.class
>>> > fs.default.name (I don't think it needs to be here but in an effort to
>>> > fix
>>> > the issues I had)
>>> > hama.mesos.executor.uri
>>> > bsp.tasks.maximum.total
>>> > hama.mesos.master
>>> > bsp.child.java.opts
>>> > bsp.system.dir
>>> > bsp.local.dir
>>> > bsp.tmp.dir
>>> > bsp.disk.queue.dir
>>> > hama.zookeeper.quorum
>>> > hama.zookeeper.property.clientPort
>>> >
>>> > When I run the example the executor is running but the task fails. The
>>> > error output on the executors log is: ERROR bsp.MesosExecutor: Caught
>>> > exception, committing suicide.
>>> > java.util.ServiceConfigurationError: org.apache.hadoop.fs.FileSystem:
>>> > Provider org.apache.hadoop.fs.LocalFileSystem not found
>>> >
>>> > I did some research on this error message but I cannot find something
>>> > that
>>> > really helps. It seems to me as a configuration issue but I don't know
>>> > how
>>> > to solve it.
>>> > Also note that I have verified that my HDFS works properly. I am
>>> > actually
>>> > already running Spark and Hadoop over Mesos without problems.
>>> >
>>> > I hope somebody can help!
>>> >
>>> > thanks,
>>> > Stratos
>>>
>>
>

Mime
View raw message