hama-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Roman Shapovalov <shapova...@graphics.cs.msu.su>
Subject Re: Problem initializing pipes in HamaStreaming
Date Thu, 26 Sep 2013 20:47:18 GMT
Martin,

Thanks for such verbose instructions.

> You can find all Hama configuration files in the *conf* folder.

OK, I thought Edward meant Hadoop configs specifically.
I have only added JAVA_HOME variable there, otherwise they are default.

> You should also find task logs in your *temp* folder.

I found the folder, but there were no .log files in the attempt*
folders (in both modes).

> Normally you should find it in *hama/logs/tasklogs*.

They appear in the pseudo-distributed mode only (which also fails).
See the attached file.

> By the way do you have python3.2 installed? :-)

Yes. "python" links to Python 2.6, but I pass "python3.2" as an
interpreter, which links to the correct version.


Roman

On Thu, Sep 26, 2013 at 4:03 PM, Martin Illecker <millecker@apache.org> wrote:
> Hi Roman,
>
> if you are running Hama in local mode, it will not use HDFS anyway.
>
> You can find all Hama configuration files in the *conf* folder.
>
> $ll hama/conf/
> total 56
> -rwxr-xr-x groomservers*
> -rwxr-xr-x hama-default.xml*
> -rwxr-xr-x hama-env.sh*
> -rwxr-xr-x hama-site.xml*
> -rwxr-xr-x log4j.properties*
>
> Probably you should setup the Pseudo Distributed Mode [1] in hama-site.xml.
>
> But the task log would be very interesting.
>
> Normally you should find it in *hama/logs/tasklogs*.
> e.g., hama/logs/tasklogs/job_201309262134_0001/attempt_201309262134_0001_000000_0.log
>
> You should also find task logs in your *temp* folder.
> But this location will depend on your operation system.
> e.g., in OSX
> /private/tmp/hadoop-YOURUSER/bsp/local/groomServer/attempt_201309262134_0001_000000_0/work/tasklogs/
>
> By the way do you have python3.2 installed? :-)
> $ python --version
> Python 3.2.5
> $ python3.2 --version
> Python 3.2.5
>
> May I ask which operation system do you use?
>
> Martin
>
> [1] http://wiki.apache.org/hama/GettingStarted#Pseudo_Distributed_Mode
>
>
>
> 2013/9/26 Roman Shapovalov <shapovalov@graphics.cs.msu.su>
>
>> Hi Edward,
>>
>> Could you please be more specific? (Sorry, I am new to this stuff)
>>
>> I run Hama in local mode. The logs/ directory is empty, and I did not
>> find any logs in HDFS as well.
>>
>> And where can I find the Hadoop configuration?
>>
>> Thank you,
>> Roman
>>
>> On Thu, Sep 26, 2013 at 12:05 PM, Edward J. Yoon <edwardyoon@apache.org>
>> wrote:
>> > Hi,
>> >
>> > That's strange. Can you attach your namenode logs and hadoop
>> configurations?
>> >
>> > On Thu, Sep 26, 2013 at 11:03 PM, Roman Shapovalov
>> > <shapovalov@graphics.cs.msu.su> wrote:
>> >> Hi again,
>> >>
>> >> I have updated both Hama (from the trunk) and Streaming (from Martin's
>> >> github), and checked that patches have been applied, but I keep
>> >> getting the same error (full log for local configuration is attached).
>> >>
>> >> Another thing may be relevant: I keep the default Hadoop libraries in
>> >> lib/. If I replace them as the tutorial says, some classes cannot be
>> >> found even if  I run pure Hama (which works perfectly with default
>> >> libs). I don't know if it is important.
>> >>
>> >> Thanks,
>> >> Roman
>> >>
>> >> On Tue, Sep 24, 2013 at 9:22 AM, Martin Illecker <millecker@apache.org>
>> wrote:
>> >>> Hi Roman,
>> >>>
>> >>> sorry for inconvenience!
>> >>> The problem has been reported [1] and will be fixed shortly to the
>> trunk.
>> >>>
>> >>> [1] https://issues.apache.org/jira/browse/HAMA-805
>> >>>
>> >>>
>> >>> 2013/9/23 Edward J. Yoon <edwardyoon@apache.org>
>> >>>
>> >>>> This looks like a bug of DistCacheUtils.
>> >>>>
>> >>>> Thanks for your report. I'll look at it tomorrow.
>> >>>>
>> >>>> On Mon, Sep 23, 2013 at 11:52 PM, Roman Shapovalov
>> >>>> <shapovalov@graphics.cs.msu.su> wrote:
>> >>>> > Hello all,
>> >>>> >
>> >>>> > I try to use Hama Streaming.
>> >>>> > I have successfully installed Hama (the Pi example works).
>> >>>> > I follow this tutorial:
>> >>>> > http://wiki.apache.org/hama/HamaStreaming
>> >>>> >
>> >>>> > When I try to run the distributed HelloWorld in the local
>> >>>> > configuration, I get the following error:
>> >>>> >
>> >>>> > $ bin/hama pipes -streaming true -bspTasks 3 -interpreter python3.2
>> >>>> > -cachefiles /tmp/PyStreaming/*.py -output /tmp/pystream-out/
>> -program
>> >>>> > /tmp/PyStreaming/BSPRunner.py -programArgs HelloWorldBSP
>> >>>> >
>> >>>> > 13/09/23 18:03:50 INFO pipes.Submitter: Streaming enabled!
>> >>>> > 13/09/23 18:03:50 WARN util.NativeCodeLoader: Unable to load
>> >>>> > native-hadoop library for your platform... using builtin-java
>> classes
>> >>>> > where applicable
>> >>>> > 13/09/23 18:03:50 WARN bsp.BSPJobClient: No job jar file set.
 User
>> >>>> > classes may not be found. See BSPJob#setJar(String) or check
Your
>> jar
>> >>>> > file.
>> >>>> > 13/09/23 18:03:50 INFO bsp.BSPJobClient: Running job:
>> >>>> job_localrunner_0001
>> >>>> > 13/09/23 18:03:50 INFO bsp.LocalBSPRunner: Setting up a new
barrier
>> for
>> >>>> 3 tasks!
>> >>>> > 13/09/23 18:03:50 ERROR bsp.LocalBSPRunner: Exception during
BSP
>> >>>> execution!
>> >>>> > java.lang.NullPointerException
>> >>>> >     at org.apache.hama.pipes.PipesBSP.setup(PipesBSP.java:44)
>> >>>> >     at
>> >>>>
>> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.run(LocalBSPRunner.java:255)
>> >>>> >     at
>> >>>>
>> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:286)
>> >>>> >     at
>> >>>>
>> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:211)
>> >>>> >     at
>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>> >>>> >     at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>> >>>> >     at
>> >>>>
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>> >>>> >     at
>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>> >>>> >     at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>> >>>> >     at
>> >>>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>> >>>> >     at
>> >>>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>> >>>> >     at java.lang.Thread.run(Thread.java:662)
>> >>>> > [output cropped]
>> >>>> >
>> >>>> > When I turn to the pseudo-distributed mode, job fails too (after
a
>> >>>> > minute of execution):
>> >>>> >
>> >>>> > 13/09/23 18:46:34 INFO pipes.Submitter: Streaming enabled!
>> >>>> > 13/09/23 18:46:34 WARN util.NativeCodeLoader: Unable to load
>> >>>> > native-hadoop library for your platform... using builtin-java
>> classes
>> >>>> > where applicable
>> >>>> > 13/09/23 18:46:34 WARN bsp.BSPJobClient: No job jar file set.
 User
>> >>>> > classes may not be found. See BSPJob#setJar(String) or check
Your
>> jar
>> >>>> > file.
>> >>>> > 13/09/23 18:46:34 INFO bsp.BSPJobClient: Running job:
>> >>>> job_201309231846_0001
>> >>>> > 13/09/23 18:47:40 INFO bsp.BSPJobClient: Job failed.
>> >>>> >
>> >>>> > Task log contains errors:
>> >>>> >
>> >>>> > 13/09/23 18:46:37 INFO ipc.Server: Starting Socket Reader #1
for
>> port
>> >>>> 43475
>> >>>> > 13/09/23 18:46:37 INFO ipc.Server: IPC Server Responder: starting
>> >>>> > 13/09/23 18:46:37 INFO ipc.Server: IPC Server listener on 43475:
>> starting
>> >>>> > 13/09/23 18:46:37 INFO message.HadoopMessageManagerImpl:  BSPPeer
>> >>>> > address:localhost.localdomain port:43475
>> >>>> > 13/09/23 18:46:37 INFO ipc.Server: IPC Server handler 0 on
43475:
>> >>>> starting
>> >>>> > 13/09/23 18:46:37 WARN util.NativeCodeLoader: Unable to load
>> >>>> > native-hadoop library for your platform... using builtin-java
>> classes
>> >>>> > where applicable
>> >>>> > 13/09/23 18:46:37 INFO sync.ZKSyncClient: Initializing ZK Sync
>> Client
>> >>>> > 13/09/23 18:46:37 INFO sync.ZooKeeperSyncClientImpl: Start
>> connecting
>> >>>> > to Zookeeper! At localhost.localdomain/127.0.0.1:43475
>> >>>> > 13/09/23 18:46:37 ERROR bsp.BSPTask: Error running bsp setup
and bsp
>> >>>> function.
>> >>>> > java.lang.NullPointerException
>> >>>> >     at java.io.File.<init>(File.java:222)
>> >>>> >     at
>> >>>>
>> org.apache.hama.pipes.PipesApplication.setupCommand(PipesApplication.java:130)
>> >>>> >     at
>> >>>>
>> org.apache.hama.pipes.PipesApplication.start(PipesApplication.java:257)
>> >>>> >     at org.apache.hama.pipes.PipesBSP.setup(PipesBSP.java:44)
>> >>>> >     at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:176)
>> >>>> >     at org.apache.hama.bsp.BSPTask.run(BSPTask.java:146)
>> >>>> >     at
>> >>>>
>> org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1246)
>> >>>> > [output cropped]
>> >>>> >
>> >>>> > I use the latest trunk version of Hama, Python 3.2.5 and Hadoop
>> >>>> 2.0.0-cdh4.1.1.
>> >>>> >
>> >>>> > Please help me to figure out the problem.
>> >>>> >
>> >>>> > Thanks in advance,
>> >>>> > Roman
>> >>>>
>> >>>>
>> >>>>
>> >>>> --
>> >>>> Best Regards, Edward J. Yoon
>> >>>> @eddieyoon
>> >>>>
>> >
>> >
>> >
>> > --
>> > Best Regards, Edward J. Yoon
>> > @eddieyoon
>>

Mime
View raw message