Return-Path: X-Original-To: apmail-hama-user-archive@www.apache.org Delivered-To: apmail-hama-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id AD956175C4 for ; Wed, 7 Oct 2015 10:50:51 +0000 (UTC) Received: (qmail 81868 invoked by uid 500); 7 Oct 2015 10:50:51 -0000 Delivered-To: apmail-hama-user-archive@hama.apache.org Received: (qmail 81838 invoked by uid 500); 7 Oct 2015 10:50:51 -0000 Mailing-List: contact user-help@hama.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hama.apache.org Delivered-To: mailing list user@hama.apache.org Received: (qmail 81827 invoked by uid 99); 7 Oct 2015 10:50:51 -0000 Received: from mail-relay.apache.org (HELO mail-relay.apache.org) (140.211.11.15) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 07 Oct 2015 10:50:51 +0000 Received: from mail-ig0-f173.google.com (mail-ig0-f173.google.com [209.85.213.173]) by mail-relay.apache.org (ASF Mail Server at mail-relay.apache.org) with ESMTPSA id ECB6D1A006D for ; Wed, 7 Oct 2015 10:50:50 +0000 (UTC) Received: by igbkq10 with SMTP id kq10so108156183igb.0 for ; Wed, 07 Oct 2015 03:50:49 -0700 (PDT) X-Gm-Message-State: ALoCoQkaj7rPLM5JI2kS1/ac981LjQ4+u+SggI5qyBs9Zrno4jHbBxSk+zYh0qXwlEEhHm+zIEop MIME-Version: 1.0 X-Received: by 10.50.61.243 with SMTP id t19mr833264igr.22.1444215049577; Wed, 07 Oct 2015 03:50:49 -0700 (PDT) Received: by 10.64.0.34 with HTTP; Wed, 7 Oct 2015 03:50:49 -0700 (PDT) In-Reply-To: References: Date: Wed, 7 Oct 2015 19:50:49 +0900 Message-ID: Subject: Re: hama streaming problem with hello world example From: "Edward J. Yoon" To: "user@hama.apache.org" Content-Type: text/plain; charset=UTF-8 Hi, We are investigating the problem. By the way, I really don't know where the string "Traceback (most recent call last):" comes from. When I tested it, I always seeing -1. java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) > > at java.lang.Integer.parseInt(Integer.java:580) > > at java.lang.Integer.parseInt(Integer.java:615) > > at On Wed, Oct 7, 2015 at 7:41 PM, Vincenzo De Leo wrote: > Thank You Edward, > can you explain to me how can I use your solution? I have tried with: > > - bin/hama pipes -streaming true -bspTasks 2 -interpreter python3.2 > -cachefiles /tmp/PyStreaming/*.py -output /tmp/pystream-out/ -inputformat > org.apache.hama.bsp.NullInputFormat -outputformat > org.apache.hama.bsp.SequenceFileOutputFormat -program > /tmp/PyStreaming/BSPRunner.py -programArgs HelloWorldBSP > > but I received the same old error messages... Do I have to install some > patch and how can I eventually do this? > > Vincenzo > > On Mon, Oct 5, 2015 at 2:41 AM, Edward J. Yoon > wrote: > >> Hi Vincenzo, >> >> I just found some code changes related to this problem at >> https://issues.apache.org/jira/browse/HAMA-839, and I was able to run >> Python program by adding the in/output formatters like: -inputformat >> org.apache.hama.bsp.NullInputFormat -outputformat >> org.apache.hama.bsp.SequenceFileOutputFormat. >> >> I'll check more and update the Wiki page. Sorry for inconvenient! >> >> >> On Fri, Oct 2, 2015 at 1:29 PM, Edward J. Yoon >> wrote: >> > I can look at it closely next week. BTW, should input be provided? >> > >> > On Thu, Oct 1, 2015 at 10:13 PM, Vincenzo De Leo >> > wrote: >> >> Thank you for your kind reply; this is the output of the K-Means >> example: >> >> >> >> $ clear;bin/hama pipes -streaming true -bspTasks 2 -interpreter >> python3.2 >> >> -cachefiles /tmp/PyStreaming/*.py -output /tmp/pystream-out/ -program >> >> /tmp/PyStreaming/BSPRunner.py -programArgs KMeansBSP >> >> 15/10/01 15:06:56 WARN util.NativeCodeLoader: Unable to load >> native-hadoop >> >> library for your platform... using builtin-java classes where applicable >> >> 15/10/01 15:06:56 INFO pipes.Submitter: Streaming enabled! >> >> 15/10/01 15:06:56 INFO Configuration.deprecation: fs.default.name is >> >> deprecated. Instead, use fs.defaultFS >> >> 15/10/01 15:06:56 INFO Configuration.deprecation: user.name is >> deprecated. >> >> Instead, use mapreduce.job.user.name >> >> 15/10/01 15:06:57 WARN conf.Configuration: >> >> >> org.apache.hadoop.fs.ChecksumFileSystem$FSDataBoundedInputStream@4ddced80 >> :an >> >> attempt to override final parameter: >> >> mapreduce.job.end-notification.max.retry.interval; Ignoring. >> >> 15/10/01 15:06:57 WARN conf.Configuration: >> >> >> org.apache.hadoop.fs.ChecksumFileSystem$FSDataBoundedInputStream@4ddced80 >> :an >> >> attempt to override final parameter: >> >> mapreduce.job.end-notification.max.attempts; Ignoring. >> >> 15/10/01 15:06:57 INFO Configuration.deprecation: user.name is >> deprecated. >> >> Instead, use mapreduce.job.user.name >> >> 15/10/01 15:06:57 INFO bsp.BSPJobClient: Running job: >> job_localrunner_0001 >> >> 15/10/01 15:06:57 INFO Configuration.deprecation: >> mapred.cache.localFiles >> >> is deprecated. Instead, use mapreduce.job.cache.local.files >> >> 15/10/01 15:06:57 INFO bsp.LocalBSPRunner: Setting up a new barrier for >> 2 >> >> tasks! >> >> java.lang.NumberFormatException: For input string: "Traceback (most >> recent >> >> call last):" >> >> at >> >> >> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) >> >> at java.lang.Integer.parseInt(Integer.java:580) >> >> at java.lang.Integer.parseInt(Integer.java:615) >> >> at >> >> >> org.apache.hama.pipes.protocol.StreamingProtocol$StreamingUplinkReaderThread.readCommand(StreamingProtocol.java:174) >> >> at >> >> org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:106) >> >> java.lang.NumberFormatException: For input string: "Traceback (most >> recent >> >> call last):" >> >> at >> >> >> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) >> >> at java.lang.Integer.parseInt(Integer.java:580) >> >> at java.lang.Integer.parseInt(Integer.java:615) >> >> at >> >> >> org.apache.hama.pipes.protocol.StreamingProtocol$StreamingUplinkReaderThread.readCommand(StreamingProtocol.java:174) >> >> at >> >> org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:106) >> >> 15/10/01 15:06:57 ERROR protocol.UplinkReader: java.lang.Exception: Bad >> >> command code: -2 >> >> at >> >> org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:172) >> >> >> >> Exception in thread "pipe-uplink-handler" java.lang.RuntimeException: >> >> java.lang.Exception: Bad command code: -2 >> >> at >> >> org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:180) >> >> Caused by: java.lang.Exception: Bad command code: -2 >> >> at >> >> org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:172) >> >> java.util.concurrent.BrokenBarrierException >> >> at >> java.util.concurrent.CyclicBarrier.dowait(CyclicBarrier.java:250) >> >> at >> java.util.concurrent.CyclicBarrier.await(CyclicBarrier.java:362) >> >> at >> >> >> org.apache.hama.pipes.protocol.StreamingProtocol.start(StreamingProtocol.java:223) >> >> at >> >> org.apache.hama.pipes.PipesApplication.start(PipesApplication.java:293) >> >> at org.apache.hama.pipes.PipesBSP.setup(PipesBSP.java:43) >> >> at >> >> >> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.run(LocalBSPRunner.java:255) >> >> at >> >> >> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:286) >> >> at >> >> >> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:211) >> >> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >> >> at >> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >> >> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >> >> at >> >> >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >> >> at >> >> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >> >> at java.lang.Thread.run(Thread.java:745) >> >> 15/10/01 15:06:57 ERROR protocol.UplinkReader: java.lang.Exception: Bad >> >> command code: -2 >> >> at >> >> org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:172) >> >> >> >> java.util.concurrent.BrokenBarrierException >> >> at >> java.util.concurrent.CyclicBarrier.dowait(CyclicBarrier.java:250) >> >> at >> java.util.concurrent.CyclicBarrier.await(CyclicBarrier.java:362) >> >> at >> >> >> org.apache.hama.pipes.protocol.StreamingProtocol.start(StreamingProtocol.java:223) >> >> at >> >> org.apache.hama.pipes.PipesApplication.start(PipesApplication.java:293) >> >> at org.apache.hama.pipes.PipesBSP.setup(PipesBSP.java:43) >> >> at >> >> >> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.run(LocalBSPRunner.java:255) >> >> at >> >> >> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:286) >> >> at >> >> >> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:211) >> >> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >> >> at >> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >> >> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >> >> at >> >> >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >> >> at >> >> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >> >> at java.lang.Thread.run(Thread.java:745) >> >> Exception in thread "pipe-uplink-handler" java.lang.RuntimeException: >> >> java.lang.Exception: Bad command code: -2 >> >> at >> >> org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:180) >> >> Caused by: java.lang.Exception: Bad command code: -2 >> >> at >> >> org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:172) >> >> 15/10/01 15:06:57 ERROR bsp.LocalBSPRunner: Exception during BSP >> execution! >> >> java.io.IOException: Stream closed >> >> at >> >> java.lang.ProcessBuilder$NullOutputStream.write(ProcessBuilder.java:433) >> >> at java.io.OutputStream.write(OutputStream.java:116) >> >> at >> >> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) >> >> at >> java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) >> >> at >> java.io.BufferedOutputStream.flush(BufferedOutputStream.java:141) >> >> at java.io.DataOutputStream.flush(DataOutputStream.java:123) >> >> at >> >> >> org.apache.hama.pipes.protocol.StreamingProtocol.writeLine(StreamingProtocol.java:287) >> >> at >> >> >> org.apache.hama.pipes.protocol.StreamingProtocol.runSetup(StreamingProtocol.java:248) >> >> at org.apache.hama.pipes.PipesBSP.setup(PipesBSP.java:45) >> >> at >> >> >> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.run(LocalBSPRunner.java:255) >> >> at >> >> >> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:286) >> >> at >> >> >> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:211) >> >> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >> >> at >> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >> >> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >> >> at >> >> >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >> >> at >> >> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >> >> at java.lang.Thread.run(Thread.java:745) >> >> 15/10/01 15:06:57 ERROR bsp.LocalBSPRunner: Exception during BSP >> execution! >> >> java.io.IOException: Stream closed >> >> at >> >> java.lang.ProcessBuilder$NullOutputStream.write(ProcessBuilder.java:433) >> >> at java.io.OutputStream.write(OutputStream.java:116) >> >> at >> >> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) >> >> at >> java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) >> >> at >> java.io.BufferedOutputStream.flush(BufferedOutputStream.java:141) >> >> at java.io.DataOutputStream.flush(DataOutputStream.java:123) >> >> at >> >> >> org.apache.hama.pipes.protocol.StreamingProtocol.writeLine(StreamingProtocol.java:287) >> >> at >> >> >> org.apache.hama.pipes.protocol.StreamingProtocol.runSetup(StreamingProtocol.java:248) >> >> at org.apache.hama.pipes.PipesBSP.setup(PipesBSP.java:45) >> >> at >> >> >> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.run(LocalBSPRunner.java:255) >> >> at >> >> >> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:286) >> >> at >> >> >> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:211) >> >> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >> >> at >> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >> >> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >> >> at >> >> >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >> >> at >> >> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >> >> at java.lang.Thread.run(Thread.java:745) >> >> 15/10/01 15:06:57 ERROR bsp.LocalBSPRunner: Error cleaning up after bsp >> >> execution. >> >> java.io.IOException: Stream closed >> >> at >> >> java.lang.ProcessBuilder$NullOutputStream.write(ProcessBuilder.java:433) >> >> at java.io.OutputStream.write(OutputStream.java:116) >> >> at >> >> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) >> >> at >> java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) >> >> at >> java.io.BufferedOutputStream.flush(BufferedOutputStream.java:141) >> >> at java.io.DataOutputStream.flush(DataOutputStream.java:123) >> >> at >> >> >> org.apache.hama.pipes.protocol.StreamingProtocol.writeLine(StreamingProtocol.java:287) >> >> at >> >> >> org.apache.hama.pipes.protocol.StreamingProtocol.runCleanup(StreamingProtocol.java:271) >> >> at org.apache.hama.pipes.PipesBSP.cleanup(PipesBSP.java:95) >> >> at >> >> >> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.run(LocalBSPRunner.java:262) >> >> at >> >> >> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:286) >> >> at >> >> >> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:211) >> >> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >> >> at >> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >> >> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >> >> at >> >> >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >> >> at >> >> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >> >> at java.lang.Thread.run(Thread.java:745) >> >> 15/10/01 15:06:57 ERROR bsp.LocalBSPRunner: Exception during BSP >> execution! >> >> java.util.concurrent.ExecutionException: java.io.IOException: Stream >> closed >> >> at java.util.concurrent.FutureTask.report(FutureTask.java:122) >> >> at java.util.concurrent.FutureTask.get(FutureTask.java:192) >> >> at >> >> >> org.apache.hama.bsp.LocalBSPRunner$ThreadObserver.run(LocalBSPRunner.java:313) >> >> at java.lang.Thread.run(Thread.java:745) >> >> Caused by: java.io.IOException: Stream closed >> >> at >> >> java.lang.ProcessBuilder$NullOutputStream.write(ProcessBuilder.java:433) >> >> at java.io.OutputStream.write(OutputStream.java:116) >> >> at >> >> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) >> >> at >> java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) >> >> at >> java.io.BufferedOutputStream.flush(BufferedOutputStream.java:141) >> >> at java.io.DataOutputStream.flush(DataOutputStream.java:123) >> >> at >> >> >> org.apache.hama.pipes.protocol.StreamingProtocol.writeLine(StreamingProtocol.java:287) >> >> at >> >> >> org.apache.hama.pipes.protocol.StreamingProtocol.runSetup(StreamingProtocol.java:248) >> >> at org.apache.hama.pipes.PipesBSP.setup(PipesBSP.java:45) >> >> at >> >> >> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.run(LocalBSPRunner.java:255) >> >> at >> >> >> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:286) >> >> at >> >> >> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:211) >> >> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >> >> at >> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >> >> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >> >> at >> >> >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >> >> at >> >> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >> >> ... 1 more >> >> 15/10/01 15:06:57 ERROR bsp.LocalBSPRunner: Error cleaning up after bsp >> >> execution. >> >> java.io.IOException: Stream closed >> >> at >> >> java.lang.ProcessBuilder$NullOutputStream.write(ProcessBuilder.java:433) >> >> at java.io.OutputStream.write(OutputStream.java:116) >> >> at >> >> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) >> >> at >> java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) >> >> at >> java.io.BufferedOutputStream.flush(BufferedOutputStream.java:141) >> >> at java.io.DataOutputStream.flush(DataOutputStream.java:123) >> >> at >> >> >> org.apache.hama.pipes.protocol.StreamingProtocol.writeLine(StreamingProtocol.java:287) >> >> at >> >> >> org.apache.hama.pipes.protocol.StreamingProtocol.runCleanup(StreamingProtocol.java:271) >> >> at org.apache.hama.pipes.PipesBSP.cleanup(PipesBSP.java:95) >> >> at >> >> >> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.run(LocalBSPRunner.java:262) >> >> at >> >> >> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:286) >> >> at >> >> >> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:211) >> >> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >> >> at >> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >> >> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >> >> at >> >> >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >> >> at >> >> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >> >> at java.lang.Thread.run(Thread.java:745) >> >> 15/10/01 15:06:58 INFO bsp.BSPJobClient: Job failed. >> >> >> >> Vincenzo >> >> >> >> On Thu, Oct 1, 2015 at 1:10 AM, Edward J. Yoon >> >> wrote: >> >> >> >>> It looks like a bug of input reader. Can you try to run K-Means >> example? >> >>> >> >>> On Wed, Sep 30, 2015 at 10:17 PM, Vincenzo De Leo >> >>> wrote: >> >>> > Hi to all, >> >>> > this is my first mail to this mailing list so please have patience >> if I >> >>> > make some bad choice in the format. >> >>> > >> >>> > I have a kubuntu-14.04 on an old Intel Core 2 Duo processor T7500 >> with 2 >> >>> GB >> >>> > of RAM. >> >>> > >> >>> > I have properly installed Hadoop-2.7.1, Sun Java JDK 1.8.0_60 e >> >>> Hama-0.7.0 >> >>> > as you can see from the following lines: >> >>> > >> >>> > >> >>> >> >> >>> >> > $ hadoop version >> >>> >> > Hadoop 2.7.1 >> >>> >> > Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r >> >>> > 15ecc87ccf4a0228f35af08fc56de536e6ce657a >> >>> >> > Compiled by jenkins on 2015-06-29T06:04Z >> >>> >> > Compiled with protoc 2.5.0 >> >>> >> > From source with checksum fc0a1a23fc1868e4d5ee7fa2b28a58a >> >>> >> > This command was run using >> >>> > >> >>> >> /home/tora/Downloads/hadoop-2.7.1/share/hadoop/common/hadoop-common-2.7.1.jar >> >>> > >> >>> > >> >>> >> > $ java -version >> >>> >> > java version "1.8.0_60" >> >>> >> > Java(TM) SE Runtime Environment (build 1.8.0_60-b27) >> >>> >> > Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode) >> >>> >> > >> >>> > >> >>> > I am able to properly run the basic Hadoop and Hama examples like the >> >>> > following: >> >>> > bin/hadoop jar >> share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar >> >>> > grep input output 'dfs[a-z.]+' >> >>> > bin/hama jar hama-examples-0.7.0.jar pi >> >>> > My problem is that I receive the following error message when I try >> to >> >>> run >> >>> > the HelloWorld Hama Streaming example with this instruction: >> >>> > bin/hama pipes -streaming true -bspTasks 2 -interpreter python3.2 >> >>> > -cachefiles /tmp/PyStreaming/*.py -output /tmp/pystream-out/ -program >> >>> > /tmp/PyStreaming/BSPRunner.py -programArgs HelloWorldBSP >> >>> > The default python interpreters of my OS were python-2.7 and >> python-3.4; >> >>> > since I had problems with this example I also tried to install >> python-3.2 >> >>> > with the following instructions but it didn't solved the problem: >> >>> > >> >>> > >> >>> >> >> >>> >> > sudo apt-get install software-properties-common >> >>> >> > sudo apt-add-repository ppa:fkrull/deadsnakes >> >>> >> > sudo apt-get update >> >>> >> > sudo apt-get install python3.2 >> >>> >> > >> >>> > >> >>> > >> >>> > The installed python version is 3.2.6 as you can see from the >> following >> >>> > lines: >> >>> > >> >>> > >> >>> >> >> >>> >> > $ python3.2 >> >>> >> > Python 3.2.6 (default, Oct 21 2014, 12:50:03) >> >>> >> > [GCC 4.8.2] on linux2 >> >>> >> > Type "help", "copyright", "credits" or "license" for more >> information. >> >>> >> > >>> >> >>> >> > >> >>> > >> >>> > >> >>> > The error message is the following (I am working in local mode so I >> >>> didn't >> >>> > run bin/start-bspd.sh): >> >>> > >> >>> > >> >>> >> >> >>> >> >> >>> >> > >> >>> >> > $ clear;bin/hama pipes -streaming true -bspTasks 2 -interpreter >> >>> > python3.2 -cachefiles /tmp/PyStreaming/*.py -output >> /tmp/pystream-out/ >> >>> > -program /tmp/PyStreaming/BSPRunner.py -programArgs HelloWorldBSP >> >>> >> > 15/09/30 12:39:11 WARN util.NativeCodeLoader: Unable to load >> >>> > native-hadoop library for your platform... using builtin-java classes >> >>> where >> >>> > applicable >> >>> >> > 15/09/30 12:39:11 INFO pipes.Submitter: Streaming enabled! >> >>> >> > 15/09/30 12:39:11 INFO Configuration.deprecation: fs.default.name >> is >> >>> > deprecated. Instead, use fs.defaultFS >> >>> >> > 15/09/30 12:39:11 INFO Configuration.deprecation: user.name is >> >>> > deprecated. Instead, use mapreduce.job.user.name >> >>> >> > 15/09/30 12:39:11 WARN conf.Configuration: >> >>> > >> org.apache.hadoop.fs.ChecksumFileSystem$FSDataBoundedInputStream@4ddced80 >> >>> :an >> >>> > attempt to override final parameter: >> >>> > mapreduce.job.end-notification.max.retry.interval; Ignoring. >> >>> >> > 15/09/30 12:39:11 WARN conf.Configuration: >> >>> > >> org.apache.hadoop.fs.ChecksumFileSystem$FSDataBoundedInputStream@4ddced80 >> >>> :an >> >>> > attempt to override final parameter: >> >>> > mapreduce.job.end-notification.max.attempts; Ignoring. >> >>> >> > 15/09/30 12:39:12 INFO Configuration.deprecation: user.name is >> >>> > deprecated. Instead, use mapreduce.job.user.name >> >>> >> > 15/09/30 12:39:12 INFO bsp.BSPJobClient: Running job: >> >>> > job_localrunner_0001 >> >>> >> > 15/09/30 12:39:12 INFO Configuration.deprecation: >> >>> > mapred.cache.localFiles is deprecated. Instead, use >> >>> > mapreduce.job.cache.local.files >> >>> >> > 15/09/30 12:39:12 INFO bsp.LocalBSPRunner: Setting up a new >> barrier >> >>> for >> >>> > 2 tasks! >> >>> >> > java.lang.NumberFormatException: For input string: "Traceback >> (most >> >>> > recent call last):" >> >>> >> > at >> >>> > >> >>> >> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) >> >>> >> > at java.lang.Integer.parseInt(Integer.java:580) >> >>> >> > at java.lang.Integer.parseInt(Integer.java:615) >> >>> >> > at >> >>> > >> >>> >> org.apache.hama.pipes.protocol.StreamingProtocol$StreamingUplinkReaderThread.readCommand(StreamingProtocol.java:174) >> >>> >> > at >> >>> > >> org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:106) >> >>> >> > java.lang.NumberFormatException: For input string: "Traceback >> (most >> >>> > recent call last):" >> >>> >> > at >> >>> > >> >>> >> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) >> >>> >> > at java.lang.Integer.parseInt(Integer.java:580) >> >>> >> > at java.lang.Integer.parseInt(Integer.java:615) >> >>> >> > at >> >>> > >> >>> >> org.apache.hama.pipes.protocol.StreamingProtocol$StreamingUplinkReaderThread.readCommand(StreamingProtocol.java:174) >> >>> >> > at >> >>> > >> org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:106) >> >>> >> > 15/09/30 12:39:12 ERROR protocol.UplinkReader: >> java.lang.Exception: >> >>> Bad >> >>> > command code: -2 >> >>> >> > at >> >>> > >> org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:172) >> >>> >> > >> >>> >> > 15/09/30 12:39:12 ERROR protocol.UplinkReader: >> java.lang.Exception: >> >>> Bad >> >>> > command code: -2 >> >>> >> > at >> >>> > >> org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:172) >> >>> >> > >> >>> >> > java.util.concurrent.BrokenBarrierException >> >>> >> > at >> >>> > java.util.concurrent.CyclicBarrier.dowait(CyclicBarrier.java:250) >> >>> >> > at >> >>> > java.util.concurrent.CyclicBarrier.await(CyclicBarrier.java:362) >> >>> >> > at >> >>> > >> >>> >> org.apache.hama.pipes.protocol.StreamingProtocol.start(StreamingProtocol.java:223) >> >>> >> > at >> >>> > >> org.apache.hama.pipes.PipesApplication.start(PipesApplication.java:293) >> >>> >> > at org.apache.hama.pipes.PipesBSP.setup(PipesBSP.java:43) >> >>> >> > at >> >>> > >> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.run(LocalBSPRunner.java:255) >> >>> >> > at >> >>> > >> >>> >> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:286) >> >>> >> > at >> >>> > >> >>> >> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:211) >> >>> >> > at >> java.util.concurrent.FutureTask.run(FutureTask.java:266) >> >>> >> > at >> >>> > >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >> >>> >> > at >> java.util.concurrent.FutureTask.run(FutureTask.java:266) >> >>> >> > at >> >>> > >> >>> >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >> >>> >> > at >> >>> > >> >>> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >> >>> >> > at java.lang.Thread.run(Thread.java:745) >> >>> >> > Exception in thread "pipe-uplink-handler" >> java.lang.RuntimeException: >> >>> > java.lang.Exception: Bad command code: -2 >> >>> >> > at >> >>> > >> org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:180) >> >>> >> > Caused by: java.lang.Exception: Bad command code: -2 >> >>> >> > at >> >>> > >> org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:172) >> >>> >> > java.util.concurrent.BrokenBarrierException >> >>> >> > at >> >>> > java.util.concurrent.CyclicBarrier.dowait(CyclicBarrier.java:250) >> >>> >> > at >> >>> > java.util.concurrent.CyclicBarrier.await(CyclicBarrier.java:362) >> >>> >> > at >> >>> > >> >>> >> org.apache.hama.pipes.protocol.StreamingProtocol.start(StreamingProtocol.java:223) >> >>> >> > at >> >>> > >> org.apache.hama.pipes.PipesApplication.start(PipesApplication.java:293) >> >>> >> > at org.apache.hama.pipes.PipesBSP.setup(PipesBSP.java:43) >> >>> >> > at >> >>> > >> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.run(LocalBSPRunner.java:255) >> >>> >> > at >> >>> > >> >>> >> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:286) >> >>> >> > at >> >>> > >> >>> >> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:211) >> >>> >> > at >> java.util.concurrent.FutureTask.run(FutureTask.java:266) >> >>> >> > at >> >>> > >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >> >>> >> > at >> java.util.concurrent.FutureTask.run(FutureTask.java:266) >> >>> >> > at >> >>> > >> >>> >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >> >>> >> > at >> >>> > >> >>> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >> >>> >> > at java.lang.Thread.run(Thread.java:745) >> >>> >> > Exception in thread "pipe-uplink-handler" >> java.lang.RuntimeException: >> >>> > java.lang.Exception: Bad command code: -2 >> >>> >> > at >> >>> > >> org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:180) >> >>> >> > Caused by: java.lang.Exception: Bad command code: -2 >> >>> >> > at >> >>> > >> org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:172) >> >>> >> > 15/09/30 12:39:12 ERROR bsp.LocalBSPRunner: Exception during BSP >> >>> > execution! >> >>> >> > java.io.IOException: Stream closed >> >>> >> > at >> >>> > >> java.lang.ProcessBuilder$NullOutputStream.write(ProcessBuilder.java:433) >> >>> >> > at java.io.OutputStream.write(OutputStream.java:116) >> >>> >> > at >> >>> > >> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) >> >>> >> > at >> >>> > java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) >> >>> >> > at >> >>> > java.io.BufferedOutputStream.flush(BufferedOutputStream.java:141) >> >>> >> > at >> java.io.DataOutputStream.flush(DataOutputStream.java:123) >> >>> >> > at >> >>> > >> >>> >> org.apache.hama.pipes.protocol.StreamingProtocol.writeLine(StreamingProtocol.java:287) >> >>> >> > at >> >>> > >> >>> >> org.apache.hama.pipes.protocol.StreamingProtocol.runSetup(StreamingProtocol.java:248) >> >>> >> > at org.apache.hama.pipes.PipesBSP.setup(PipesBSP.java:45) >> >>> >> > at >> >>> > >> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.run(LocalBSPRunner.java:255) >> >>> >> > at >> >>> > >> >>> >> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:286) >> >>> >> > at >> >>> > >> >>> >> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:211) >> >>> >> > at >> java.util.concurrent.FutureTask.run(FutureTask.java:266) >> >>> >> > at >> >>> > >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >> >>> >> > at >> java.util.concurrent.FutureTask.run(FutureTask.java:266) >> >>> >> > at >> >>> > >> >>> >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >> >>> >> > at >> >>> > >> >>> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >> >>> >> > at java.lang.Thread.run(Thread.java:745) >> >>> >> > 15/09/30 12:39:12 ERROR bsp.LocalBSPRunner: Exception during BSP >> >>> > execution! >> >>> >> > java.lang.InterruptedException: java.lang.Exception: Bad command >> code: >> >>> > -2 >> >>> >> > at >> >>> > >> org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:172) >> >>> >> > >> >>> >> > at >> >>> > >> >>> >> org.apache.hama.pipes.protocol.BinaryProtocol.waitForFinish(BinaryProtocol.java:335) >> >>> >> > at >> >>> > >> >>> >> org.apache.hama.pipes.PipesApplication.waitForFinish(PipesApplication.java:329) >> >>> >> > at org.apache.hama.pipes.PipesBSP.setup(PipesBSP.java:47) >> >>> >> > at >> >>> > >> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.run(LocalBSPRunner.java:255) >> >>> >> > at >> >>> > >> >>> >> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:286) >> >>> >> > at >> >>> > >> >>> >> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:211) >> >>> >> > at >> java.util.concurrent.FutureTask.run(FutureTask.java:266) >> >>> >> > at >> >>> > >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >> >>> >> > at >> java.util.concurrent.FutureTask.run(FutureTask.java:266) >> >>> >> > at >> >>> > >> >>> >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >> >>> >> > at >> >>> > >> >>> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >> >>> >> > at java.lang.Thread.run(Thread.java:745) >> >>> >> > 15/09/30 12:39:12 ERROR bsp.LocalBSPRunner: Error cleaning up >> after >> >>> bsp >> >>> > execution. >> >>> >> > java.io.IOException: Stream closed >> >>> >> > at >> >>> > >> java.lang.ProcessBuilder$NullOutputStream.write(ProcessBuilder.java:433) >> >>> >> > at java.io.OutputStream.write(OutputStream.java:116) >> >>> >> > at >> >>> > >> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) >> >>> >> > at >> >>> > java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) >> >>> >> > at >> >>> > java.io.BufferedOutputStream.flush(BufferedOutputStream.java:141) >> >>> >> > at >> java.io.DataOutputStream.flush(DataOutputStream.java:123) >> >>> >> > at >> >>> > >> >>> >> org.apache.hama.pipes.protocol.StreamingProtocol.writeLine(StreamingProtocol.java:287) >> >>> >> > at >> >>> > >> >>> >> org.apache.hama.pipes.protocol.StreamingProtocol.runCleanup(StreamingProtocol.java:271) >> >>> >> > at >> org.apache.hama.pipes.PipesBSP.cleanup(PipesBSP.java:95) >> >>> >> > at >> >>> > >> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.run(LocalBSPRunner.java:262) >> >>> >> > at >> >>> > >> >>> >> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:286) >> >>> >> > at >> >>> > >> >>> >> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:211) >> >>> >> > at >> java.util.concurrent.FutureTask.run(FutureTask.java:266) >> >>> >> > at >> >>> > >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >> >>> >> > at >> java.util.concurrent.FutureTask.run(FutureTask.java:266) >> >>> >> > at >> >>> > >> >>> >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >> >>> >> > at >> >>> > >> >>> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >> >>> >> > at java.lang.Thread.run(Thread.java:745) >> >>> >> > 15/09/30 12:39:12 ERROR bsp.LocalBSPRunner: Error cleaning up >> after >> >>> bsp >> >>> > execution. >> >>> >> > java.io.IOException: Stream closed >> >>> >> > at >> >>> > >> java.lang.ProcessBuilder$NullOutputStream.write(ProcessBuilder.java:433) >> >>> >> > at java.io.OutputStream.write(OutputStream.java:116) >> >>> >> > at >> >>> > >> java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) >> >>> >> > at >> >>> > java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) >> >>> >> > at >> >>> > java.io.BufferedOutputStream.flush(BufferedOutputStream.java:141) >> >>> >> > at >> java.io.DataOutputStream.flush(DataOutputStream.java:123) >> >>> >> > at >> >>> > >> >>> >> org.apache.hama.pipes.protocol.StreamingProtocol.writeLine(StreamingProtocol.java:287) >> >>> >> > at >> >>> > >> >>> >> org.apache.hama.pipes.protocol.StreamingProtocol.runCleanup(StreamingProtocol.java:271) >> >>> >> > at >> org.apache.hama.pipes.PipesBSP.cleanup(PipesBSP.java:95) >> >>> >> > at >> >>> > >> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.run(LocalBSPRunner.java:262) >> >>> >> > at >> >>> > >> >>> >> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:286) >> >>> >> > at >> >>> > >> >>> >> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:211) >> >>> >> > at >> java.util.concurrent.FutureTask.run(FutureTask.java:266) >> >>> >> > at >> >>> > >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >> >>> >> > at >> java.util.concurrent.FutureTask.run(FutureTask.java:266) >> >>> >> > at >> >>> > >> >>> >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >> >>> >> > at >> >>> > >> >>> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >> >>> >> > at java.lang.Thread.run(Thread.java:745) >> >>> >> > 15/09/30 12:39:12 ERROR bsp.LocalBSPRunner: Exception during BSP >> >>> > execution! >> >>> >> > java.util.concurrent.ExecutionException: >> >>> > java.lang.InterruptedException: java.lang.Exception: Bad command >> code: -2 >> >>> >> > at >> >>> > >> org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:172) >> >>> >> > >> >>> >> > at >> java.util.concurrent.FutureTask.report(FutureTask.java:122) >> >>> >> > at >> java.util.concurrent.FutureTask.get(FutureTask.java:192) >> >>> >> > at >> >>> > >> >>> >> org.apache.hama.bsp.LocalBSPRunner$ThreadObserver.run(LocalBSPRunner.java:313) >> >>> >> > at java.lang.Thread.run(Thread.java:745) >> >>> >> > Caused by: java.lang.InterruptedException: java.lang.Exception: >> Bad >> >>> > command code: -2 >> >>> >> > at >> >>> > >> org.apache.hama.pipes.protocol.UplinkReader.run(UplinkReader.java:172) >> >>> >> > >> >>> >> > at >> >>> > >> >>> >> org.apache.hama.pipes.protocol.BinaryProtocol.waitForFinish(BinaryProtocol.java:335) >> >>> >> > at >> >>> > >> >>> >> org.apache.hama.pipes.PipesApplication.waitForFinish(PipesApplication.java:329) >> >>> >> > at org.apache.hama.pipes.PipesBSP.setup(PipesBSP.java:47) >> >>> >> > at >> >>> > >> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.run(LocalBSPRunner.java:255) >> >>> >> > at >> >>> > >> >>> >> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:286) >> >>> >> > at >> >>> > >> >>> >> org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:211) >> >>> >> > at >> java.util.concurrent.FutureTask.run(FutureTask.java:266) >> >>> >> > at >> >>> > >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >> >>> >> > at >> java.util.concurrent.FutureTask.run(FutureTask.java:266) >> >>> >> > at >> >>> > >> >>> >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >> >>> >> > at >> >>> > >> >>> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >> >>> >> > ... 1 more >> >>> >> > 15/09/30 12:39:13 INFO bsp.BSPJobClient: Job failed. >> >>> >> > >> >>> > >> >>> > >> >>> > I tried to searching for these error messages with google but I >> didn't >> >>> find >> >>> > any solution. >> >>> > >> >>> > Does anyone of you have any idea of how can I solve this problem? >> >>> > >> >>> > Any help is appreciated :) >> >>> > >> >>> > Thank you >> >>> > Best >> >>> > >> >>> > Vincenzo >> >>> >> >>> >> >>> >> >>> -- >> >>> Best Regards, Edward J. Yoon >> >>> >> >> >> >> >> >> >> >> -- >> >> >> >> >> ----------------------------------------------------------------------------- >> >> Dr. Vincenzo De Leo, PhD >> >> Research and Development @ Linkalab >> >> >> ----------------------------------------------------------------------------- >> >> Home Page: http://www.linkalab.it/people/vincenzo.deleo >> >> e-mail: vincenzo.deleo@linkalab.it >> >> Phone: (+39)0706753140 >> >> Mobile: (+39)3406673544 >> >> Skype contact: vincenzo.deleo77 >> >> GTalk contact: nagatobimaru >> >> >> ----------------------------------------------------------------------------- >> >> Linkalab (http://www.linkalab.it) >> >> CoMPLeX SySTeMS CoMPuTaTioNaL LaBoRaToRy >> >> Via San Benedetto, 88 >> >> 09129 Cagliari (CA), ITALY >> >> P. IVA: 03148020922 >> >> Phone: (+39) 0707966021 >> >> FAX: (+39) 0707962000 >> >> >> ----------------------------------------------------------------------------- >> > >> > >> > >> > -- >> > Best Regards, Edward J. Yoon >> >> >> >> -- >> Best Regards, Edward J. Yoon >> -- Best Regards, Edward J. Yoon