Return-Path: Delivered-To: apmail-incubator-pig-dev-archive@locus.apache.org Received: (qmail 43721 invoked from network); 7 Apr 2008 23:00:11 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 7 Apr 2008 23:00:11 -0000 Received: (qmail 29626 invoked by uid 500); 7 Apr 2008 23:00:11 -0000 Delivered-To: apmail-incubator-pig-dev-archive@incubator.apache.org Received: (qmail 29589 invoked by uid 500); 7 Apr 2008 23:00:11 -0000 Mailing-List: contact pig-dev-help@incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: pig-dev@incubator.apache.org Delivered-To: mailing list pig-dev@incubator.apache.org Received: (qmail 29561 invoked by uid 99); 7 Apr 2008 23:00:11 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Apr 2008 16:00:11 -0700 X-ASF-Spam-Status: No, hits=-1998.5 required=10.0 tests=ALL_TRUSTED,WEIRD_PORT X-Spam-Check-By: apache.org Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 07 Apr 2008 22:59:26 +0000 Received: from brutus (localhost [127.0.0.1]) by brutus.apache.org (Postfix) with ESMTP id E4A64234C0BF for ; Mon, 7 Apr 2008 15:57:24 -0700 (PDT) Message-ID: <1252342836.1207609044922.JavaMail.jira@brutus> Date: Mon, 7 Apr 2008 15:57:24 -0700 (PDT) From: "Olga Natkovich (JIRA)" To: pig-dev@incubator.apache.org Subject: [jira] Updated: (PIG-182) Broken pipe if excuting the streaming script via the stream command directory In-Reply-To: <1140154720.1207187064218.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/PIG-182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Olga Natkovich updated PIG-182: ------------------------------- Resolution: Fixed Status: Resolved (was: Patch Available) patch committed. thanks arun for fixing and xu for testing > Broken pipe if excuting the streaming script via the stream command directory > ----------------------------------------------------------------------------- > > Key: PIG-182 > URL: https://issues.apache.org/jira/browse/PIG-182 > Project: Pig > Issue Type: Bug > Reporter: Xu Zhang > Assignee: Arun C Murthy > Priority: Blocker > Attachments: MySimpleStreamApp.pl, PIG-182_0_20080404.patch, PIG-182_1_20080404.patch, PIG-182_2_20080407.patch, script.pig > > > I got "broken pipe" exception with the following Pig script. I also attached the Pig script and the perl script to this bug report. > {code} > A = load '/user/pig/tests/data/singlefile/studenttab10k'; > B = stream A through `perl /home/xu/streamingscript/MySimpleStreamApp.pl` as (name, age, gpa); > store B into 'results_9'; > {code} > Here is Pig's console output > {noformat} > I can't find HOD configuration for piglet, hopefully you weren't planning on using HOD. > 2008-04-02 18:37:29,214 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: wilbur11.labs.corp.sp1.yahoo.com:8020 > 2008-04-02 18:37:30,030 [main] INFO org.apache.pig.backend.hadoop.executionengine.POMapreduce - ----- MapReduce Job ----- > 2008-04-02 18:37:30,030 [main] INFO org.apache.pig.backend.hadoop.executionengine.POMapreduce - Input: [/user/pig/tests/data/singlefile/studenttab10k:org.apache.pig.builtin.PigStorage()] > 2008-04-02 18:37:30,031 [main] INFO org.apache.pig.backend.hadoop.executionengine.POMapreduce - Map: [[*]->org.apache.pig.impl.eval.StreamSpec@121f1d] > 2008-04-02 18:37:30,031 [main] INFO org.apache.pig.backend.hadoop.executionengine.POMapreduce - Group: null > 2008-04-02 18:37:30,032 [main] INFO org.apache.pig.backend.hadoop.executionengine.POMapreduce - Combine: null > 2008-04-02 18:37:30,032 [main] INFO org.apache.pig.backend.hadoop.executionengine.POMapreduce - Reduce: null > 2008-04-02 18:37:30,032 [main] INFO org.apache.pig.backend.hadoop.executionengine.POMapreduce - Output: results_9:org.apache.pig.builtin.BinaryStorage > 2008-04-02 18:37:30,032 [main] INFO org.apache.pig.backend.hadoop.executionengine.POMapreduce - Split: null > 2008-04-02 18:37:30,032 [main] INFO org.apache.pig.backend.hadoop.executionengine.POMapreduce - Map parallelism: -1 > 2008-04-02 18:37:30,033 [main] INFO org.apache.pig.backend.hadoop.executionengine.POMapreduce - Reduce parallelism: -1 > 219190 hdfs://wilbur11.labs.corp.sp1.yahoo.com:8020/user/pig/tests/data/singlefile/studenttab10k > 2008-04-02 18:37:32,889 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Pig progress = 0% > 2008-04-02 18:37:53,985 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (map) tip_200803281454_0803_m_000000 java.lang.RuntimeException: java.io.IOException: Broken pipe > at org.apache.pig.impl.eval.StreamSpec$StreamDataCollector.finish(StreamSpec.java:152) > at org.apache.pig.impl.eval.collector.DataCollector.finishPipe(DataCollector.java:131) > at org.apache.pig.backend.hadoop.executionengine.mapreduceExec.PigMapReduce.run(PigMapReduce.java:119) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:208) > at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2071) > Caused by: java.io.IOException: Broken pipe > at java.io.FileOutputStream.writeBytes(Native Method) > at java.io.FileOutputStream.write(FileOutputStream.java:260) > at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65) > at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109) > at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65) > at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123) > at java.io.DataOutputStream.flush(DataOutputStream.java:106) > at org.apache.pig.impl.streaming.DefaultInputHandler.close(DefaultInputHandler.java:56) > at org.apache.pig.impl.streaming.ExecutableManager.close(ExecutableManager.java:128) > at org.apache.pig.backend.hadoop.streaming.HadoopExecutableManager.close(HadoopExecutableManager.java:115) > at org.apache.pig.impl.eval.StreamSpec$StreamDataCollector.finish(StreamSpec.java:148) > ... 4 more > java.lang.RuntimeException: java.io.IOException: Broken pipe > at org.apache.pig.impl.eval.StreamSpec$StreamDataCollector.finish(StreamSpec.java:152) > at org.apache.pig.impl.eval.collector.DataCollector.finishPipe(DataCollector.java:131) > at org.apache.pig.backend.hadoop.executionengine.mapreduceExec.PigMapReduce.run(PigMapReduce.java:119) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:208) > at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2071) > Caused by: java.io.IOException: Broken pipe > at java.io.FileOutputStream.writeBytes(Native Method) > at java.io.FileOutputStream.write(FileOutputStream.java:260) > at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65) > at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109) > at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65) > at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123) > at java.io.DataOutputStream.flush(DataOutputStream.java:106) > at org.apache.pig.impl.streaming.DefaultInputHandler.close(DefaultInputHandler.java:56) > at org.apache.pig.impl.streaming.ExecutableManager.close(ExecutableManager.java:128) > at org.apache.pig.backend.hadoop.streaming.HadoopExecutableManager.close(HadoopExecutableManager.java:115) > at org.apache.pig.impl.eval.StreamSpec$StreamDataCollector.finish(StreamSpec.java:148) > ... 4 more > java.lang.RuntimeException: java.io.IOException: Broken pipe > at org.apache.pig.impl.eval.StreamSpec$StreamDataCollector.finish(StreamSpec.java:152) > at org.apache.pig.impl.eval.collector.DataCollector.finishPipe(DataCollector.java:131) > at org.apache.pig.backend.hadoop.executionengine.mapreduceExec.PigMapReduce.run(PigMapReduce.java:119) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:208) > at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2071) > Caused by: java.io.IOException: Broken pipe > at java.io.FileOutputStream.writeBytes(Native Method) > at java.io.FileOutputStream.write(FileOutputStream.java:260) > at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65) > at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109) > at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65) > at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123) > at java.io.DataOutputStream.flush(DataOutputStream.java:106) > at org.apache.pig.impl.streaming.DefaultInputHandler.close(DefaultInputHandler.java:56) > at org.apache.pig.impl.streaming.ExecutableManager.close(ExecutableManager.java:128) > at org.apache.pig.backend.hadoop.streaming.HadoopExecutableManager.close(HadoopExecutableManager.java:115) > at org.apache.pig.impl.eval.StreamSpec$StreamDataCollector.finish(StreamSpec.java:148) > ... 4 more > java.lang.RuntimeException: java.io.IOException: Broken pipe > at org.apache.pig.impl.eval.StreamSpec$StreamDataCollector.finish(StreamSpec.java:152) > at org.apache.pig.impl.eval.collector.DataCollector.finishPipe(DataCollector.java:131) > at org.apache.pig.backend.hadoop.executionengine.mapreduceExec.PigMapReduce.run(PigMapReduce.java:119) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:208) > at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2071) > Caused by: java.io.IOException: Broken pipe > at java.io.FileOutputStream.writeBytes(Native Method) > at java.io.FileOutputStream.write(FileOutputStream.java:260) > at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65) > at java.io.BufferedOutputStream.write(BufferedOutputStream.java:109) > at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65) > at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123) > at java.io.DataOutputStream.flush(DataOutputStream.java:106) > at org.apache.pig.impl.streaming.DefaultInputHandler.close(DefaultInputHandler.java:56) > at org.apache.pig.impl.streaming.ExecutableManager.close(ExecutableManager.java:128) > at org.apache.pig.backend.hadoop.streaming.HadoopExecutableManager.close(HadoopExecutableManager.java:115) > at org.apache.pig.impl.eval.StreamSpec$StreamDataCollector.finish(StreamSpec.java:148) > ... 4 more > 2008-04-02 18:37:53,998 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_0803_r_000000 > 2008-04-02 18:37:53,998 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_0803_r_000001 > 2008-04-02 18:37:53,998 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_0803_r_000002 > 2008-04-02 18:37:53,998 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_0803_r_000003 > 2008-04-02 18:37:53,998 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_0803_r_000004 > 2008-04-02 18:37:53,999 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_0803_r_000005 > 2008-04-02 18:37:53,999 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_0803_r_000006 > 2008-04-02 18:37:53,999 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_0803_r_000007 > 2008-04-02 18:37:53,999 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_0803_r_000008 > 2008-04-02 18:37:53,999 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_0803_r_000009 > 2008-04-02 18:37:53,999 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_0803_r_000010 > 2008-04-02 18:37:53,999 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_0803_r_000011 > 2008-04-02 18:37:53,999 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_0803_r_000012 > 2008-04-02 18:37:53,999 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_0803_r_000013 > 2008-04-02 18:37:53,999 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_0803_r_000014 > 2008-04-02 18:37:54,000 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_0803_r_000015 > 2008-04-02 18:37:54,001 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_0803_r_000016 > 2008-04-02 18:37:54,001 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_0803_r_000017 > 2008-04-02 18:37:54,001 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_0803_r_000018 > 2008-04-02 18:37:54,001 [main] ERROR org.apache.pig.backend.hadoop.executionengine.mapreduceExec.MapReduceLauncher - Error message from task (reduce) tip_200803281454_0803_r_000019 > 2008-04-02 18:37:54,005 [main] ERROR org.apache.pig.tools.grunt.Grunt - java.io.IOException: Unable to store alias null > at org.apache.pig.impl.util.WrappedIOException.wrap(WrappedIOException.java:16) > at org.apache.pig.PigServer.registerQuery(PigServer.java:283) > at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:446) > at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:226) > at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:62) > at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:60) > at org.apache.pig.Main.main(Main.java:265) > Caused by: org.apache.pig.backend.executionengine.ExecException: java.io.IOException: Job failed > at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:288) > at org.apache.pig.PigServer.optimizeAndRunQuery(PigServer.java:400) > at org.apache.pig.PigServer.registerQuery(PigServer.java:280) > ... 5 more > Caused by: java.io.IOException: Job failed > at org.apache.pig.backend.hadoop.executionengine.POMapreduce.open(POMapreduce.java:179) > at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.execute(HExecutionEngine.java:274) > ... 7 more > 2008-04-02 18:37:54,005 [main] ERROR org.apache.pig.tools.grunt.Grunt - Unable to store alias null > {noformat} -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.