pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "pi song" <pi.so...@gmail.com>
Subject Re: Use of MiniCluster
Date Thu, 01 May 2008 02:57:03 GMT
Shravan,

Have you got the solution on this yet? I'm also expecting the answer from
someone else but seems like nobody knows. I guess what you have to do is to
access MiniCluster via PigServer API instead (=just a tiny : )pointer).

Please tell me how if you've already got this working.

Pi



On 4/30/08, Shravan M Narayanamurthy <shravanm@yahoo-inc.com> wrote:
>
> Hi,
>   I am working on the types branch and am trying to use the MiniCluster
> class for testing Job Submission and Monitoring in the redesigned pipeline.
> I have hit into an issue.
>
> The MiniCluster's Map Reduce part does not seem to work. I tried to submit
> a wordCount MR job that exists in the hadoop examples jar. It does not work.
> Here are the steps I followed:
>
>   (i) Started the MiniCluster using this:
>
>       package org.apache.pig.test;
>
>       public class TestMiniCluster {
>           public static void main(String[] args) throws
>       InterruptedException {
>               MiniCluster cluster = MiniCluster.buildCluster();
>               while(true){
>                   Thread.sleep(10000);
>               }
>           }
>       }
>
>   (ii) hadoop --config ~/pigtest/conf dfs -put /etc/passwd passwd
>
>   (iii) hadoop --config ~/pigtest/conf dfs -ls
>   Found 1 items
>   /user/shravanm/passwd   <r 3>   2768    2008-04-30 04:37
>
>   Verified that the file had been created
>
>   (iv) hadoop --config ~/pigtest/conf jar hadoop-0.15.3-examples.jar
>   wordcount passwd passwdCnt
>   Got the following exceptions:
>   08/04/30 04:37:41 INFO mapred.FileInputFormat: Total input paths to
>   process : 1
>   08/04/30 04:37:41 INFO mapred.JobClient: Running job:
>   job_200804300435_0001
>   08/04/30 04:37:42 INFO mapred.JobClient:  map 0% reduce 0%
>   08/04/30 04:37:54 INFO mapred.JobClient: Task Id :
>   task_200804300435_0001_m_000000_0, Status : FAILED
>   08/04/30 04:37:55 WARN mapred.JobClient: Error reading task
>   outputhttp://
> richbreak-lr.eglbp.corp.yahoo.com:50061/tasklog?plaintext=true&taskid=task_200804300435_0001_m_000000_0&filter=stdout
>   08/04/30 04:37:55 WARN mapred.JobClient: Error reading task
>   outputhttp://
> richbreak-lr.eglbp.corp.yahoo.com:50061/tasklog?plaintext=true&taskid=task_200804300435_0001_m_000000_0&filter=stderr
>   08/04/30 04:37:55 INFO mapred.JobClient: Task Id :
>   task_200804300435_0001_m_000001_0, Status : FAILED
>   08/04/30 04:37:55 WARN mapred.JobClient: Error reading task
>   outputhttp://
> richbreak-lr.eglbp.corp.yahoo.com:50063/tasklog?plaintext=true&taskid=task_200804300435_0001_m_000001_0&filter=stdout
>   08/04/30 04:37:55 WARN mapred.JobClient: Error reading task
>   outputhttp://
> richbreak-lr.eglbp.corp.yahoo.com:50063/tasklog?plaintext=true&taskid=task_200804300435_0001_m_000001_0&filter=stderr
>   08/04/30 04:37:55 INFO mapred.JobClient: Task Id :
>   task_200804300435_0001_r_000000_0, Status : FAILED
>   08/04/30 04:37:55 WARN mapred.JobClient: Error reading task
>   outputhttp://
> richbreak-lr.eglbp.corp.yahoo.com:50062/tasklog?plaintext=true&taskid=task_200804300435_0001_r_000000_0&filter=stdout
>   08/04/30 04:37:55 WARN mapred.JobClient: Error reading task
>   outputhttp://
> richbreak-lr.eglbp.corp.yahoo.com:50062/tasklog?plaintext=true&taskid=task_200804300435_0001_r_000000_0&filter=stderr
>   08/04/30 04:37:58 INFO mapred.JobClient: Task Id :
>   task_200804300435_0001_m_000001_1, Status : FAILED
>   08/04/30 04:37:58 WARN mapred.JobClient: Error reading task
>   outputhttp://
> richbreak-lr.eglbp.corp.yahoo.com:50062/tasklog?plaintext=true&taskid=task_200804300435_0001_m_000001_1&filter=stdout
>   08/04/30 04:37:58 WARN mapred.JobClient: Error reading task
>   outputhttp://
> richbreak-lr.eglbp.corp.yahoo.com:50062/tasklog?plaintext=true&taskid=task_200804300435_0001_m_000001_1&filter=stderr
>   08/04/30 04:37:58 INFO mapred.JobClient: Task Id :
>   task_200804300435_0001_m_000000_1, Status : FAILED
>   08/04/30 04:37:58 WARN mapred.JobClient: Error reading task
>   outputhttp://
> richbreak-lr.eglbp.corp.yahoo.com:50063/tasklog?plaintext=true&taskid=task_200804300435_0001_m_000000_1&filter=stdout
>   08/04/30 04:37:58 WARN mapred.JobClient: Error reading task
>   outputhttp://
> richbreak-lr.eglbp.corp.yahoo.com:50063/tasklog?plaintext=true&taskid=task_200804300435_0001_m_000000_1&filter=stderr
>   08/04/30 04:38:01 INFO mapred.JobClient: Task Id :
>   task_200804300435_0001_r_000000_1, Status : FAILED
>   08/04/30 04:38:01 WARN mapred.JobClient: Error reading task
>   outputhttp://
> richbreak-lr.eglbp.corp.yahoo.com:50063/tasklog?plaintext=true&taskid=task_200804300435_0001_r_000000_1&filter=stdout
>   08/04/30 04:38:01 WARN mapred.JobClient: Error reading task
>   outputhttp://
> richbreak-lr.eglbp.corp.yahoo.com:50063/tasklog?plaintext=true&taskid=task_200804300435_0001_r_000000_1&filter=stderr
>   08/04/30 04:38:05 INFO mapred.JobClient: Task Id :
>   task_200804300435_0001_m_000000_2, Status : FAILED
>   08/04/30 04:38:05 WARN mapred.JobClient: Error reading task
>   outputhttp://
> richbreak-lr.eglbp.corp.yahoo.com:50060/tasklog?plaintext=true&taskid=task_200804300435_0001_m_000000_2&filter=stdout
>   08/04/30 04:38:05 WARN mapred.JobClient: Error reading task
>   outputhttp://
> richbreak-lr.eglbp.corp.yahoo.com:50060/tasklog?plaintext=true&taskid=task_200804300435_0001_m_000000_2&filter=stderr
>   08/04/30 04:38:05 INFO mapred.JobClient: Task Id :
>   task_200804300435_0001_r_000000_2, Status : FAILED
>   08/04/30 04:38:05 WARN mapred.JobClient: Error reading task
>   outputhttp://
> richbreak-lr.eglbp.corp.yahoo.com:50060/tasklog?plaintext=true&taskid=task_200804300435_0001_r_000000_2&filter=stdout
>   08/04/30 04:38:05 WARN mapred.JobClient: Error reading task
>   outputhttp://
> richbreak-lr.eglbp.corp.yahoo.com:50060/tasklog?plaintext=true&taskid=task_200804300435_0001_r_000000_2&filter=stderr
>   08/04/30 04:38:05 INFO mapred.JobClient: Task Id :
>   task_200804300435_0001_m_000001_2, Status : FAILED
>   08/04/30 04:38:05 WARN mapred.JobClient: Error reading task
>   outputhttp://
> richbreak-lr.eglbp.corp.yahoo.com:50061/tasklog?plaintext=true&taskid=task_200804300435_0001_m_000001_2&filter=stdout
>   08/04/30 04:38:05 WARN mapred.JobClient: Error reading task
>   outputhttp://
> richbreak-lr.eglbp.corp.yahoo.com:50061/tasklog?plaintext=true&taskid=task_200804300435_0001_m_000001_2&filter=stderr
>   08/04/30 04:38:08 INFO mapred.JobClient:  map 100% reduce 100%
>   08/04/30 04:38:08 INFO mapred.JobClient: Task Id :
>   task_200804300435_0001_r_000000_3, Status : TIPFAILED
>   08/04/30 04:38:08 WARN mapred.JobClient: Error reading task
>   outputhttp://
> richbreak-lr.eglbp.corp.yahoo.com:50061/tasklog?plaintext=true&taskid=task_200804300435_0001_r_000000_3&filter=stdout
>   08/04/30 04:38:08 WARN mapred.JobClient: Error reading task
>   outputhttp://
> richbreak-lr.eglbp.corp.yahoo.com:50061/tasklog?plaintext=true&taskid=task_200804300435_0001_r_000000_3&filter=stderr
>   08/04/30 04:38:08 INFO mapred.JobClient: Task Id :
>   task_200804300435_0001_m_000000_3, Status : TIPFAILED
>   08/04/30 04:38:08 WARN mapred.JobClient: Error reading task
>   outputhttp://
> richbreak-lr.eglbp.corp.yahoo.com:50062/tasklog?plaintext=true&taskid=task_200804300435_0001_m_000000_3&filter=stdout
>   08/04/30 04:38:08 WARN mapred.JobClient: Error reading task
>   outputhttp://
> richbreak-lr.eglbp.corp.yahoo.com:50062/tasklog?plaintext=true&taskid=task_200804300435_0001_m_000000_3&filter=stderr
>   java.io.IOException: Job failed!
>           at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:831)
>           at org.apache.hadoop.examples.WordCount.run(WordCount.java:147)
>           at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>           at org.apache.hadoop.examples.WordCount.main(WordCount.java:153)
>           at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>           at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>           at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>           at java.lang.reflect.Method.invoke(Method.java:597)
>           at
>
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
>           at
>   org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
>           at
>   org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:49)
>           at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>           at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>           at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>           at java.lang.reflect.Method.invoke(Method.java:597)
>           at org.apache.hadoop.util.RunJar.main(RunJar.java:155)
>
> The same sequence when tried with an actual cluster setup on my local
> machine works perfectly.
>
> Am I doing something wrong? Should I use the MiniCluster in a different
> way?
> I am using revision 631443, a bit old but this is the one that exists in
> the type branch..
>
> Thanks,
> --Shravan
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message