hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shravan M Narayanamurthy <shrav...@yahoo-inc.com>
Subject Use of MiniCluster
Date Wed, 30 Apr 2008 00:01:06 GMT
Hi,
    I am working on the types branch and am trying to use the 
MiniCluster class for testing Job Submission and Monitoring in the 
redesigned pipeline. I have hit into an issue.

The MiniCluster's Map Reduce part does not seem to work. I tried to 
submit a wordCount MR job that exists in the hadoop examples jar. It 
does not work. Here are the steps I followed:

    (i) Started the MiniCluster using this:

        package org.apache.pig.test;

        public class TestMiniCluster {
            public static void main(String[] args) throws
        InterruptedException {
                MiniCluster cluster = MiniCluster.buildCluster();
                while(true){
                    Thread.sleep(10000);
                }
            }
        }

    (ii) hadoop --config ~/pigtest/conf dfs -put /etc/passwd passwd

    (iii) hadoop --config ~/pigtest/conf dfs -ls
    Found 1 items
    /user/shravanm/passwd   <r 3>   2768    2008-04-30 04:37

    Verified that the file had been created

    (iv) hadoop --config ~/pigtest/conf jar hadoop-0.15.3-examples.jar
    wordcount passwd passwdCnt
    Got the following exceptions:
    08/04/30 04:37:41 INFO mapred.FileInputFormat: Total input paths to
    process : 1
    08/04/30 04:37:41 INFO mapred.JobClient: Running job:
    job_200804300435_0001
    08/04/30 04:37:42 INFO mapred.JobClient:  map 0% reduce 0%
    08/04/30 04:37:54 INFO mapred.JobClient: Task Id :
    task_200804300435_0001_m_000000_0, Status : FAILED
    08/04/30 04:37:55 WARN mapred.JobClient: Error reading task
    outputhttp://richbreak-lr.eglbp.corp.yahoo.com:50061/tasklog?plaintext=true&taskid=task_200804300435_0001_m_000000_0&filter=stdout
    08/04/30 04:37:55 WARN mapred.JobClient: Error reading task
    outputhttp://richbreak-lr.eglbp.corp.yahoo.com:50061/tasklog?plaintext=true&taskid=task_200804300435_0001_m_000000_0&filter=stderr
    08/04/30 04:37:55 INFO mapred.JobClient: Task Id :
    task_200804300435_0001_m_000001_0, Status : FAILED
    08/04/30 04:37:55 WARN mapred.JobClient: Error reading task
    outputhttp://richbreak-lr.eglbp.corp.yahoo.com:50063/tasklog?plaintext=true&taskid=task_200804300435_0001_m_000001_0&filter=stdout
    08/04/30 04:37:55 WARN mapred.JobClient: Error reading task
    outputhttp://richbreak-lr.eglbp.corp.yahoo.com:50063/tasklog?plaintext=true&taskid=task_200804300435_0001_m_000001_0&filter=stderr
    08/04/30 04:37:55 INFO mapred.JobClient: Task Id :
    task_200804300435_0001_r_000000_0, Status : FAILED
    08/04/30 04:37:55 WARN mapred.JobClient: Error reading task
    outputhttp://richbreak-lr.eglbp.corp.yahoo.com:50062/tasklog?plaintext=true&taskid=task_200804300435_0001_r_000000_0&filter=stdout
    08/04/30 04:37:55 WARN mapred.JobClient: Error reading task
    outputhttp://richbreak-lr.eglbp.corp.yahoo.com:50062/tasklog?plaintext=true&taskid=task_200804300435_0001_r_000000_0&filter=stderr
    08/04/30 04:37:58 INFO mapred.JobClient: Task Id :
    task_200804300435_0001_m_000001_1, Status : FAILED
    08/04/30 04:37:58 WARN mapred.JobClient: Error reading task
    outputhttp://richbreak-lr.eglbp.corp.yahoo.com:50062/tasklog?plaintext=true&taskid=task_200804300435_0001_m_000001_1&filter=stdout
    08/04/30 04:37:58 WARN mapred.JobClient: Error reading task
    outputhttp://richbreak-lr.eglbp.corp.yahoo.com:50062/tasklog?plaintext=true&taskid=task_200804300435_0001_m_000001_1&filter=stderr
    08/04/30 04:37:58 INFO mapred.JobClient: Task Id :
    task_200804300435_0001_m_000000_1, Status : FAILED
    08/04/30 04:37:58 WARN mapred.JobClient: Error reading task
    outputhttp://richbreak-lr.eglbp.corp.yahoo.com:50063/tasklog?plaintext=true&taskid=task_200804300435_0001_m_000000_1&filter=stdout
    08/04/30 04:37:58 WARN mapred.JobClient: Error reading task
    outputhttp://richbreak-lr.eglbp.corp.yahoo.com:50063/tasklog?plaintext=true&taskid=task_200804300435_0001_m_000000_1&filter=stderr
    08/04/30 04:38:01 INFO mapred.JobClient: Task Id :
    task_200804300435_0001_r_000000_1, Status : FAILED
    08/04/30 04:38:01 WARN mapred.JobClient: Error reading task
    outputhttp://richbreak-lr.eglbp.corp.yahoo.com:50063/tasklog?plaintext=true&taskid=task_200804300435_0001_r_000000_1&filter=stdout
    08/04/30 04:38:01 WARN mapred.JobClient: Error reading task
    outputhttp://richbreak-lr.eglbp.corp.yahoo.com:50063/tasklog?plaintext=true&taskid=task_200804300435_0001_r_000000_1&filter=stderr
    08/04/30 04:38:05 INFO mapred.JobClient: Task Id :
    task_200804300435_0001_m_000000_2, Status : FAILED
    08/04/30 04:38:05 WARN mapred.JobClient: Error reading task
    outputhttp://richbreak-lr.eglbp.corp.yahoo.com:50060/tasklog?plaintext=true&taskid=task_200804300435_0001_m_000000_2&filter=stdout
    08/04/30 04:38:05 WARN mapred.JobClient: Error reading task
    outputhttp://richbreak-lr.eglbp.corp.yahoo.com:50060/tasklog?plaintext=true&taskid=task_200804300435_0001_m_000000_2&filter=stderr
    08/04/30 04:38:05 INFO mapred.JobClient: Task Id :
    task_200804300435_0001_r_000000_2, Status : FAILED
    08/04/30 04:38:05 WARN mapred.JobClient: Error reading task
    outputhttp://richbreak-lr.eglbp.corp.yahoo.com:50060/tasklog?plaintext=true&taskid=task_200804300435_0001_r_000000_2&filter=stdout
    08/04/30 04:38:05 WARN mapred.JobClient: Error reading task
    outputhttp://richbreak-lr.eglbp.corp.yahoo.com:50060/tasklog?plaintext=true&taskid=task_200804300435_0001_r_000000_2&filter=stderr
    08/04/30 04:38:05 INFO mapred.JobClient: Task Id :
    task_200804300435_0001_m_000001_2, Status : FAILED
    08/04/30 04:38:05 WARN mapred.JobClient: Error reading task
    outputhttp://richbreak-lr.eglbp.corp.yahoo.com:50061/tasklog?plaintext=true&taskid=task_200804300435_0001_m_000001_2&filter=stdout
    08/04/30 04:38:05 WARN mapred.JobClient: Error reading task
    outputhttp://richbreak-lr.eglbp.corp.yahoo.com:50061/tasklog?plaintext=true&taskid=task_200804300435_0001_m_000001_2&filter=stderr
    08/04/30 04:38:08 INFO mapred.JobClient:  map 100% reduce 100%
    08/04/30 04:38:08 INFO mapred.JobClient: Task Id :
    task_200804300435_0001_r_000000_3, Status : TIPFAILED
    08/04/30 04:38:08 WARN mapred.JobClient: Error reading task
    outputhttp://richbreak-lr.eglbp.corp.yahoo.com:50061/tasklog?plaintext=true&taskid=task_200804300435_0001_r_000000_3&filter=stdout
    08/04/30 04:38:08 WARN mapred.JobClient: Error reading task
    outputhttp://richbreak-lr.eglbp.corp.yahoo.com:50061/tasklog?plaintext=true&taskid=task_200804300435_0001_r_000000_3&filter=stderr
    08/04/30 04:38:08 INFO mapred.JobClient: Task Id :
    task_200804300435_0001_m_000000_3, Status : TIPFAILED
    08/04/30 04:38:08 WARN mapred.JobClient: Error reading task
    outputhttp://richbreak-lr.eglbp.corp.yahoo.com:50062/tasklog?plaintext=true&taskid=task_200804300435_0001_m_000000_3&filter=stdout
    08/04/30 04:38:08 WARN mapred.JobClient: Error reading task
    outputhttp://richbreak-lr.eglbp.corp.yahoo.com:50062/tasklog?plaintext=true&taskid=task_200804300435_0001_m_000000_3&filter=stderr
    java.io.IOException: Job failed!
            at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:831)
            at org.apache.hadoop.examples.WordCount.run(WordCount.java:147)
            at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
            at org.apache.hadoop.examples.WordCount.main(WordCount.java:153)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
            at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:597)
            at
    org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
            at
    org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
            at
    org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:49)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at
    sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
            at
    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:597)
            at org.apache.hadoop.util.RunJar.main(RunJar.java:155)

The same sequence when tried with an actual cluster setup on my local 
machine works perfectly.

Am I doing something wrong? Should I use the MiniCluster in a different way?
I am using revision 631443, a bit old but this is the one that exists in 
the type branch..

Thanks,
--Shravan



Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message