Mailing-List: contact core-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=rogers.com;
  h=X-YMail-OSG:Received:Date:From:Subject:To:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-ID;
  b=Dc6gcmaLnuEEPKpyjYxPJDA/+ZAraz3HkFwSMvmGY1KelSM0TKmPozA0jlYD+kEh0XB6ZSu0bceq1higidLLvHRvlTVNVBtSQJFvkKrWoAjXB9uTGFULU2IeN1oUS3Co4C8+enL7SBsGzYGqOdgHlkfHqoWrFvmtXraL5rHoDXg=;
Date: Wed, 7 May 2008 19:38:10 -0400 (EDT)
From: Derek Shaw <derek.shaw@rogers.com>
Subject: Fwd: Collecting output not to file
To: core-user@hadoop.apache.org
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="0-475996443-1210203490=:47651"
Content-Transfer-Encoding: 8bit
Message-ID: <362726.47651.qm@web88105.mail.re2.yahoo.com>

--0-475996443-1210203490=:47651
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: 8bit

To clarify:
 
     static class TestOutputFormat
         implements OutputFormat <Text, Text>
     {
         static class TestRecordWriter
             implements RecordWriter <Text, Text>
         {
             TestOutputFormat output;
             
             public TestRecordWriter (TestOutputFormat output, org.apache.hadoop.fs.FileSystem ignored, JobConf job, String name, Progressable progress)
             {
                 this.output = output;
             }
             
             public void close (Reporter reporter)
             {}
             
             public void write (Text key, Text value)
             {
                 output.addResults (value.toString ());
             }
         }
         
         protected String results = "";
                 
         public void checkOutputSpecs (org.apache.hadoop.fs.FileSystem ignored, JobConf job)
             throws IOException
         {}
         
         public RecordWriter <Text, Text> getRecordWriter (org.apache.hadoop.fs.FileSystem ignored, JobConf job, String name, Progressable progress)
         {
             return new TestRecordWriter (this, ignored, job, name, progress);
         }
         
         public void addResults (String r)
         {
             results += r + ",";
         }
         
         public String getResults ()
         {
             return results;
         }
     }

 And then running the task:
 public int run(String[] args) 
         throws Exception 
     {
     ....
     JobClient.runJob(job);
         
     // getOutputFormatcreates a new instance of the outputformat. I want to get the instance of the output format that the reduce function wrote to
 // The recordWriter that reduce wrote to would be just as good
         TestOutputFormat results = (TestOutputFormat) job.getOutputFormat ();  
   
 // Always prints the empty string, not the populated results
         System.out.println ("results: " + results.getResults ());   
         
         return 0;
     }

Derek Shaw <derek.shaw@rogers.com> wrote: Date: Tue, 6 May 2008 23:26:30 -0400 (EDT)
From: Derek Shaw <derek.shaw@rogers.com>
Subject: Collecting output not to file
To: core-user@hadoop.apache.org

 Hey,

>From the examples that I have seen thus far, all of the results from the reduce function are being written to a file. Instead of writing results to a file, I want to store them and inspect them after the job is completed. (I think that I need to implement my own OutputCollector, but I don't know how to tell hadoop to use it.) How can I do this?

-Derek


--0-475996443-1210203490=:47651--