hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alejandro Abdelnur <tuc...@gmail.com>
Subject Re: Hadoop multiple output files
Date Tue, 29 Jun 2010 04:10:34 GMT
Adam,

Yes, you have do some configuration in the jobconf before submitting
the job. if the javadocs are not clear enough, check the testcase

A

On Mon, Jun 28, 2010 at 8:42 PM, Adam Silberstein
<silberst@yahoo-inc.com> wrote:
> Hi Alejandro,
> Thanks for the tip.  I tried the class, but got the following error:
>
> java.lang.IllegalArgumentException: Undefined named output 'text'
>  at
> org.apache.hadoop.mapred.lib.MultipleOutputs.getCollector(MultipleOutputs.ja
> va:496)
>  at
> org.apache.hadoop.mapred.lib.MultipleOutputs.getCollector(MultipleOutputs.ja
> va:476)
>  at
> com.yahoo.shadoop.applications.SyntheticToHdfsTableAndIndex$SortReducer.redu
> ce(Unknown Source)
>  at
> com.yahoo.shadoop.applications.SyntheticToHdfsTableAndIndex$SortReducer.redu
> ce(Unknown Source)
>  at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:463)
>  at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:411)
>  at org.apache.hadoop.mapred.Child.main(Child.java:170)
>
>
> I am trying to write to an output file named 'text.'  Do I need to
> initialize that file in some way?  I tried making a directory with the name,
> but that didn't do anything.
>
> Thanks,
> Adam
>
>
> On 6/28/10 6:17 PM, "Alejandro Abdelnur" <tucu00@gmail.com> wrote:
>
>> Check the MultipleOutputs class
>>
>> http://hadoop.apache.org/common/docs/r0.20.0/api/org/apache/hadoop/mapred/lib/
>> MultipleOutputs.html
>>
>> On Mon, Jun 28, 2010 at 5:31 PM, Adam Silberstein
>> <silberst@yahoo-inc.com> wrote:
>>>
>>> Hi,
>>> I would like to run a hadoop job that write to multiple output files.  I see
>>> a class called MultipleOutputFormat that looks like what I want, but I have
>>> not been able to find any sample code showing how to use it.  I see
>>> discussion of it in a JIRA, where the idea is to choose output file based on
>>> key.  If someone could me to a sample, that would be great.
>>>
>>> Thanks,
>>> Adam
>>>
>>>
>
>

Mime
View raw message