Mailing-List: contact core-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-user@hadoop.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
In-Reply-To: <48B6A227.1020800@in.tum.de>
References: <BBDD0DDF-C9A2-443B-9888-CC4BBA0043BB@cs.utexas.edu>
 <b76e3e240808280041o5eade1b8xd1893eab29bc55fc@mail.gmail.com>
 <F9082F3D-62D7-4425-A122-0AC7683B28D1@cis.upenn.edu>
 <48B6A227.1020800@in.tum.de>
Mime-Version: 1.0 (Apple Message framework v753.1)
Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed
Message-Id: <0EFDE18E-65C9-4CB4-809E-B0F5DBB889A5@cis.upenn.edu>
Cc: Benjamin Gufler <benjamin.gufler@in.tum.de>
Content-Transfer-Encoding: 7bit
From: Shirley Cohen <shirleyc@cis.upenn.edu>
Subject: Re: MultipleOutputFormat versus MultipleOutputs
Date: Fri, 29 Aug 2008 10:13:49 -0500
To: core-user@hadoop.apache.org

Thanks, Benjamin. Your example saved me a lot of time :))

Shirley

On Aug 28, 2008, at 8:03 AM, Benjamin Gufler wrote:

> Hi Shirley,
>
> On 2008-08-28 14:32, Shirley Cohen wrote:
>> Do you have an example that shows how to use MultipleOutputFormat?
>
> using MultipleOutputFormat is actually pretty easy. Derive a class  
> from
> it, overriding - if you want to base the destination file name on the
> key and/or value - the method "generateFileNameForKeyValue". I'm using
> it this way:
>
> protected String generateFileNameForKeyValue(K key, V value,
>         String name) {
>     return name + "-" + key.toString();
> }
>
> Pay attention at not generating too many different file names,  
> however:
> All the files are kept open until the Reducer terminates, and  
> operating
> systems usually impose a limit on open files you can have.
>
> Also, if you haven't done so yet, please upgrade to the latest  
> release,
> 0.18, if you want to use MultipleOutputFormat. Up to 0.17.2, there was
> some trouble with Reducers having more than one output file (see
> HADOOP-3639 for the details).
>
> Benjamin