Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of idryman@gmail.com designates
 209.85.192.172 as permitted sender)
Content-Type: text/plain; charset=iso-8859-1
Mime-Version: 1.0 (Mac OS X Mail 6.6 \(1510\))
Subject: Re: Capture Directory Context in Hadoop Mapper
From: Felix Chern <idryman@gmail.com>
In-Reply-To: 
 <CAOcnVr0dYve=AcbvNDZm2jY1qgh21DpOZMnK9JhPr+TpxT=cxw@mail.gmail.com>
Date: Thu, 30 Jan 2014 10:15:28 -0800
Content-Transfer-Encoding: quoted-printable
Message-Id: <00730058-BBF7-43A4-8975-771D2FC62B32@gmail.com>
References: <C4008E8F-062A-4A79-B7F5-AE4849B0A226@gmail.com>
 <CAOcnVr0dYve=AcbvNDZm2jY1qgh21DpOZMnK9JhPr+TpxT=cxw@mail.gmail.com>
To: user@hadoop.apache.org

MultipleInputs is nice. Most of the time, I use it for reduce-side join.
It's great, however, you'll need to specify different Mapper class per =
input directory.
In our case, we try to let the Mapper itself to capture the directory =
information, because these directories might contain
data across months, and the the file structures may differ a bit time by =
time.
Finally, this is the solution I came up with, and it's fun to hack on =
lower level APIs. :D

Yet, thanks for suggesting!

Felix

On Jan 29, 2014, at 10:15 PM, Harsh J <harsh@cloudera.com> wrote:

> Hi,
>=20
> These posts are nicely written - thanks for sharing! Have you also
> taken a look at the MultipleInputs feature, which gives you a cleaner
> approach? =
http://hadoop.apache.org/docs/stable2/api/org/apache/hadoop/mapreduce/lib/=
input/MultipleInputs.html
>=20
> On Thu, Jan 30, 2014 at 4:16 AM, Felix Chern <idryman@gmail.com> =
wrote:
>> Hi all,
>>=20
>> I wrote a tutorial of how to receive path information in Mapper =
class. It's
>> useful in our hadoop use case where we need to apply different logic =
on
>> different input source directory. Enjoy!
>>=20
>> =
http://www.idryman.org/blog/2014/01/26/capture-directory-context-in-hadoop=
-mapper/
>> =
http://www.idryman.org/blog/2014/01/27/capture-path-info-in-hadoop-inputfo=
rmat-class/
>>=20
>> Felix
>=20
>=20
>=20
> --=20
> Harsh J