Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of harsh@cloudera.com designates
 209.85.223.173 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAPwWofFGaErjKuvOx98igkMA82jXgQfTCbaYy4+OoX7BaRYDyA@mail.gmail.com>
References: 
 <CAPwWofFGaErjKuvOx98igkMA82jXgQfTCbaYy4+OoX7BaRYDyA@mail.gmail.com>
From: Harsh J <harsh@cloudera.com>
Date: Fri, 22 Feb 2013 11:45:01 +0530
Message-ID: 
 <CAOcnVr0iODZG5Fjh9W=n_HWVA2SBCRodOmZo3QLiTgH5s1Y-_A@mail.gmail.com>
Subject: Re: MapReduce processing with extra (possibly non-serializable)
 configuration
To: "<user@hadoop.apache.org>" <user@hadoop.apache.org>
Content-Type: text/plain; charset=ISO-8859-1

How do you imagine sending "data" of any kind (be it in object form,
etc.) over the network to other nodes, without implementing or relying
on a serialization for it? Are you looking for "easy" Java ways such
as the distributed cache from Hazelcast, etc., where this may be taken
care for you automatically in some way? :)

On Fri, Feb 22, 2013 at 2:40 AM, Public Network Services
<publicnetworkservices@gmail.com> wrote:
> Hi...
>
> I am trying to put an existing file processing application into Hadoop and
> need to find the best way of propagating some extra configuration per split,
> in the form of complex and proprietary custom Java objects.
>
> The general idea is
>
> A custom InputFormat splits the input data
> The same InputFormat prepares the appropriate configuration for each split
> Hadoop processes each split in MapReduce, using the split itself and the
> corresponding configuration
>
> The problem is that these configuration objects contain a lot of properties
> and references to other complex objects, and so on, therefore it will take a
> lot of work to cover all the possible combinations and make the whole thing
> serializable (if it can be done in the first place).
>
> Most probably this is the only way forward, but if anyone has ever dealt
> with this problem, please suggest the best approach to follow.
>
> Thanks!
>


--
Harsh J