Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of publicnetworkservices@gmail.com
 designates 209.85.128.45 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAOcnVr0iODZG5Fjh9W=n_HWVA2SBCRodOmZo3QLiTgH5s1Y-_A@mail.gmail.com>
References: 
 <CAPwWofFGaErjKuvOx98igkMA82jXgQfTCbaYy4+OoX7BaRYDyA@mail.gmail.com>
	<CAOcnVr0iODZG5Fjh9W=n_HWVA2SBCRodOmZo3QLiTgH5s1Y-_A@mail.gmail.com>
Date: Thu, 21 Feb 2013 22:25:31 -0800
Message-ID: 
 <CAPwWofE19nfzPVf1Kyd+2LDoBZYCmSD3iYgr_0RnSy3790i4_g@mail.gmail.com>
Subject: Re: MapReduce processing with extra (possibly non-serializable)
 configuration
From: Public Network Services <publicnetworkservices@gmail.com>
To: user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00248c768dc612eaf004d64a424f

--00248c768dc612eaf004d64a424f
Content-Type: text/plain; charset=ISO-8859-1

Hazelcast is an interesting idea, but I was hoping that there is a way of
doing this in MapReduce. :-)

It didn't seem like that from the start, but I posted here just to make
sure I was not missing something.

So, I will serialize my data objects and use them accordingly.

Thanks!


On Thu, Feb 21, 2013 at 10:15 PM, Harsh J <harsh@cloudera.com> wrote:

> How do you imagine sending "data" of any kind (be it in object form,
> etc.) over the network to other nodes, without implementing or relying
> on a serialization for it? Are you looking for "easy" Java ways such
> as the distributed cache from Hazelcast, etc., where this may be taken
> care for you automatically in some way? :)
>
> On Fri, Feb 22, 2013 at 2:40 AM, Public Network Services
> <publicnetworkservices@gmail.com> wrote:
> > Hi...
> >
> > I am trying to put an existing file processing application into Hadoop
> and
> > need to find the best way of propagating some extra configuration per
> split,
> > in the form of complex and proprietary custom Java objects.
> >
> > The general idea is
> >
> > A custom InputFormat splits the input data
> > The same InputFormat prepares the appropriate configuration for each
> split
> > Hadoop processes each split in MapReduce, using the split itself and the
> > corresponding configuration
> >
> > The problem is that these configuration objects contain a lot of
> properties
> > and references to other complex objects, and so on, therefore it will
> take a
> > lot of work to cover all the possible combinations and make the whole
> thing
> > serializable (if it can be done in the first place).
> >
> > Most probably this is the only way forward, but if anyone has ever dealt
> > with this problem, please suggest the best approach to follow.
> >
> > Thanks!
> >
>
>
>
> --
> Harsh J
>

--00248c768dc612eaf004d64a424f
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hazelcast is an interesting idea, but I was hoping that there is a way of d=
oing this in MapReduce. :-)<div><br></div><div>It didn&#39;t seem like that=
 from the start, but I posted here just to make sure I was not missing some=
thing.</div>
<div><br></div><div>So, I will serialize my data objects and use them accor=
dingly.</div><div><br></div><div>Thanks!</div><div><br><br><div class=3D"gm=
ail_quote">On Thu, Feb 21, 2013 at 10:15 PM, Harsh J <span dir=3D"ltr">&lt;=
<a href=3D"mailto:harsh@cloudera.com" target=3D"_blank">harsh@cloudera.com<=
/a>&gt;</span> wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">How do you imagine sending &quot;data&quot; =
of any kind (be it in object form,<br>
etc.) over the network to other nodes, without implementing or relying<br>
on a serialization for it? Are you looking for &quot;easy&quot; Java ways s=
uch<br>
as the distributed cache from Hazelcast, etc., where this may be taken<br>
care for you automatically in some way? :)<br>
<div><div class=3D"h5"><br>
On Fri, Feb 22, 2013 at 2:40 AM, Public Network Services<br>
&lt;<a href=3D"mailto:publicnetworkservices@gmail.com">publicnetworkservice=
s@gmail.com</a>&gt; wrote:<br>
&gt; Hi...<br>
&gt;<br>
&gt; I am trying to put an existing file processing application into Hadoop=
 and<br>
&gt; need to find the best way of propagating some extra configuration per =
split,<br>
&gt; in the form of complex and proprietary custom Java objects.<br>
&gt;<br>
&gt; The general idea is<br>
&gt;<br>
&gt; A custom InputFormat splits the input data<br>
&gt; The same InputFormat prepares the appropriate configuration for each s=
plit<br>
&gt; Hadoop processes each split in MapReduce, using the split itself and t=
he<br>
&gt; corresponding configuration<br>
&gt;<br>
&gt; The problem is that these configuration objects contain a lot of prope=
rties<br>
&gt; and references to other complex objects, and so on, therefore it will =
take a<br>
&gt; lot of work to cover all the possible combinations and make the whole =
thing<br>
&gt; serializable (if it can be done in the first place).<br>
&gt;<br>
&gt; Most probably this is the only way forward, but if anyone has ever dea=
lt<br>
&gt; with this problem, please suggest the best approach to follow.<br>
&gt;<br>
&gt; Thanks!<br>
&gt;<br>
<br>
<br>
<br>
</div></div>--<br>
Harsh J<br>
</blockquote></div><br></div>

--00248c768dc612eaf004d64a424f--