Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of dextermorgan4u@gmail.com
 designates 209.85.215.48 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CABtCYFzx4k36x0-V8M8T_ovSc6sd4EL-Hhn56jNHRbG5UGbQSA@mail.gmail.com>
References: 
 <CABtCYFzx4k36x0-V8M8T_ovSc6sd4EL-Hhn56jNHRbG5UGbQSA@mail.gmail.com>
Date: Mon, 27 Aug 2012 23:24:21 +0300
Message-ID: 
 <CABtCYFwY3qoO75m6=0u1TOqN+G452tMbM-4fpJDFc9=WmcUwtg@mail.gmail.com>
Subject: Re: best way to join?
From: dexter morgan <dextermorgan4u@gmail.com>
To: user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=f46d04088ef56d09fe04c8451c01

--f46d04088ef56d09fe04c8451c01
Content-Type: text/plain; charset=ISO-8859-1

Dear list,
>
> Lets say i have a file, like this:
> id \t at,tlng <-- structure
>
> 1\t40.123,-50.432
> 2\t41.431,-43.32
> ...
> ...
> lets call it: 'points.txt'
> I'm trying to build a map-reduce job that runs over this BIG points file
> and it should output
> a file, that will look like:
> id[lat,lng] \t [list of points in JSON standard] <--- structure
>
> 1[40.123,-50.432]\t[[41.431,-43.32],[...,...],...,[...]]
> 2[41.431,-43.32]\t[[40.123,-50.432],...[,]]
> ...
>
> Basically it should run on ITSELF, and grab for each point the N (it will
> be an argument for the job) CLOSEST points (the mappers should calculate
> the distance)..
>
> Distributed cache is not an option, what else?  not sure if to classify it
> as a map-join , reduce-join or both?
> Would you do this in HIVE some how?
> Is it feasible in a single job?
>
> Would LOVE to hear your suggestions, code (if you think its not that hard)
> or what not.
> BTW using CDH3 - rev 3 (20.23)
>
> Thanks!
>

--f46d04088ef56d09fe04c8451c01
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><br><br><div class=3D"gmail_quote"><br><blockquote class=
=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padd=
ing-left:1ex"><div dir=3D"ltr">Dear list,<div><br></div><div>Lets say i hav=
e a file, like this:</div>
<div>id \t at,tlng &lt;-- structure</div><div><br></div><div>1\t40.123,-50.=
432</div><div>2\t41.431,-43.32</div><div>...</div><div>...</div>
<div>lets call it: &#39;points.txt&#39;</div><div>I&#39;m trying to build a=
 map-reduce job that runs over this BIG points file and it should output</d=
iv><div>a file, that will look like:</div><div>id[lat,lng] \t [list of poin=
ts in JSON standard] &lt;--- structure</div>

<div><br></div><div>1[40.123,-50.432]\t[[41.431,-43.32],[...,...],...,[...]=
]</div><div>2[41.431,-43.32]\t[[40.123,-50.432],...[,]]</div><div>...</div>=
<div><br></div><div>Basically it should run on ITSELF, and grab for each po=
int the N (it will be an argument for the job) CLOSEST points (the mappers =
should calculate the distance)..</div>

<div><br></div><div>Distributed cache is not an option, what else? =A0not s=
ure if to classify it as a map-join , reduce-join or both?</div><div>Would =
you do this in HIVE some how?=A0</div><div>Is it feasible in a single job?<=
/div>

<div><br></div><div>Would LOVE to hear your suggestions, code (if you think=
 its not that hard) or what not.</div><div>BTW using CDH3 - rev 3 (20.23)</=
div><div><br></div><div>Thanks!</div></div>
</blockquote></div><br></div>

--f46d04088ef56d09fe04c8451c01--