Mailing-List: contact user-help@ignite.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@ignite.apache.org
MIME-Version: 1.0
In-Reply-To: <1461144698190-4364.post@n6.nabble.com>
References: <1461138097037-4357.post@n6.nabble.com>
	<CAGt6o8n9aXXtQbo=yyksAeg0KFF3EXC-CCZoubs16WVbxic9qg@mail.gmail.com>
	<1461144698190-4364.post@n6.nabble.com>
Date: Wed, 20 Apr 2016 15:07:44 +0300
Message-ID: 
 <CAGt6o8kPO-9CwJMYABG6C_N=fqzRy5ENF10Uiz+nZoSR0WXzUA@mail.gmail.com>
Subject: Re: Map-reduce proceesing
From: Vladimir Ozerov <vozerov@gridgain.com>
To: user@ignite.apache.org
Content-Type: multipart/alternative; boundary=001a11432512f75ff90530e9708d

--001a11432512f75ff90530e9708d
Content-Type: text/plain; charset=UTF-8

Hi,

If you broadcast the job and want to iterate over cache inside it, then
please make sure that you iterate only over local entries (e.g.
IgniteCache.localEntries(), ScanQuery.setLocal(true), etc.). Otherwise your
jobs will duplicate work and performance will suffer.

Also please note that returned result set might be incomplete if one of the
nodes failed during job processing. If you care about it, you should either
implement some failover, or use Ignite's built-in queries (ScanQuery,
SqlQuery) which already take care of it.

Anyway, I strongly recommend you to focus on SqlQuery first. You can
configure indexes on cache and they could give you great boost, because
instead of iterating over the whole cache, Ignite will use indexes for fast
data lookup.

Vladimir.

On Wed, Apr 20, 2016 at 12:31 PM, dmreshet <dmreshet@gmail.com> wrote:

> Yes, I know.
> I want to compare performance of SQL,  SQL with indexes and MapReduce job.
> I have found that I can use broadcast to garantie that my MapReduce job
> will
> be executed on each node exactly once.
> So now my job uses code:
> /Collection<List&lt;Person> result =
>
> ignite.compute(ignite.cluster()).broadcast((IgniteCallable<List&lt;Person>>)
> () -> {...});/
>
> And than I will reduce the result.
>
> Is that the best practise to implement MapReduce job in case that I should
> process data from cache?
>
>
>
> --
> View this message in context:
> http://apache-ignite-users.70518.x6.nabble.com/Map-reduce-proceesing-tp4357p4364.html
> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>

--001a11432512f75ff90530e9708d
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi,<div><br></div><div>If you broadcast the job and want t=
o iterate over cache inside it, then please make sure that you iterate only=
 over local entries (e.g. IgniteCache.localEntries(), ScanQuery.setLocal(tr=
ue), etc.). Otherwise your jobs will duplicate work and performance will su=
ffer.</div><div><br></div><div>Also please note that returned result set mi=
ght be incomplete if one of the nodes failed during job processing. If you =
care about it, you should either implement some failover, or use Ignite&#39=
;s built-in queries (ScanQuery, SqlQuery) which already take care of it.</d=
iv><div><br></div><div>Anyway, I strongly recommend you to focus on SqlQuer=
y first. You can configure indexes on cache and they could give you great b=
oost, because instead of iterating over the whole cache, Ignite will use in=
dexes for fast data lookup.</div><div><br></div><div>Vladimir.</div></div><=
div class=3D"gmail_extra"><br><div class=3D"gmail_quote">On Wed, Apr 20, 20=
16 at 12:31 PM, dmreshet <span dir=3D"ltr">&lt;<a href=3D"mailto:dmreshet@g=
mail.com" target=3D"_blank">dmreshet@gmail.com</a>&gt;</span> wrote:<br><bl=
ockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #=
ccc solid;padding-left:1ex">Yes, I know.<br>
I want to compare performance of SQL,=C2=A0 SQL with indexes and MapReduce =
job.<br>
I have found that I can use broadcast to garantie that my MapReduce job wil=
l<br>
be executed on each node exactly once.<br>
So now my job uses code:<br>
/Collection&lt;List&amp;lt;Person&gt; result =3D<br>
ignite.compute(ignite.cluster()).broadcast((IgniteCallable&lt;List&amp;lt;P=
erson&gt;&gt;)<br>
() -&gt; {...});/<br>
<br>
And than I will reduce the result.<br>
<br>
Is that the best practise to implement MapReduce job in case that I should<=
br>
process data from cache?<br>
<br>
<br>
<br>
--<br>
View this message in context: <a href=3D"http://apache-ignite-users.70518.x=
6.nabble.com/Map-reduce-proceesing-tp4357p4364.html" rel=3D"noreferrer" tar=
get=3D"_blank">http://apache-ignite-users.70518.x6.nabble.com/Map-reduce-pr=
oceesing-tp4357p4364.html</a><br>
<div class=3D"HOEnZb"><div class=3D"h5">Sent from the Apache Ignite Users m=
ailing list archive at Nabble.com.<br>
</div></div></blockquote></div><br></div>

--001a11432512f75ff90530e9708d--