Mailing-List: contact dev-help@ignite.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@ignite.apache.org
Date: Mon, 21 Nov 2016 12:19:47 -0800
From: Konstantin Boudnik <cos@apache.org>
To: dev@ignite.apache.org
Subject: Re: ignite-spark module in Hadoop Accelerator
Message-ID: <20161121201947.GP15754@boudnik.org>
Mail-Followup-To: dev@ignite.apache.org
References: <CABuYRcpe=nasjvhukfg6ixnLsUNuy8i0G-VTJsv6NAd=VX6cbw@mail.gmail.com>
 <CA+0=VoUndNG-3RUGkDMr6+VU7apBnhq3nqB9wzxTtDTTvc514g@mail.gmail.com>
 <CABuYRcrf6dxxSuS5BTr20eD9vGJmwb0Hd0Hv1t6ZRCK-xTQ9XQ@mail.gmail.com>
 <9E40FB60-4D04-4C10-8EC4-54C5DF63CF37@apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <9E40FB60-4D04-4C10-8EC4-54C5DF63CF37@apache.org>
User-Agent: Mutt/1.5.21 (2010-09-15)
archived-at: Mon, 21 Nov 2016 20:17:40 -0000

What Denis said: spark has been added to the Hadoop accelerator as a way to
boost the performance of more than just MR compute of the Hadoop stack, IIRC.
For what it worth, Spark is considered a part of Hadoop at large. 

On the separate note, in the Bigtop, we start looking into changing the way we
deliver Ignite and we'll likely to start offering the whole 'data fabric'
experience instead of the mere "hadoop-acceleration".

Cos

On Mon, Nov 21, 2016 at 09:54AM, Denis Magda wrote:
> Val,
> 
> Ignite Hadoop module includes not only the map-reduce accelerator but Ignite
> Hadoop File System component as well. The latter can be used in deployments
> like HDFS+IGFS+Ignite Spark + Spark. 
> 
> Considering this I’m for the second solution proposed by you: put both 2.10
> and 2.11 ignite-spark modules under ‘optional’ folder of Ignite Hadoop
> Accelerator distribution.
> https://issues.apache.org/jira/browse/IGNITE-4254 <https://issues.apache.org/jira/browse/IGNITE-4254>
> 
> BTW, this task may be affected or related to the following ones:
> https://issues.apache.org/jira/browse/IGNITE-3596 <https://issues.apache.org/jira/browse/IGNITE-3596>
> https://issues.apache.org/jira/browse/IGNITE-3822
> 
> —
> Denis
> 
> > On Nov 19, 2016, at 1:26 PM, Valentin Kulichenko <valentin.kulichenko@gmail.com> wrote:
> > 
> > Hadoop Accelerator is a plugin to Ignite and this plugin is used by Hadoop
> > when running its jobs. ignite-spark module only provides IgniteRDD which
> > Hadoop obviously will never use.
> > 
> > Is there another use case for Hadoop Accelerator which I'm missing?
> > 
> > -Val
> > 
> > On Sat, Nov 19, 2016 at 3:12 AM, Dmitriy Setrakyan <dsetrakyan@apache.org>
> > wrote:
> > 
> >> Why do you think that spark module is not needed in our hadoop build?
> >> 
> >> On Fri, Nov 18, 2016 at 5:44 PM, Valentin Kulichenko <
> >> valentin.kulichenko@gmail.com> wrote:
> >> 
> >>> Folks,
> >>> 
> >>> Is there anyone who understands the purpose of including ignite-spark
> >>> module in the Hadoop Accelerator build? I can't figure out a use case for
> >>> which it's needed.
> >>> 
> >>> In case we actually need it there, there is an issue then. We actually
> >> have
> >>> two ignite-spark modules, for 2.10 and 2.11. In Fabric build everything
> >> is
> >>> good, we put both in 'optional' folder and user can enable either one.
> >> But
> >>> in Hadoop Accelerator there is only 2.11 which means that the build
> >> doesn't
> >>> work with 2.10 out of the box.
> >>> 
> >>> We should either remove the module from the build, or fix the issue.
> >>> 
> >>> -Val
> >>> 
> >> 
>