ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Denis Magda <dma...@apache.org>
Subject Re: ignite-spark module in Hadoop Accelerator
Date Thu, 01 Dec 2016 06:54:01 GMT
Vovan,

As one of hadoop maintainers, please share your point of view on this.

—
Denis

> On Nov 30, 2016, at 10:49 PM, Sergey Kozlov <skozlov@gridgain.com> wrote:
> 
> Denis
> 
> I agree that at the moment there's no reason to split into fabric and
> hadoop editions.
> 
> On Thu, Dec 1, 2016 at 4:45 AM, Denis Magda <dmagda@apache.org> wrote:
> 
>> Hadoop Accelerator doesn’t require any additional libraries in compare to
>> those we have in the fabric build. It only lacks some of them as Val
>> mentioned below.
>> 
>> Wouldn’t it better to discontinue Hadoop Accelerator edition and simply
>> deliver hadoop jar and its configs as a part of the fabric?
>> 
>> —
>> Denis
>> 
>>> On Nov 27, 2016, at 3:12 PM, Dmitriy Setrakyan <dsetrakyan@apache.org>
>> wrote:
>>> 
>>> Separate edition for the Hadoop Accelerator was primarily driven by the
>>> default libraries. Hadoop Accelerator requires many more libraries as
>> well
>>> as configuration settings compared to the standard fabric download.
>>> 
>>> Now, as far as spark integration is concerned, I am not sure which
>> edition
>>> it belongs in, Hadoop Accelerator or standard fabric.
>>> 
>>> D.
>>> 
>>> On Sat, Nov 26, 2016 at 7:39 PM, Denis Magda <dmagda@apache.org> wrote:
>>> 
>>>> *Dmitriy*,
>>>> 
>>>> I do believe that you should know why the community decided to a
>> separate
>>>> edition for the Hadoop Accelerator. What was the reason for that?
>>>> Presently, as I see, it brings more confusion and difficulties rather
>> then
>>>> benefit.
>>>> 
>>>> —
>>>> Denis
>>>> 
>>>> On Nov 26, 2016, at 2:14 PM, Konstantin Boudnik <cos@apache.org> wrote:
>>>> 
>>>> In fact I am very much agree with you. Right now, running the
>> "accelerator"
>>>> component in Bigtop disto gives one a pretty much complete fabric
>> anyway.
>>>> But
>>>> in order to make just an accelerator component we perform quite a bit of
>>>> woodoo magic during the packaging stage of the Bigtop build, shuffling
>> jars
>>>> from here and there. And that's quite crazy, honestly ;)
>>>> 
>>>> Cos
>>>> 
>>>> On Mon, Nov 21, 2016 at 03:33PM, Valentin Kulichenko wrote:
>>>> 
>>>> I tend to agree with Denis. I see only these differences between Hadoop
>>>> Accelerator and Fabric builds (correct me if I miss something):
>>>> 
>>>> - Limited set of available modules and no optional modules in Hadoop
>>>> Accelerator.
>>>> - No ignite-hadoop module in Fabric.
>>>> - Additional scripts, configs and instructions included in Hadoop
>>>> Accelerator.
>>>> 
>>>> And the list of included modules frankly looks very weird. Here are only
>>>> some of the issues I noticed:
>>>> 
>>>> - ignite-indexing and ignite-spark are mandatory. Even if we need them
>>>> for Hadoop Acceleration (which I doubt), are they really required or
>> can
>>>> be
>>>> optional?
>>>> - We force to use ignite-log4j module without providing other logger
>>>> options (e.g., SLF).
>>>> - We don't include ignite-aws module. How to use Hadoop Accelerator
>> with
>>>> S3 discovery?
>>>> - Etc.
>>>> 
>>>> It seems to me that if we try to fix all this issue, there will be
>>>> virtually no difference between Fabric and Hadoop Accelerator builds
>> except
>>>> couple of scripts and config files. If so, there is no reason to have
>> two
>>>> builds.
>>>> 
>>>> -Val
>>>> 
>>>> On Mon, Nov 21, 2016 at 3:13 PM, Denis Magda <dmagda@apache.org> wrote:
>>>> 
>>>> On the separate note, in the Bigtop, we start looking into changing the
>>>> 
>>>> way we
>>>> 
>>>> deliver Ignite and we'll likely to start offering the whole 'data
>> fabric'
>>>> experience instead of the mere "hadoop-acceleration”.
>>>> 
>>>> 
>>>> And you still will be using hadoop-accelerator libs of Ignite, right?
>>>> 
>>>> I’m thinking of if there is a need to keep releasing Hadoop Accelerator
>> as
>>>> a separate delivery.
>>>> What if we start releasing the accelerator as a part of the standard
>>>> fabric binary putting hadoop-accelerator libs under ‘optional’ folder?
>>>> 
>>>> —
>>>> Denis
>>>> 
>>>> On Nov 21, 2016, at 12:19 PM, Konstantin Boudnik <cos@apache.org>
>> wrote:
>>>> 
>>>> What Denis said: spark has been added to the Hadoop accelerator as a way
>>>> 
>>>> to
>>>> 
>>>> boost the performance of more than just MR compute of the Hadoop stack,
>>>> 
>>>> IIRC.
>>>> 
>>>> For what it worth, Spark is considered a part of Hadoop at large.
>>>> 
>>>> On the separate note, in the Bigtop, we start looking into changing the
>>>> 
>>>> way we
>>>> 
>>>> deliver Ignite and we'll likely to start offering the whole 'data
>> fabric'
>>>> experience instead of the mere "hadoop-acceleration".
>>>> 
>>>> Cos
>>>> 
>>>> On Mon, Nov 21, 2016 at 09:54AM, Denis Magda wrote:
>>>> 
>>>> Val,
>>>> 
>>>> Ignite Hadoop module includes not only the map-reduce accelerator but
>>>> 
>>>> Ignite
>>>> 
>>>> Hadoop File System component as well. The latter can be used in
>>>> 
>>>> deployments
>>>> 
>>>> like HDFS+IGFS+Ignite Spark + Spark.
>>>> 
>>>> Considering this I’m for the second solution proposed by you: put both
>>>> 
>>>> 2.10
>>>> 
>>>> and 2.11 ignite-spark modules under ‘optional’ folder of Ignite Hadoop
>>>> Accelerator distribution.
>>>> https://issues.apache.org/jira/browse/IGNITE-4254 <
>>>> 
>>>> https://issues.apache.org/jira/browse/IGNITE-4254>
>>>> 
>>>> 
>>>> BTW, this task may be affected or related to the following ones:
>>>> https://issues.apache.org/jira/browse/IGNITE-3596 <
>>>> 
>>>> https://issues.apache.org/jira/browse/IGNITE-3596>
>>>> 
>>>> https://issues.apache.org/jira/browse/IGNITE-3822
>>>> 
>>>> —
>>>> Denis
>>>> 
>>>> On Nov 19, 2016, at 1:26 PM, Valentin Kulichenko <
>>>> 
>>>> valentin.kulichenko@gmail.com> wrote:
>>>> 
>>>> 
>>>> Hadoop Accelerator is a plugin to Ignite and this plugin is used by
>>>> 
>>>> Hadoop
>>>> 
>>>> when running its jobs. ignite-spark module only provides IgniteRDD
>>>> 
>>>> which
>>>> 
>>>> Hadoop obviously will never use.
>>>> 
>>>> Is there another use case for Hadoop Accelerator which I'm missing?
>>>> 
>>>> -Val
>>>> 
>>>> On Sat, Nov 19, 2016 at 3:12 AM, Dmitriy Setrakyan <
>>>> 
>>>> dsetrakyan@apache.org>
>>>> 
>>>> wrote:
>>>> 
>>>> Why do you think that spark module is not needed in our hadoop build?
>>>> 
>>>> On Fri, Nov 18, 2016 at 5:44 PM, Valentin Kulichenko <
>>>> valentin.kulichenko@gmail.com> wrote:
>>>> 
>>>> Folks,
>>>> 
>>>> Is there anyone who understands the purpose of including ignite-spark
>>>> module in the Hadoop Accelerator build? I can't figure out a use
>>>> 
>>>> case for
>>>> 
>>>> which it's needed.
>>>> 
>>>> In case we actually need it there, there is an issue then. We
>>>> 
>>>> actually
>>>> 
>>>> have
>>>> 
>>>> two ignite-spark modules, for 2.10 and 2.11. In Fabric build
>>>> 
>>>> everything
>>>> 
>>>> is
>>>> 
>>>> good, we put both in 'optional' folder and user can enable either
>>>> 
>>>> one.
>>>> 
>>>> But
>>>> 
>>>> in Hadoop Accelerator there is only 2.11 which means that the build
>>>> 
>>>> doesn't
>>>> 
>>>> work with 2.10 out of the box.
>>>> 
>>>> We should either remove the module from the build, or fix the issue.
>>>> 
>>>> -Val
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>> 
>> 
> 
> 
> -- 
> Sergey Kozlov
> GridGain Systems
> www.gridgain.com


Mime
View raw message