ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vladimir Ozerov <voze...@gridgain.com>
Subject Re: Using HDFS as a secondary FS
Date Mon, 14 Dec 2015 08:23:25 GMT
Valya,

Because we decide whether to load Hadoop module based on its availability
in classpath. And when Hadoop module is loaded, certain restrictions are
applied to configuration, e.g. peerClassLoadingEnabled must be false.
All this looks very inconvenient for me, but this is how things currently
work.

Vladimir.

On Mon, Dec 14, 2015 at 11:21 AM, Valentin Kulichenko <
valentin.kulichenko@gmail.com> wrote:

> Guys,
>
> Why don't we include ignite-hadoop module in Fabric? This user simply wants
> to configure HDFS as a secondary file system to ensure persistence. Not
> having the opportunity to do this in Fabric looks weird to me. And actually
> I don't think this is a use case for Hadoop Accelerator.
>
> -Val
>
> On Mon, Dec 14, 2015 at 12:11 AM, Denis Magda <dmagda@gridgain.com> wrote:
>
> > Hi Ivan,
> >
> > 1) Yes, I think that it makes sense to have the old versions of the docs
> > while an old version is still considered to be used by someone.
> >
> > 2) Absolutely, the time to add a corresponding article on the readme.io
> > has come. It's not the first time I see the question related to HDFS as a
> > secondary FS.
> > Before and now it's not clear for me what exact steps I should follow to
> > enable such a configuration. Our current suggestions look like a puzzle.
> > I'll assemble the puzzle on my side and prepare the article. Ivan if you
> > don't mind I would reaching you out directly asking for any technical
> > assistance if needed.
> >
> > Regards,
> > Denis
> >
> >
> > On 12/14/2015 10:25 AM, Ivan V. wrote:
> >
> >> Hi, Valentin,
> >>
> >> 1) first of all note that the author of the question uses not the latest
> >> doc page, namely
> >> http://apacheignite.gridgain.org/v1.0/docs/igfs-secondary-file-system .
> >> This is version 1.0, while the latest is 1.5:
> >> https://apacheignite.readme.io/docs/hadoop-accelerator. Besides, it
> >> appeared that some links from the latest doc version point to 1.0 doc
> >> version. I fixed that in several places where I found that. Do we really
> >> need old doc versions (1.0 -1.4)?
> >>
> >> 2) our documentation (
> >> http://apacheignite.gridgain.org/docs/secondary-file-system) does not
> >> provide any special setup instructions to configure HDFS as secondary
> file
> >> system in Ignite. Our docs assume that if a user wants to integrate with
> >> Hadoop, (s)he follows generic Hadoop integration instruction (e.g.
> >> http://apacheignite.gridgain.org/docs/installing-on-apache-hadoop). It
> >> looks like the page
> >> http://apacheignite.gridgain.org/docs/secondary-file-system should be
> >> more
> >> clear regarding the required configuration steps (in fact, setting up
> >> HADOOP_HOME variable for Ignite node process).
> >>
> >> 3) Hadoop jars are correctly found by Ignite if the following conditions
> >> are met:
> >> (a) The "Hadoop Edition" distribution is used (not a "Fabric" edition).
> >> (b) Either HADOOP_HOME environment variable is set up (for Apache Hadoop
> >> distribution), or file "/etc/default/hadoop" exists and matches the
> Hadoop
> >> distribution used (BigTop, Cloudera, HDP, etc.)
> >>
> >> The exact mechanism of the Hadoop classpath composition can be found in
> >> files
> >> IGNITE_HOME/bin/include/hadoop-classpath.sh
> >> IGNITE_HOME/bin/include/setenv.sh .
> >>
> >> The issue is discussed in
> >> https://issues.apache.org/jira/browse/IGNITE-372
> >> , https://issues.apache.org/jira/browse/IGNITE-483 .
> >>
> >> On Sat, Dec 12, 2015 at 3:45 AM, Valentin Kulichenko <
> >> valentin.kulichenko@gmail.com> wrote:
> >>
> >> Igniters,
> >>>
> >>> I'm looking at the question on SO [1] and I'm a bit confused.
> >>>
> >>> We ship ignite-hadoop module only in Hadoop Accelerator and without
> >>> Hadoop
> >>> JARs, assuming that user will include them from the Hadoop distribution
> >>> he
> >>> uses. It seems OK for me when accelerator is plugged in to Hadoop to
> run
> >>> mapreduce jobs, but I can't figure out steps required to configure HDFS
> >>> as
> >>> a secondary FS for IGFS. Which Hadoop JARs should be on classpath? Is
> >>> user
> >>> supposed to add them manually?
> >>>
> >>> Can someone with more expertise in our Hadoop integration clarify
> this? I
> >>> believe there is not enough documentation on this topic.
> >>>
> >>> BTW, any ideas why user gets exception for JobConf class which is in
> >>> 'mapred' package? Why map-reduce class is being used?
> >>>
> >>> [1]
> >>>
> >>>
> >>>
> http://stackoverflow.com/questions/34221355/apache-ignite-what-are-the-dependencies-of-ignitehadoopigfssecondaryfilesystem
> >>>
> >>> -Val
> >>>
> >>>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message