ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Valentin Kulichenko <valentin.kuliche...@gmail.com>
Subject Re: Using HDFS as a secondary FS
Date Mon, 14 Dec 2015 08:21:05 GMT
Guys,

Why don't we include ignite-hadoop module in Fabric? This user simply wants
to configure HDFS as a secondary file system to ensure persistence. Not
having the opportunity to do this in Fabric looks weird to me. And actually
I don't think this is a use case for Hadoop Accelerator.

-Val

On Mon, Dec 14, 2015 at 12:11 AM, Denis Magda <dmagda@gridgain.com> wrote:

> Hi Ivan,
>
> 1) Yes, I think that it makes sense to have the old versions of the docs
> while an old version is still considered to be used by someone.
>
> 2) Absolutely, the time to add a corresponding article on the readme.io
> has come. It's not the first time I see the question related to HDFS as a
> secondary FS.
> Before and now it's not clear for me what exact steps I should follow to
> enable such a configuration. Our current suggestions look like a puzzle.
> I'll assemble the puzzle on my side and prepare the article. Ivan if you
> don't mind I would reaching you out directly asking for any technical
> assistance if needed.
>
> Regards,
> Denis
>
>
> On 12/14/2015 10:25 AM, Ivan V. wrote:
>
>> Hi, Valentin,
>>
>> 1) first of all note that the author of the question uses not the latest
>> doc page, namely
>> http://apacheignite.gridgain.org/v1.0/docs/igfs-secondary-file-system .
>> This is version 1.0, while the latest is 1.5:
>> https://apacheignite.readme.io/docs/hadoop-accelerator. Besides, it
>> appeared that some links from the latest doc version point to 1.0 doc
>> version. I fixed that in several places where I found that. Do we really
>> need old doc versions (1.0 -1.4)?
>>
>> 2) our documentation (
>> http://apacheignite.gridgain.org/docs/secondary-file-system) does not
>> provide any special setup instructions to configure HDFS as secondary file
>> system in Ignite. Our docs assume that if a user wants to integrate with
>> Hadoop, (s)he follows generic Hadoop integration instruction (e.g.
>> http://apacheignite.gridgain.org/docs/installing-on-apache-hadoop). It
>> looks like the page
>> http://apacheignite.gridgain.org/docs/secondary-file-system should be
>> more
>> clear regarding the required configuration steps (in fact, setting up
>> HADOOP_HOME variable for Ignite node process).
>>
>> 3) Hadoop jars are correctly found by Ignite if the following conditions
>> are met:
>> (a) The "Hadoop Edition" distribution is used (not a "Fabric" edition).
>> (b) Either HADOOP_HOME environment variable is set up (for Apache Hadoop
>> distribution), or file "/etc/default/hadoop" exists and matches the Hadoop
>> distribution used (BigTop, Cloudera, HDP, etc.)
>>
>> The exact mechanism of the Hadoop classpath composition can be found in
>> files
>> IGNITE_HOME/bin/include/hadoop-classpath.sh
>> IGNITE_HOME/bin/include/setenv.sh .
>>
>> The issue is discussed in
>> https://issues.apache.org/jira/browse/IGNITE-372
>> , https://issues.apache.org/jira/browse/IGNITE-483 .
>>
>> On Sat, Dec 12, 2015 at 3:45 AM, Valentin Kulichenko <
>> valentin.kulichenko@gmail.com> wrote:
>>
>> Igniters,
>>>
>>> I'm looking at the question on SO [1] and I'm a bit confused.
>>>
>>> We ship ignite-hadoop module only in Hadoop Accelerator and without
>>> Hadoop
>>> JARs, assuming that user will include them from the Hadoop distribution
>>> he
>>> uses. It seems OK for me when accelerator is plugged in to Hadoop to run
>>> mapreduce jobs, but I can't figure out steps required to configure HDFS
>>> as
>>> a secondary FS for IGFS. Which Hadoop JARs should be on classpath? Is
>>> user
>>> supposed to add them manually?
>>>
>>> Can someone with more expertise in our Hadoop integration clarify this? I
>>> believe there is not enough documentation on this topic.
>>>
>>> BTW, any ideas why user gets exception for JobConf class which is in
>>> 'mapred' package? Why map-reduce class is being used?
>>>
>>> [1]
>>>
>>>
>>> http://stackoverflow.com/questions/34221355/apache-ignite-what-are-the-dependencies-of-ignitehadoopigfssecondaryfilesystem
>>>
>>> -Val
>>>
>>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message