Mailing-List: contact dev-help@ignite.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@ignite.apache.org
Subject: Re: Using HDFS as a secondary FS
To: dev@ignite.apache.org
References: 
 <CABuYRcp1fM6_80jVuyUBOwFsPruWnvpEHz9PSfF9zP-mQwODLQ@mail.gmail.com>
 <CAC15qqV1kX4TWRZaY5mjDsLo0sefp8ryGnJa4teN3AWZvpU1tw@mail.gmail.com>
From: Denis Magda <dmagda@gridgain.com>
Message-ID: <566E79CC.8040303@gridgain.com>
Date: Mon, 14 Dec 2015 11:11:56 +0300
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101
 Thunderbird/38.4.0
MIME-Version: 1.0
In-Reply-To: 
 <CAC15qqV1kX4TWRZaY5mjDsLo0sefp8ryGnJa4teN3AWZvpU1tw@mail.gmail.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit

Hi Ivan,

1) Yes, I think that it makes sense to have the old versions of the docs 
while an old version is still considered to be used by someone.

2) Absolutely, the time to add a corresponding article on the readme.io 
has come. It's not the first time I see the question related to HDFS as 
a secondary FS.
Before and now it's not clear for me what exact steps I should follow to 
enable such a configuration. Our current suggestions look like a puzzle.
I'll assemble the puzzle on my side and prepare the article. Ivan if you 
don't mind I would reaching you out directly asking for any technical 
assistance if needed.

Regards,
Denis

On 12/14/2015 10:25 AM, Ivan V. wrote:
> Hi, Valentin,
>
> 1) first of all note that the author of the question uses not the latest
> doc page, namely
> http://apacheignite.gridgain.org/v1.0/docs/igfs-secondary-file-system .
> This is version 1.0, while the latest is 1.5:
> https://apacheignite.readme.io/docs/hadoop-accelerator. Besides, it
> appeared that some links from the latest doc version point to 1.0 doc
> version. I fixed that in several places where I found that. Do we really
> need old doc versions (1.0 -1.4)?
>
> 2) our documentation (
> http://apacheignite.gridgain.org/docs/secondary-file-system) does not
> provide any special setup instructions to configure HDFS as secondary file
> system in Ignite. Our docs assume that if a user wants to integrate with
> Hadoop, (s)he follows generic Hadoop integration instruction (e.g.
> http://apacheignite.gridgain.org/docs/installing-on-apache-hadoop). It
> looks like the page
> http://apacheignite.gridgain.org/docs/secondary-file-system should be more
> clear regarding the required configuration steps (in fact, setting up
> HADOOP_HOME variable for Ignite node process).
>
> 3) Hadoop jars are correctly found by Ignite if the following conditions
> are met:
> (a) The "Hadoop Edition" distribution is used (not a "Fabric" edition).
> (b) Either HADOOP_HOME environment variable is set up (for Apache Hadoop
> distribution), or file "/etc/default/hadoop" exists and matches the Hadoop
> distribution used (BigTop, Cloudera, HDP, etc.)
>
> The exact mechanism of the Hadoop classpath composition can be found in
> files
> IGNITE_HOME/bin/include/hadoop-classpath.sh
> IGNITE_HOME/bin/include/setenv.sh .
>
> The issue is discussed in https://issues.apache.org/jira/browse/IGNITE-372
> , https://issues.apache.org/jira/browse/IGNITE-483 .
>
> On Sat, Dec 12, 2015 at 3:45 AM, Valentin Kulichenko <
> valentin.kulichenko@gmail.com> wrote:
>
>> Igniters,
>>
>> I'm looking at the question on SO [1] and I'm a bit confused.
>>
>> We ship ignite-hadoop module only in Hadoop Accelerator and without Hadoop
>> JARs, assuming that user will include them from the Hadoop distribution he
>> uses. It seems OK for me when accelerator is plugged in to Hadoop to run
>> mapreduce jobs, but I can't figure out steps required to configure HDFS as
>> a secondary FS for IGFS. Which Hadoop JARs should be on classpath? Is user
>> supposed to add them manually?
>>
>> Can someone with more expertise in our Hadoop integration clarify this? I
>> believe there is not enough documentation on this topic.
>>
>> BTW, any ideas why user gets exception for JobConf class which is in
>> 'mapred' package? Why map-reduce class is being used?
>>
>> [1]
>>
>> http://stackoverflow.com/questions/34221355/apache-ignite-what-are-the-dependencies-of-ignitehadoopigfssecondaryfilesystem
>>
>> -Val
>>