ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ivan Veselovsky (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (IGNITE-345) documentation: better describe secondary filesystem configuration
Date Wed, 25 Feb 2015 17:06:05 GMT

     [ https://issues.apache.org/jira/browse/IGNITE-345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Ivan Veselovsky updated IGNITE-345:
-----------------------------------
    Description: 
Currently the documentation on how to configure the secondary HDFS (http://hadoop.gridgain.org/latest/GGFS)
is not quite clear. 

1) The key point there is that this mode requires two different Hadoop configurations (and,
perhaps, installations): Hadoop server should have ordinary hdfs filesystem configured, like
hdfs://aaa:9000/ , while Hadoop client should use igfs:// filesystem , that, in turn, runs
upon the hdfs filesystem configured in the Hadoop server through the Ignite node(s). This
way, the Hadoop client  and Hadoop server have its own core-site.xml each. In case of local
deployment (everything on one machine) this may be done as 2 different  Hadoop configurations
or 2 different copies of Hadoop installation. We'd better provide comprehensive examples of
the configuration in the documentation.

2) the second point that is quite unclear is the purpose of "cfgPath" constructor attribute
(see below) of the Ignite node configuration. Actually it supposed to mean the path to the
*server* Hadoop configuration (core-site.xml), but actually it is optional (currently it is
impossible to omit this paramater, but it is possible to use empty config file.) This config
is used (1) to get the secondary hdfs URI, if the one is not given directly as "uri" paramater
, and (2) to get additional parameters of that filesystem, if any. The difficulty there is
that this config file may (and should?) actually reside on a different node (host), so it
may be tricky to provide local URL to this configuration. (We'll discuss possibility to omit
this paramater in configuration at all.) 
        <property name="igfsConfiguration">
            <list>
                <bean class="org.apache.ignite.configuration.IgfsConfiguration" parent="igfsCfgBase">
                    <property name="name" value="igfs"/>
                    ....
                    <property name="defaultMode" value="PROXY"/>

                    <property name="secondaryFileSystem">
                        <bean class="org.apache.ignite.internal.igfs.hadoop.IgfsHadoopFileSystemWrapper">
                            <constructor-arg name="uri"     value="hdfs://172.17.0.4:9000"/>
                            <constructor-arg name="cfgPath" value="....../etc/hadoop/core-site.xml"/>

                        </bean>
                    </property>

                </bean>
            </list>
        </property>


  was:
Currently the documentation on how to configure the secondary HDFS (http://hadoop.gridgain.org/latest/GGFS)
is not quite clear. 

1) The key point there is that this mode requires two different Hadoop configurations (and,
perhaps, installations): Hadoop server should have ordinary hdfs filesystem configured, like
hdfs://aaa:9000/ , while Hadoop client should use igfs:// filesystem , that, in turn, runs
upon the hdfs filesystem configured in the Hadoop server. This way, the Hadoop client  and
Hadoop server have its own core-site.xml each. In case of local deployment (everything on
one machine) this may be done as 2 different  Hadoop configurations or 2 different copies
of Hadoop installation.

2) the second point that is quite unclear is the purpose of "cfgPath" constructor attribute
(see below) of the Ignite node configuration. Actually it supposed to mean the path to server
Hadoop configuration (core-site.xml), but actually it is optional (currently it is impossible
to omit this paramater, but it is possible to use empty config file.) This config is used
(1) to get the secondary hdfs URI, if the one is not given directly as "uri" paramater , and
(2) to get additional parameters of that filesystem, if any. The difficulty there is that
this config file may actually reside on a different node (host), so it may be tricky to provide
local URL to this configuration. (We'll discuss possibility to omit this paramater in configuration
at all.) 
        <property name="igfsConfiguration">
            <list>
                <bean class="org.apache.ignite.configuration.IgfsConfiguration" parent="igfsCfgBase">
                    <property name="name" value="igfs"/>
                    ....
                    <property name="defaultMode" value="PROXY"/>

                    <property name="secondaryFileSystem">
                        <bean class="org.apache.ignite.internal.igfs.hadoop.IgfsHadoopFileSystemWrapper">
                            <constructor-arg name="uri"     value="hdfs://172.17.0.4:9000"/>
                            <constructor-arg name="cfgPath" value="....../etc/hadoop/core-site.xml"/>

                        </bean>
                    </property>

                </bean>
            </list>
        </property>



> documentation: better describe secondary filesystem configuration
> -----------------------------------------------------------------
>
>                 Key: IGNITE-345
>                 URL: https://issues.apache.org/jira/browse/IGNITE-345
>             Project: Ignite
>          Issue Type: Wish
>          Components: documentation
>    Affects Versions: sprint-2
>            Reporter: Ivan Veselovsky
>
> Currently the documentation on how to configure the secondary HDFS (http://hadoop.gridgain.org/latest/GGFS)
is not quite clear. 
> 1) The key point there is that this mode requires two different Hadoop configurations
(and, perhaps, installations): Hadoop server should have ordinary hdfs filesystem configured,
like hdfs://aaa:9000/ , while Hadoop client should use igfs:// filesystem , that, in turn,
runs upon the hdfs filesystem configured in the Hadoop server through the Ignite node(s).
This way, the Hadoop client  and Hadoop server have its own core-site.xml each. In case of
local deployment (everything on one machine) this may be done as 2 different  Hadoop configurations
or 2 different copies of Hadoop installation. We'd better provide comprehensive examples of
the configuration in the documentation.
> 2) the second point that is quite unclear is the purpose of "cfgPath" constructor attribute
(see below) of the Ignite node configuration. Actually it supposed to mean the path to the
*server* Hadoop configuration (core-site.xml), but actually it is optional (currently it is
impossible to omit this paramater, but it is possible to use empty config file.) This config
is used (1) to get the secondary hdfs URI, if the one is not given directly as "uri" paramater
, and (2) to get additional parameters of that filesystem, if any. The difficulty there is
that this config file may (and should?) actually reside on a different node (host), so it
may be tricky to provide local URL to this configuration. (We'll discuss possibility to omit
this paramater in configuration at all.) 
>         <property name="igfsConfiguration">
>             <list>
>                 <bean class="org.apache.ignite.configuration.IgfsConfiguration" parent="igfsCfgBase">
>                     <property name="name" value="igfs"/>
>                     ....
>                     <property name="defaultMode" value="PROXY"/>
>                     <property name="secondaryFileSystem">
>                         <bean class="org.apache.ignite.internal.igfs.hadoop.IgfsHadoopFileSystemWrapper">
>                             <constructor-arg name="uri"     value="hdfs://172.17.0.4:9000"/>
>                             <constructor-arg name="cfgPath" value="....../etc/hadoop/core-site.xml"/>

>                         </bean>
>                     </property>
>                 </bean>
>             </list>
>         </property>



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message