ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vladimir Ozerov <voze...@gridgain.com>
Subject Re: Apache Drill querying IGFS-accelerated (H)DFS?
Date Fri, 05 Feb 2016 10:39:04 GMT
Petar,

IGFS configuration consists of two steps: starting Ignite node and
adjusting Hadoop configuration.

*1) Starting Ignite node:*

   - Download Apache Ignite Hadoop Accelerator (
   http://ignite.apache.org/download.cgi#binaries) and unpack it.
   - If you want to link IGFS and HDFS, please add the following property
   to the bean *org.apache.ignite.configuration.FileSystemConfiguration*
    inside *config/default-config.xml*:

<property name="secondaryFileSystem">
    <bean class="org.apache.ignite.hadoop.fs.IgniteHadoopIgfsSecondaryFileSystem">
        <property name="fileSystemFactory">
            <bean
class="org.apache.ignite.hadoop.fs.CachingHadoopFileSystemFactory">
                <property name="uri" value="hdfs://localhost:9000"/>
                <property name="configPaths">
                    <list>
                        <value>*/path/to/your/core-site.xml*</value>
                    </list>
                </property>
            </bean>
        </property>
    </bean>
</property>



   - Run bin/ignite.sh

At this point you will have Ignite node which is able to accept IGFS
requests at port 10500 (see *config/default-config.xml*, section
*ipcEndpointConfiguration*).

*2) Configuring Hadoop:*

   - Copy the following 3 Ignite JARS to
   *${HADOOP_HOME}/share/hadoop/common/lib* so that Hadoop is able to
   instantiate IGFS client:
      - libs/ignite-core-[VERSION].jar
      - libs/ignite-shmem-1.0.0.jar
      - libs/ignite-hadoop/ignite-hadoop-[VERSION].jar


   - Go to your *core-site.xml* and set map "igfs" URL to IGFS client class:
   -

   <property>
       <name>fs.igfs.impl</name>
       <value>org.apache.ignite.hadoop.fs.v1.IgniteHadoopFileSystem</value></property>



   - At this point you should be able to start querying IGFS. E.g. "hadoop
   -fs ls igfs://igfs@/"

Once link between IGFS and HDFS is set, you can add IGFS URL "igfs://igfs@/"
to Apache Drill configuration and try quering data.

Please let me know if you have any further problems during setup.

Vladimir.


On Fri, Feb 5, 2016 at 12:34 PM, pshomov <petar@activitystream.com> wrote:

> Hi Vladimir,
>
> Changed back fs.defaultFS (in my case to hdfs://localhost:9000) and the
> HDFS service is back on its feet ;)
> My problem is that Apache Drill does not accept igfs:// scheme (No
> FileSystem for scheme: figs . It knows pretty well the hdfs:// schema
> though. I am guessing that accessing the hdfs:// bypasses the IGFS
> acceleration altogether, right?
>
> Best regards,
>
> Petar
>
>
>
> ------------------------------
> View this message in context: Re: Apache Drill querying IGFS-accelerated
> (H)DFS?
> <http://apache-ignite-users.70518.x6.nabble.com/Apache-Drill-querying-IGFS-accelerated-H-DFS-tp2840p2850.html>
>
> Sent from the Apache Ignite Users mailing list archive
> <http://apache-ignite-users.70518.x6.nabble.com/> at Nabble.com.
>

Mime
View raw message