ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vladimir Ozerov <voze...@gridgain.com>
Subject Re: Apache Drill querying IGFS-accelerated (H)DFS?
Date Fri, 05 Feb 2016 12:40:25 GMT

Yes, I mean setting igfs://igfs@localhost:10500 to Drill's config. I see in
your email that you typed "ifgs" instead of "igfs". Is it a typo in email
or in Drill configuration as well? Please try changing it to "igfs" and
possibly restart Drill instance because may be it simply didn't pick latest
changes yet.

Anyways, the general rule is that you need to register IGFS file system in
some Hadoop configuration file (usually this is core-site.xml):


... and then "feed" this configuration file to Drill somehow. As you tried
to execute a command over IGFS and it worked fine, it means that *Hadoop is
already configured correctly*. The main question is why Drill do not
respect Hadoop settings.

I do not have much experience with Drill, so could please try to looking
for any Drill properties which point to Hadoop configuration file(s) and
then check if these files really contain mentioned "fs.igfs.impl" property?


On Fri, Feb 5, 2016 at 3:14 PM, pshomov <petar@activitystream.com> wrote:

> ​Hi Vladimir,
> Thank you for keeping very speedy responses to my questions! Much, much
> appreciated!
> I apparently missed the point where I am supposed to run an ignite node
> outside of hadoop, I thought it would spin one in process. Anyways I
> followed your instructions and setup a secondaryFileSystem (btw, your
> sample was for code that is not released yet, so I used this instead
>                     <property name="secondaryFileSystem">
>                         <bean
> class="org.apache.ignite.hadoop.fs.IgniteHadoopIgfsSecondaryFileSystem">
>                             <constructor-arg name="uri"
> value="hdfs://localhost:9000"/>
>                             <constructor-arg name="cfgPath"
> value="/Users/petar/src/as/igfs/hadoop-2.7.1/etc/hadoop/core-site.xml"/>
>                         </bean>
>                     </property>
> I think this should be fine, right?
> ), ran an ignite node and then started hadoop and was able to list my
> files using
> bin/hadoop fs -ls igfs://igfs@localhost:10500/
> However the last part did not happen:
> >Once link between IGFS and HDFS is set, you can add IGFS URL
> "igfs://igfs@/" to Apache Drill configuration and try quering data.
> Drill keeps insisting it knows nothing about igfs:// schema. When you say
> apache Drill configuration do you mean opening
> http://localhost:8047/storage/dfs and modifying that to be like this
> {
>   "type": "file",
>   "enabled": true,
>   "connection": "ifgs://igfs@localhost:10500",
>   "workspaces": {
>     "petar": {
>       "location": "/",
>       "writable": false,
>       "defaultInputFormat": null
>     }
>   },
>  ….
> }
> Or do mean some .conf file in the config folder of Drill?
> Thank you for your tremendous help once again!
> Best regards,
> Petar
> ------------------------------
> View this message in context: Re: Apache Drill querying IGFS-accelerated
> (H)DFS?
> <http://apache-ignite-users.70518.x6.nabble.com/Apache-Drill-querying-IGFS-accelerated-H-DFS-tp2840p2855.html>
> Sent from the Apache Ignite Users mailing list archive
> <http://apache-ignite-users.70518.x6.nabble.com/> at Nabble.com.

View raw message