Return-Path: X-Original-To: apmail-ignite-user-archive@minotaur.apache.org Delivered-To: apmail-ignite-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 9F47018AAD for ; Fri, 5 Feb 2016 12:44:10 +0000 (UTC) Received: (qmail 93481 invoked by uid 500); 5 Feb 2016 12:44:10 -0000 Delivered-To: apmail-ignite-user-archive@ignite.apache.org Received: (qmail 93434 invoked by uid 500); 5 Feb 2016 12:44:10 -0000 Mailing-List: contact user-help@ignite.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@ignite.apache.org Delivered-To: mailing list user@ignite.apache.org Received: (qmail 93424 invoked by uid 99); 5 Feb 2016 12:44:10 -0000 Received: from Unknown (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 05 Feb 2016 12:44:10 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 09CB5C0D58 for ; Fri, 5 Feb 2016 12:44:10 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.28 X-Spam-Level: * X-Spam-Status: No, score=1.28 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, WEIRD_PORT=0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gridgain-com.20150623.gappssmtp.com Received: from mx1-us-east.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id XkufhTqfMRW6 for ; Fri, 5 Feb 2016 12:44:08 +0000 (UTC) Received: from mail-lb0-f179.google.com (mail-lb0-f179.google.com [209.85.217.179]) by mx1-us-east.apache.org (ASF Mail Server at mx1-us-east.apache.org) with ESMTPS id A32D342BC3 for ; Fri, 5 Feb 2016 12:44:07 +0000 (UTC) Received: by mail-lb0-f179.google.com with SMTP id dx2so48857011lbd.3 for ; Fri, 05 Feb 2016 04:44:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gridgain-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=vcLZ48o8KsS052TFSk9hCn6tqdoWxyxLYmAfPATWjSU=; b=KsdGB9kPNsLa4WKtgLuffTYGu0KTvQmIwaUYHOz6OnizjmfxDHFdQ/1TOU+TXybZCe k31ZEsZDmPCPyaOEIx5WfqSItdkA59SELrXEAb2h9CmAKX9xijPxIz+wJHmdZjWVJ7hG PTqBYtCAnsCAjPhBqBDB/CHz690jYAznFg1XGIoYhCGML/AsItrMQAhnNG/dOAa0Rtnk w8GMWZRdKHpSmkN1vaUMBHIinSTRAeRlXQJFYxVXXo4VbzmOoZUTzmDgS77ObNxjue5R YecnqhS1/5ss1ON0bOxpBffd0NtYxYlXQd7qy7q9//x09C4ES8VbM6aAwz1NpOrDjoOm yUVw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=vcLZ48o8KsS052TFSk9hCn6tqdoWxyxLYmAfPATWjSU=; b=V1J/b6Mj9kx0N9AJQ68Ci50RdOrP4tk4MsOSIXQ+y1HgnQHn/wkkokrTd/eOHBe5/B Q14xyBjBrJluOSfVyY5B9J8aBFA8BwKK6o4s8a4VR9RwQDIrIE1fLBT7k3iNLIZyMM8d mS8dXY6A4ItbP5Kr6mRWhl6DxFy3H06sRESzSTVcKGsZr3XFTpjmeCrMkn67wGRTpX8Q 0GpmpwOPy3DrmV5I8BMk/g4I8Z2H+dmwZ6UaQsrC974CGFw9qDWe9h43uFnUJeU8SK4g B0DD5twoxZMfeAKTcxXsjI+6PF597E9xPA1MuRTzmWxdYBnupBBU7e73Qy5uzOoMoEDn KSIA== X-Gm-Message-State: AG10YOTHufl6MKlbIKAb6ukTRzVSHhfb+K0jj26SjN7PoA8HO8ky5Gpteab4kIZTAc0iF9W16WRIrVUmhjYJazO8 MIME-Version: 1.0 X-Received: by 10.112.172.233 with SMTP id bf9mr5917392lbc.121.1454676240522; Fri, 05 Feb 2016 04:44:00 -0800 (PST) Received: by 10.114.77.102 with HTTP; Fri, 5 Feb 2016 04:44:00 -0800 (PST) In-Reply-To: References: <1454622967955-2840.post@n6.nabble.com> <1454625724982-2841.post@n6.nabble.com> <1454626634024-2842.post@n6.nabble.com> Date: Fri, 5 Feb 2016 15:44:00 +0300 Message-ID: Subject: Re: Apache Drill querying IGFS-accelerated (H)DFS? From: Vladimir Ozerov To: user@ignite.apache.org Content-Type: multipart/alternative; boundary=001a11c3277e947e3f052b053453 --001a11c3277e947e3f052b053453 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Petar, Looks like this could be what we need - Storage Plugin - https://drill.apache.org/docs/plugin-configuration-basics/ Could you please try configuring new plugin for IGFS? If the problem is still there, could you please provide detailed error description and possibly logs? Vladimir. On Fri, Feb 5, 2016 at 3:40 PM, Vladimir Ozerov wrote: > Petar, > > Yes, I mean setting igfs://igfs@localhost:10500 to Drill's config. I see > in your email that you typed "ifgs" instead of "igfs". Is it a typo in > email or in Drill configuration as well? Please try changing it to "igfs" > and possibly restart Drill instance because may be it simply didn't pick > latest changes yet. > > Anyways, the general rule is that you need to register IGFS file system i= n > some Hadoop configuration file (usually this is core-site.xml): > > > fs.igfs.impl > org.apache.ignite.hadoop.fs.v1.IgniteHadoopFileSystem<= /property> > > > ... and then "feed" this configuration file to Drill somehow. As you trie= d > to execute a command over IGFS and it worked fine, it means that *Hadoop > is already configured correctly*. The main question is why Drill do not > respect Hadoop settings. > > I do not have much experience with Drill, so could please try to looking > for any Drill properties which point to Hadoop configuration file(s) and > then check if these files really contain mentioned "fs.igfs.impl" propert= y? > > Vladimir. > > On Fri, Feb 5, 2016 at 3:14 PM, pshomov wrote: > >> >> =E2=80=8BHi Vladimir, >> >> Thank you for keeping very speedy responses to my questions! Much, much >> appreciated! >> >> I apparently missed the point where I am supposed to run an ignite node >> outside of hadoop, I thought it would spin one in process. Anyways I >> followed your instructions and setup a secondaryFileSystem (btw, your >> sample was for code that is not released yet, so I used this instead >> >> > class=3D"org.apache.ignite.hadoop.fs.IgniteHadoopIgfsSecondaryFileSystem= "> >> > value=3D"hdfs://localhost:9000"/> >> > value=3D"/Users/petar/src/as/igfs/hadoop-2.7.1/etc/hadoop/core-site.xml"= /> >> >> >> I think this should be fine, right? >> ), ran an ignite node and then started hadoop and was able to list my >> files using >> >> bin/hadoop fs -ls igfs://igfs@localhost:10500/ >> >> >> However the last part did not happen: >> >> >Once link between IGFS and HDFS is set, you can add IGFS URL >> "igfs://igfs@/" to Apache Drill configuration and try quering data. >> >> Drill keeps insisting it knows nothing about igfs:// schema. When you sa= y >> apache Drill configuration do you mean opening >> http://localhost:8047/storage/dfs and modifying that to be like this >> >> { >> "type": "file", >> "enabled": true, >> "connection": "ifgs://igfs@localhost:10500", >> "workspaces": { >> "petar": { >> "location": "/", >> "writable": false, >> "defaultInputFormat": null >> } >> }, >> =E2=80=A6. >> } >> >> >> Or do mean some .conf file in the config folder of Drill? >> Thank you for your tremendous help once again! >> >> Best regards, >> >> Petar >> >> ------------------------------ >> View this message in context: Re: Apache Drill querying IGFS-accelerated >> (H)DFS? >> >> Sent from the Apache Ignite Users mailing list archive >> at Nabble.com. >> > > --001a11c3277e947e3f052b053453 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Petar,

Looks like this could be what we= need - Storage Plugin -=C2=A0https://drill.apache.org/docs/plugin-configuratio= n-basics/
Could you please try configuring new plugin for IGF= S?

If the problem is still there, could you p= lease provide detailed error description and possibly logs?

<= /div>
Vladimir.

On Fri, Feb 5, 2016 at 3:40 PM, Vladimir Ozerov <vozerov@= gridgain.com> wrote:
Petar,

Yes, I mean setting igfs://igfs@localh= ost:10500 to Drill's config. I see in your email that you typed "i= fgs" instead of "igfs". Is it= a typo in email or in Drill configuration as well? Please try changing it = to "igfs" and possibly restart Drill instance because may be it s= imply didn't pick latest changes yet.

Anyways,= the general rule is that you need to register IGFS file system in some Had= oop configuration file (usually this is core-site.xml):

<property>
    <name>fs=
.igfs.impl</name>
    <value>o=
rg.apache.ignite.hadoop.fs.v1.IgniteHadoopFileSystem</value=
>
</property> =

... and then "feed" this = configuration file to Drill somehow. As you tried to execute a command over= IGFS and it worked fine, it means that Hadoop is already configured cor= rectly. The main question is why Drill do not respect Hadoop settings.= =C2=A0

I do not have much experience with Drill, s= o could please try to looking for any Drill properties which point to Hadoo= p configuration file(s) and then check if these files really contain mentio= ned "fs.igfs.impl" property?

Vladimir.

On Fri, Feb 5, 2016 at 3:14 PM, pshomov <petar@a= ctivitystream.com> wrote:
<= div dir=3D"ltr">
=E2=80=8BHi Vladimir,

Thank you for = keeping very speedy responses to my questions! Much, much appreciated!

I apparently missed the point where I am supposed to r= un an ignite node outside of hadoop, I thought it would spin one in process= . Anyways I followed your instructions and setup a secondaryFileSystem (btw= , your sample was for code that is not released yet, so I used this instead=
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 <property name=3D"secondaryFileSystem">
=
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 <bean class=3D"org.apache.ignite.hadoop.fs.IgniteHado= opIgfsSecondaryFileSystem">
=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 <constructor-arg name=3D"uri" value=3D"hdfs://localho= st:9000"/>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 <constructor-arg na= me=3D"cfgPath" value=3D"/Users/petar/src/as/igfs/hadoop-2.7.= 1/etc/hadoop/core-site.xml"/>
=C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 </bean>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= </property>
I think this should be fine, right?
= ), ran an ignite node and then started hadoop and was able to list my files= using=C2=A0

bin/hadoop fs -ls igfs://igfs@localho= st:10500/


However the last part did not happen:

>Once link between IGFS and HDFS is set, you can add IGFS URL "igfs://= igfs@/" to Apache Drill configuration and try quering data.

=

Drill k= eeps insisting it knows nothing about igfs:// schema. When you say apache D= rill configuration do you mean opening=C2=A0= http://localhost:8047/storage/dfs and modifying that to be like this

{
=C2=A0 "type": "file= ",
=C2=A0 "enabled": true,
=C2=A0 "= connection": "ifgs://igfs@localhost:10500",
=C2=A0= "workspaces": {
=C2=A0 =C2=A0 "petar": {
=C2=A0 =C2=A0 =C2=A0 "location": "/",
= =C2=A0 =C2=A0 =C2=A0 "writable": false,
=C2=A0 =C2=A0 = =C2=A0 "defaultInputFormat": null
=C2=A0 =C2=A0 }
=
=C2=A0 },
=C2=A0=E2=80=A6.
}

<= /div>

Or do mean some .conf file in the config folder of= Drill?
Thank you for your tremendous help once again!
=
Best regards,

Petar
=09 =09 =09
View this message in context: Re: Apache Drill querying IGFS-accelerated (H)DFS? Sent from the Apache Ignite Users mailing list archive at Nabble.com= .


--001a11c3277e947e3f052b053453--