incubator-drill-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Dunning <ted.dunn...@gmail.com>
Subject Re: Test Case for Inability to run a Distributed Drill Query (on JSON files on FS).
Date Fri, 07 Nov 2014 16:04:54 GMT
Does anybody have a thought about this?

It seems to me like Chris has drillbits that are running their own ZK nodes
internally.

Is that true?

Would it be possible to have drill clients check for consistent ZK cluster
definitions when connecting to drill bits?



On Wed, Oct 29, 2014 at 4:30 AM, Chris Drawater <chris.drawater@jdsu.com>
wrote:

> Hi Ramana,
>
>
>
>
>
> We've isolated the problem!   Below is our simple test case whereby a SQL
> query only runs local and not distributed.
>
>
>
> Our single zookeeper instance is running only on node Drill1  amd the
> foreman drillbit is running on Drill2.
>
>
>
> SQL -->
>
>         use dfs;
>
>         select xbin,ybin,numberOfConnections  from
> `/geo/appdata/rawdata/2014/SameNamedata.json`;
>
>
>
> where SameNamedata.json is present on  nodes : Drill1, Drill2 and Drill3,
> and is of the form -->
>
>
>
> { "NumberOfConnections": 1, "StartTime": "2014-10-
> 02T14:30:00Z", "EndTime": "2014-10-02T14:45:00Z", "XBin": 566150, "YBin":
> 3270100 }
>
> { "NumberOfUniqueSubscribers": 2, "NumberOfConnections":
> 2, "StartTime": "2014-10-02T14:30:00Z", "EndTime": "2014-10-
> 02T14:45:00Z", "XBin": 571350, "YBin": 3268100, "AverageSnr": -
> 6.2870720195528573 }
>
>
>
> On the 'foreman' node (Drill2) ,  drillbit.log  reveals the following
> trace -->
>
>
>
> 2014-10-28 10:55:06,883 [UserServer-1] DEBUG
> o.a.drill.exec.rpc.user.UserServer - Received query to run.  Returning
> query handle.
>
> 2014-10-28 10:55:06,893 [WorkManager Event Thread] DEBUG
> o.apache.drill.exec.work.WorkManager - Starting pending task
> org.apache.drill.exec.work.WorkManager$RunnableWrapper@bec4752
>
> 2014-10-28 10:55:06,911 [091c7c1d-fb6b-4685-9f1d-04f08afd1053:foreman]
> DEBUG o.a.d.e.s.h.HBaseStoragePluginConfig - Initializing HBase
> StoragePlugin configuration with zookeeper quorum 'localhost', port '2181'.
>
> 2014-10-28 10:55:06,912 [091c7c1d-fb6b-4685-9f1d-04f08afd1053:foreman]
> DEBUG o.a.drill.exec.store.SchemaFactory - Took 17 ms to register schemas.
>
> 2014-10-28 10:55:06,953 [091c7c1d-fb6b-4685-9f1d-04f08afd1053:foreman]
> DEBUG o.a.d.e.p.s.h.DefaultSqlHandler - Optiq Logical :
>
> ProjectRel(xbin=[$1], ybin=[$2], numberOfConnections=[$3]): rowcount =
> 100.0, cumulative cost = {200.0 rows, 401.0 cpu, 0.0 io, 0.0 network, 0.0
> memory}, id = 1259
>
>   EnumerableTableAccessRel(table=
> [[dfs, /geo/appdata/rawdata/2014/SameNamedata.json]]): rowcount = 100.0,
> cumulative cost = {100.0 rows, 101.0 cpu, 0.0 io, 0.0 network, 0.0 memory},
> id = 1256
>
>
>
> The line -->
>
>         2014-10-28 10:55:06,911 [091c7c1d-fb6b-4685-9f1d-
> 04f08afd1053:foreman] DEBUG o.a.d.e.s.h.HBaseStoragePluginConfig -
> Initializing HBase StoragePlugin configuration with zookeeper
> quorum 'localhost', port '2181'.
>
> may be the key.
>
>
>
> We're not using HBase and their is no Zookeeper instance running local on
> Drill2 !!
>
>
>
> May be a bug ?  (Although it might still be our configuration I suppose).
>
>
>
>
>
> Cheers,
>
>    Chris
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message