drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aditya <adityakish...@gmail.com>
Subject Re: DRILL-4199: Add Support for HBase 1.X - planning to merge
Date Tue, 21 Jun 2016 17:04:17 GMT
There should be another stack trace in drillbit.out when this happens,
could you please check that file.

On Mon, Jun 20, 2016 at 8:14 PM, qiang li <tiredqiang@gmail.com> wrote:

> Another issue is some time when I restart the node, the node can not be
> startup.
>
> Here is the exception.
> ache-drill-1.7.0/jars/drill-gis-1.7.0-SNAPSHOT.jar!/,
> jar:file:/usr/lib/apache-drill-1.7.0/jars/drill-memory-base-1.7.0-SNAPSHOT.jar!/]
> took 2800ms
> 2016-06-20 19:10:18,313 [main] INFO  o.a.d.e.s.s.PersistentStoreRegistry -
> Using the configured PStoreProvider class:
> 'org.apache.drill.exec.store.sys.store.provider.ZookeeperPersistentStoreProvider'.
> 2016-06-20 19:10:19,221 [main] INFO  o.apache.drill.exec.server.Drillbit -
> Construction completed (1529 ms).
> 2016-06-20 19:10:31,136 [main] WARN  o.apache.drill.exec.server.Drillbit -
> Failure on close()
> java.lang.NullPointerException: null
>         at
> org.apache.drill.exec.work.WorkManager.close(WorkManager.java:153)
> ~[drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
>         at
> org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:76)
> ~[drill-common-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
>         at
> org.apache.drill.common.AutoCloseables.close(AutoCloseables.java:64)
> ~[drill-common-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
>         at org.apache.drill.exec.server.Drillbit.close(Drillbit.java:159)
> [drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
>         at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:293)
> [drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
>         at org.apache.drill.exec.server.Drillbit.start(Drillbit.java:271)
> [drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
>         at org.apache.drill.exec.server.Drillbit.main(Drillbit.java:267)
> [drill-java-exec-1.7.0-SNAPSHOT.jar:1.7.0-SNAPSHOT]
> 2016-06-20 19:10:31,137 [main] INFO  o.apache.drill.exec.server.Drillbit -
> Shutdown completed (1914 ms).
>
> I did nothing and start it at next day, then it can startup.
>
> 2016-06-21 9:48 GMT+08:00 qiang li <tiredqiang@gmail.com>:
>
>> Hi Aman,
>>
>> I did not fully test with the old version.
>>
>> Cloud you please help me create the JIRA issue,  I think my account have
>> not the privilege, my account is griffinli and can not find the place to
>> create new issue. Below is the explain detail for the same SQL in different
>> nodes of cluster.
>>
>>
>> This is the correct plan which only have two nodes:
>> 0: jdbc:drill:zk=xxx:> explain plan for select
>> CONVERT_FROM(BYTE_SUBSTR(`ref0`.row_key,-8,8),'BIGINT_BE') as uid,
>> convert_from(`ref0`.`v`.`v`,'UTF8') as v  from hbase.`offers_nation_idx` as
>> `nation` join hbase.offers_ref0 as `ref0` on
>>  BYTE_SUBSTR(`ref0`.row_key,-8,8) = nation.`v`.`v` where `nation`.row_key
>>  > '0br' and `nation`.row_key  < '0bs' limit 10;
>> +------+------+
>> | text | json |
>> +------+------+
>> | 00-00    Screen
>> 00-01      Project(uid=[$0], v=[$1])
>> 00-02        SelectionVectorRemover
>> 00-03          Limit(fetch=[10])
>> 00-04            UnionExchange
>> 01-01              SelectionVectorRemover
>> 01-02                Limit(fetch=[10])
>> 01-03                  Project(uid=[CONVERT_FROMBIGINT_BE(BYTE_SUBSTR($3,
>> -8, 8))], v=[CONVERT_FROMUTF8(ITEM($4, 'v'))])
>> 01-04                    Project(row_key=[$3], v=[$4], ITEM=[$5],
>> row_key0=[$0], v0=[$1], $f2=[$2])
>> 01-05                      HashJoin(condition=[=($2, $5)],
>> joinType=[inner])
>> 01-07                        Project(row_key=[$0], v=[$1],
>> $f2=[BYTE_SUBSTR($0, -8, 8)])
>> 01-09                          Scan(groupscan=[HBaseGroupScan
>> [HBaseScanSpec=HBaseScanSpec [tableName=offers_ref0, startRow=null,
>> stopRow=null, filter=null], columns=[`*`]]])
>> 01-06                        Project(row_key0=[$0], v0=[$1], ITEM=[$2])
>> 01-08                          *BroadcastExchange*
>> 02-01                            Project(row_key=[$0], v=[$1],
>> ITEM=[ITEM($1, 'v')])
>> 02-02                              Scan(groupscan=[HBaseGroupScan
>> [HBaseScanSpec=HBaseScanSpec [tableName=offers_nation_idx,
>> startRow=0br\x00, stopRow=0bs, filter=FilterList AND (2/2): [RowFilter
>> (GREATER, 0br), RowFilter (LESS, 0bs)]], columns=[`row_key`, `v`,
>> `v`.`v`]]])
>>
>>
>> This is the plan that fails which have more than 5 nodes:
>> 0: jdbc:drill:zk=xxx:> explain plan for select
>> CONVERT_FROM(BYTE_SUBSTR(`ref0`.row_key,-8,8),'BIGINT_BE') as uid,
>> convert_from(`ref0`.`v`.`v`,'UTF8') as v  from hbase.`offers_nation_idx` as
>> `nation` join hbase.offers_ref0 as `ref0` on
>>  BYTE_SUBSTR(`ref0`.row_key,-8,8) = nation.`v`.`v` where `nation`.row_key
>>  > '0br' and `nation`.row_key  < '0bs' limit 10;
>> +------+------+
>> | text | json |
>> +------+------+
>> | 00-00    Screen
>> 00-01      Project(uid=[$0], v=[$1])
>> 00-02        SelectionVectorRemover
>> 00-03          Limit(fetch=[10])
>> 00-04            UnionExchange
>> 01-01              SelectionVectorRemover
>> 01-02                Limit(fetch=[10])
>> 01-03                  Project(uid=[CONVERT_FROMBIGINT_BE(BYTE_SUBSTR($3,
>> -8, 8))], v=[CONVERT_FROMUTF8(ITEM($4, 'v'))])
>> 01-04                    Project(row_key=[$3], v=[$4], ITEM=[$5],
>> row_key0=[$0], v0=[$1], $f2=[$2])
>> 01-05                      HashJoin(condition=[=($2, $5)],
>> joinType=[inner])
>> 01-07                        Project(row_key=[$0], v=[$1], $f2=[$2])
>> 01-09                          *HashToRandomExchange*(dist0=[[$2]])
>> 02-01                            UnorderedMuxExchange
>> 04-01                              Project(row_key=[$0], v=[$1],
>> $f2=[$2], E_X_P_R_H_A_S_H_F_I_E_L_D=[hash32AsDouble($2)])
>> 04-02                                Project(row_key=[$0], v=[$1],
>> $f2=[BYTE_SUBSTR($0, -8, 8)])
>> 04-03                                  Scan(groupscan=[HBaseGroupScan
>> [HBaseScanSpec=HBaseScanSpec [tableName=offers_ref0, startRow=null,
>> stopRow=null, filter=null], columns=[`*`]]])
>> 01-06                        Project(row_key0=[$0], v0=[$1], ITEM=[$2])
>> 01-08                          Project(row_key=[$0], v=[$1], ITEM=[$2])
>> 01-10                            *HashToRandomExchange*(dist0=[[$2]])
>> 03-01                              UnorderedMuxExchange
>> 05-01                                Project(row_key=[$0], v=[$1],
>> ITEM=[$2], E_X_P_R_H_A_S_H_F_I_E_L_D=[hash32AsDouble($2)])
>> 05-02                                  Project(row_key=[$0], v=[$1],
>> ITEM=[ITEM($1, 'v')])
>> 05-03                                    Scan(groupscan=[HBaseGroupScan
>> [HBaseScanSpec=HBaseScanSpec [tableName=offers_nation_idx,
>> startRow=0br\x00, stopRow=0bs, filter=FilterList AND (2/2): [RowFilter
>> (GREATER, 0br), RowFilter (LESS, 0bs)]], columns=[`row_key`, `v`,
>> `v`.`v`]]])
>>
>> The difference is use *BroadcastExchange *and  *HashToRandomExchange.*
>>
>> You can create the JIRA and send me the link .
>>
>> Thanks.
>>
>>
>> 2016-06-20 23:44 GMT+08:00 Aman Sinha <amansinha@apache.org>:
>>
>>> Hi Qiang,
>>> were you seeing this same issue with the prior HBase version also ?  (I
>>> would think this is not a regression).  It would be best to create a new
>>> JIRA and attach the EXPLAIN plans for the successful and failed runs.
>>> With
>>> more nodes some minor fragments of the hash join may be getting empty
>>> input
>>> batches and I am guessing that has something to do with the
>>> SchemaChangeException.   Someone would need to debug once you create the
>>> JIRA with relevant details.
>>>
>>> -Aman
>>>
>>> On Mon, Jun 20, 2016 at 5:13 AM, qiang li <tiredqiang@gmail.com> wrote:
>>>
>>> > Thanks Aditya.
>>> >
>>> > By the way, I found another issue.
>>> >
>>> > Let say I have two tables.
>>> >
>>> > offers_ref0 : rowkey salt(1byte)+long uid(8 byte ) , family: v,
>>> qualifier:
>>> > v(string)
>>> > offers_nation_idx: rowkey salt(1byte) + string, family:v, qualifier:
>>> v(long
>>> > 8 byte)
>>> >
>>> > there is the SQL:
>>> >
>>> > select CONVERT_FROM(BYTE_SUBSTR(`ref0`.row_key,-8,8),'BIGINT_BE') as
>>> uid,
>>> > convert_from(`ref0`.`v`.`v`,'UTF8') as v  from
>>> hbase.`offers_nation_idx` as
>>> > `nation` join hbase.offers_ref0 as `ref0` on
>>> > CONVERT_FROM(BYTE_SUBSTR(`ref0`.row_key,-8,8),'BIGINT_BE') =
>>> > CONVERT_FROM(nation.`v`.`v`,'BIGINT_BE') where `nation`.row_key  >
>>> '0br'
>>> > and `nation`.row_key  < '0bs' limit 10
>>> >
>>> > When I execute the query with single node or less than 5 nodes, its
>>> working
>>> > good. But when I execute it in cluster which have about 14 nodes, its
>>> throw
>>> > a exception:
>>> >
>>> > First time will throw this exception:
>>> > *Caused by: java.sql.SQLException: SYSTEM ERROR: SchemaChangeException:
>>> > Hash join does not support schema changes*
>>> >
>>> > Then if I query again, it will always throw below exception:
>>> > *Query Failed: An Error Occurred*
>>> > *org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR:
>>> > IllegalStateException: Failure while reading vector. Expected vector
>>> class
>>> > of org.apache.drill.exec.vector.NullableIntVector but was holding
>>> vector
>>> > class org.apache.drill.exec.vector.complex.MapVector, field=
>>> > v(MAP:REQUIRED)[v(VARBINARY:OPTIONAL)[$bits$(UINT1:REQUIRED),
>>> > v(VARBINARY:OPTIONAL)[$offsets$(UINT4:REQUIRED)]]] Fragment 12:4
>>> [Error Id:
>>> > 06c6eae4-0822-4714-b0bf-a6e04ebfec79 on xxx:31010]*
>>> >
>>> > Its very strange, and I do not know how to solve it.
>>> > I tried add node to the cluster one by one, it will reproduce when I
>>> added
>>> > 5 nodes. Can anyone help me solve this issue?
>>> >
>>> >
>>> >
>>> >
>>> > 2016-06-17 4:39 GMT+08:00 Aditya <adityakishore@gmail.com>:
>>> >
>>> > > https://issues.apache.org/jira/browse/DRILL-4727
>>> > >
>>> > > On Thu, Jun 16, 2016 at 11:39 AM, Aman Sinha <amansinha@apache.org>
>>> > wrote:
>>> > >
>>> > >> Qiang/Aditya can you create a JIRA for this and mark it for 1.7.
>>> > thanks.
>>> > >>
>>> > >> On Thu, Jun 16, 2016 at 11:25 AM, Aditya <adityakishore@gmail.com>
>>> > wrote:
>>> > >>
>>> > >> > Thanks for reporting, I'm looking into it and will post a
patch
>>> soon.
>>> > >> >
>>> > >> > On Wed, Jun 15, 2016 at 7:27 PM, qiang li <tiredqiang@gmail.com>
>>> > wrote:
>>> > >> >
>>> > >> > > Hi Aditya,
>>> > >> > >
>>> > >> > > I tested the latest version and got this exception and
the
>>> drillbit
>>> > >> fail
>>> > >> > > to startup .
>>> > >> > >
>>> > >> > > Exception in thread "main" java.lang.NoSuchMethodError:
>>> > >> > > io.netty.util.UniqueName.<init>(Ljava/lang/String;)V
>>> > >> > >         at
>>> > >> io.netty.channel.ChannelOption.<init>(ChannelOption.java:136)
>>> > >> > >         at
>>> > >> io.netty.channel.ChannelOption.valueOf(ChannelOption.java:99)
>>> > >> > >         at
>>> > >> io.netty.channel.ChannelOption.<clinit>(ChannelOption.java:42)
>>> > >> > >         at
>>> > >> > >
>>> org.apache.drill.exec.rpc.BasicServer.<init>(BasicServer.java:63)
>>> > >> > >         at
>>> > >> > >
>>> org.apache.drill.exec.rpc.user.UserServer.<init>(UserServer.java:74)
>>> > >> > >         at
>>> > >> > >
>>> > >>
>>> >
>>> org.apache.drill.exec.service.ServiceEngine.<init>(ServiceEngine.java:78)
>>> > >> > >         at
>>> > >> > org.apache.drill.exec.server.Drillbit.<init>(Drillbit.java:108)
>>> > >> > >         at
>>> > >> org.apache.drill.exec.server.Drillbit.start(Drillbit.java:285)
>>> > >> > >         at
>>> > >> org.apache.drill.exec.server.Drillbit.start(Drillbit.java:271)
>>> > >> > >         at
>>> > >> org.apache.drill.exec.server.Drillbit.main(Drillbit.java:267)
>>> > >> > >
>>> > >> > > It will working if I remove
>>> > jars/3rdparty/netty-all-4.0.23.Final.jar,
>>> > >> the
>>> > >> > > drill can startup. I think there have some package dependency
>>> > version
>>> > >> > > issue, do you think so ?
>>> > >> > >
>>> > >> > >
>>> > >> > >
>>> > >> > > 2016-06-15 8:14 GMT+08:00 Aditya <adityakishore@gmail.com>:
>>> > >> > >
>>> > >> > >> HBase 1.x support has been merged and is available
in latest
>>> > >> > >> 1.7.0-SNAPSHOT
>>> > >> > >> builds.
>>> > >> > >>
>>> > >> > >> On Wed, Jun 1, 2016 at 1:23 PM, Aditya <
>>> adityakishore@gmail.com>
>>> > >> wrote:
>>> > >> > >>
>>> > >> > >> > Thanks Jacques for promptly reviewing my long
series of
>>> patches!
>>> > >> > >> >
>>> > >> > >> > I'm planning to merge the HBase 1.x support
some time in
>>> next 48
>>> > >> > hours.
>>> > >> > >> >
>>> > >> > >> > If anyone else is interested and willing, please
review the
>>> > latest
>>> > >> > patch
>>> > >> > >> > here[1].
>>> > >> > >> >
>>> > >> > >> > aditya...
>>> > >> > >> >
>>> > >> > >> > [1] https://github.com/apache/drill/pull/443/files
>>> > >> > >> >
>>> > >> > >>
>>> > >> > >
>>> > >> > >
>>> > >> >
>>> > >>
>>> > >
>>> > >
>>> >
>>>
>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message