kudu-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Todd Lipcon <t...@cloudera.com>
Subject Re: Error when creating partitions with BIGINT
Date Mon, 16 Jan 2017 23:05:00 GMT
Hi Amit,

AFAIK that's a known issue in the latest impala-kudu parcel, which will be
addressed in the upcoming 5.10 release (at which point the special
"impala-kudu" build will no longer be necessary). It should have no serious
effect on functionality.

-Todd

On Mon, Jan 16, 2017 at 4:27 AM, Amit Adhau <amit.adhau@globant.com> wrote:

> Thanks Todd,
>
> We have installed the latest parcel as suggested and now, we do not have
> this issue.
> However, on Impala side it is showing us error as per below screen shot;
>
> [image: Inline image 1]
>
> This error in the dashboard is consistently coming up even after repeated
> Impala re-start.
> Our query seems to be working fine but, there is not much on the Impala
> log except the below mentioned observation.
>
> This seems to be specific to latest Impala_Kudu parcel because, if we goes
> ahead and use the old parcel '2.7.0-1.cdh5.9.0.p0.23' then we do not get
> this error.
>
> In logs, it shows us below message repeatedly.
>
> Query id 0:0 not found.
>     @           0x835ba9  (unknown)
>     @           0xaa0c45  (unknown)
>     @           0xace1ec  (unknown)
>     @           0xbd5955  (unknown)
>     @           0xbd6d75  (unknown)
>     @           0xbea330  (unknown)
>     @           0xbecaad  (unknown)
>     @           0xbed13d  (unknown)
>     @     0x7ff1ac47daa1  start_thread
>     @     0x7ff1ac1caaad  clone
>
> Can you please help us on this, if you have any idea about this issue or
> any impact of this error on the functionality.
>
> Thank you so much for your help.
>
> Thanks,
> Amit
>
> On Tue, Jan 3, 2017 at 8:14 AM, Todd Lipcon <todd@cloudera.com> wrote:
>
>> Hi Pablo,
>>
>> Sorry for the delay. I think many list members (myself included) have
>> been on vacation the last couple weeks for the holidays.
>>
>> I checked this on the latest development versions of Impala and Kudu
>> (corresponding to upcoming releases in January) and with the new syntax as
>> follows it seems to be working correctly:
>>
>> create table todd_test (
>>   a bigint,
>>   primary key (a)
>> ) partition by range(a) (
>>     partition values < 1480550400000,
>>     partition 1480550400000 <= values < 1483228800000,
>>     partition 1483228800000 <= values < 1485907200000,
>>     partition 1485907200000 <= values)
>> stored as kudu;
>>
>> I didn't get a chance to try the latest released version of Impala-Kudu,
>> but I'm guessing you might be on a slightly older one. Can you check the
>> installed Parcel version and see if it matches
>> IMPALA_KUDU-2.7.0-3.cdh5.9.0.p0.10? This version is available from
>> http://archive.cloudera.com/beta/impala-kudu/parcels/latest/
>>
>> -Todd
>>
>> On Tue, Dec 27, 2016 at 1:44 PM, Pablo Vazquez <pablo.vazquez@globant.com
>> > wrote:
>>
>>> Hi Kudu team.
>>>
>>> Just wondering if you have any comment related to the issue with the
>>> BIGINT partitions.
>>>
>>> Best Regards.
>>>
>>> On Fri, Dec 23, 2016 at 4:05 PM, Pablo Vazquez <
>>> pablo.vazquez@globant.com> wrote:
>>>
>>>> Hi Kudu fellowship!
>>>>
>>>> Happy holidays for all you guys.
>>>>
>>>> Recently I tried to create a simple 4-column table using a BIGINT as
>>>> part of the PK and use the same column for partitioning by RANGE.
>>>>
>>>> This is the script:
>>>>
>>>> *CREATE EXTERNAL TABLE `kd_clicks` (*
>>>> *`channel_id` BIGINT,*
>>>> *`timestamp` BIGINT,*
>>>> *`channel_name` STRING,*
>>>> *`cookie_hash` STRING*
>>>> *)*
>>>> *DISTRIBUTE BY RANGE (`timestamp`)*
>>>> *SPLIT ROWS ( *
>>>> *(1480550400000),*
>>>> *(1483228800000),*
>>>> *(1485907200000)*
>>>> *)*
>>>> *TBLPROPERTIES(*
>>>> *  'storage_handler' = 'com.cloudera.kudu.hive.KuduStorageHandler',*
>>>> *  'kudu.table_name' = 'kd_clicks',*
>>>> *  'kudu.master_addresses' = 'XXXXXX',*
>>>> *  'kudu.key_columns' = 'channel_id, timestamp'*
>>>> *)*
>>>> *;*
>>>>
>>>> However, in the Kudu UI the partitions are not created properly:
>>>>
>>>> Partition schema & range bounds
>>>>
>>>> Range component:
>>>>   (timestamp int64)
>>>> Range bounds:
>>>>   [<start>, (int64 timestamp=-1213317120))
>>>>   [(int64 timestamp=-1213317120), (int64 timestamp=-151484416))
>>>>   [(int64 timestamp=-151484416), (int64 timestamp=1465082880))
>>>>   [(int64 timestamp=1465082880), <end>)
>>>>
>>>>
>>>> We made some tests and we found the issue appears when the partition is
>>>> defined above 2,147,483,647 (upper limit for Int-32). It seems the range
>>>> partition only accept INT32 boundaries although the column is INT64.
>>>>
>>>> I'd appreciate your comments. We're not sure if we are missing
>>>> something during table creation.
>>>>
>>>> Regards.
>>>> --
>>>> *Pablo Quetzalcóatl Vázquez*| Software Designer
>>>> *GLOBANT*
>>>>
>>>> [image: Facebook]
>>>> <http://s.wisestamp.com/links?url=https%3A%2F%2Fwww.facebook.com%2FGlobant&sn=b3psb3BlekB5YWhvby5jb20%3D>
[image:
>>>> Twitter]
>>>> <http://s.wisestamp.com/links?url=http%3A%2F%2Fwww.twitter.com%2Fglobant&sn=b3psb3BlekB5YWhvby5jb20%3D>
[image:
>>>> Youtube]
>>>> <http://s.wisestamp.com/links?url=http%3A%2F%2Fwww.youtube.com%2FGlobant&sn=b3psb3BlekB5YWhvby5jb20%3D>
[image:
>>>> Linkedin]
>>>> <http://s.wisestamp.com/links?url=http%3A%2F%2Fwww.linkedin.com%2Fcompany%2Fglobant&sn=b3psb3BlekB5YWhvby5jb20%3D>
[image:
>>>> Pinterest]
>>>> <http://s.wisestamp.com/links?url=http%3A%2F%2Fpinterest.com%2Fglobant%2F&sn=b3psb3BlekB5YWhvby5jb20%3D>
[image:
>>>> Globant]
>>>> <http://s.wisestamp.com/links?url=http%3A%2F%2Fwww.globant.com%2F&sn=b3psb3BlekB5YWhvby5jb20%3D>
>>>>
>>>
>>>
>>>
>>> --
>>> *Pablo Quetzalcóatl Vázquez*| Software Designer
>>> *GLOBANT*
>>>
>>> [image: Facebook]
>>> <http://s.wisestamp.com/links?url=https%3A%2F%2Fwww.facebook.com%2FGlobant&sn=b3psb3BlekB5YWhvby5jb20%3D>
[image:
>>> Twitter]
>>> <http://s.wisestamp.com/links?url=http%3A%2F%2Fwww.twitter.com%2Fglobant&sn=b3psb3BlekB5YWhvby5jb20%3D>
[image:
>>> Youtube]
>>> <http://s.wisestamp.com/links?url=http%3A%2F%2Fwww.youtube.com%2FGlobant&sn=b3psb3BlekB5YWhvby5jb20%3D>
[image:
>>> Linkedin]
>>> <http://s.wisestamp.com/links?url=http%3A%2F%2Fwww.linkedin.com%2Fcompany%2Fglobant&sn=b3psb3BlekB5YWhvby5jb20%3D>
[image:
>>> Pinterest]
>>> <http://s.wisestamp.com/links?url=http%3A%2F%2Fpinterest.com%2Fglobant%2F&sn=b3psb3BlekB5YWhvby5jb20%3D>
[image:
>>> Globant]
>>> <http://s.wisestamp.com/links?url=http%3A%2F%2Fwww.globant.com%2F&sn=b3psb3BlekB5YWhvby5jb20%3D>
>>>
>>> The information contained in this e-mail may be confidential. It has
>>> been sent for the sole use of the intended recipient(s). If the reader of
>>> this message is not an intended recipient, you are hereby notified that any
>>> unauthorized review, use, disclosure, dissemination, distribution or
>>> copying of this communication, or any of its contents,
>>> is strictly prohibited. If you have received it by mistake please let
>>> us know by e-mail immediately and delete it from your system. Many
>>> thanks.
>>>
>>>
>>>
>>> La información contenida en este mensaje puede ser confidencial. Ha sido
>>> enviada para el uso exclusivo del destinatario(s) previsto. Si el lector de
>>> este mensaje no fuera el destinatario previsto, por el presente queda Ud.
>>> notificado que cualquier lectura, uso, publicación, diseminación,
>>> distribución o copiado de esta comunicación o su contenido está
>>> estrictamente prohibido. En caso de que Ud. hubiera recibido este mensaje
>>> por error le agradeceremos notificarnos por e-mail inmediatamente y
>>> eliminarlo de su sistema. Muchas gracias.
>>>
>>>
>>
>>
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>>
>
>
>
> --
> Thanks & Regards,
>
> *Amit Adhau* | Data Architect
>
> *GLOBANT* | IND:+91 9821518132 <+91%2098215%2018132>
>
> [image: Facebook] <https://www.facebook.com/Globant>
>
> [image: Twitter] <http://www.twitter.com/globant>
>
> [image: Youtube] <http://www.youtube.com/Globant>
>
> [image: Linkedin] <http://www.linkedin.com/company/globant>
>
> [image: Pinterest] <http://pinterest.com/globant/>
>
> [image: Globant] <http://www.globant.com/>
>
> The information contained in this e-mail may be confidential. It has been
> sent for the sole use of the intended recipient(s). If the reader of this
> message is not an intended recipient, you are hereby notified that any
> unauthorized review, use, disclosure, dissemination, distribution or
> copying of this communication, or any of its contents,
> is strictly prohibited. If you have received it by mistake please let us
> know by e-mail immediately and delete it from your system. Many thanks.
>
>
>
> La información contenida en este mensaje puede ser confidencial. Ha sido
> enviada para el uso exclusivo del destinatario(s) previsto. Si el lector de
> este mensaje no fuera el destinatario previsto, por el presente queda Ud.
> notificado que cualquier lectura, uso, publicación, diseminación,
> distribución o copiado de esta comunicación o su contenido está
> estrictamente prohibido. En caso de que Ud. hubiera recibido este mensaje
> por error le agradeceremos notificarnos por e-mail inmediatamente y
> eliminarlo de su sistema. Muchas gracias.
>
>


-- 
Todd Lipcon
Software Engineer, Cloudera

Mime
View raw message