Return-Path: X-Original-To: apmail-kylin-dev-archive@minotaur.apache.org Delivered-To: apmail-kylin-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 0C369182F7 for ; Tue, 1 Dec 2015 08:13:24 +0000 (UTC) Received: (qmail 66904 invoked by uid 500); 1 Dec 2015 08:13:23 -0000 Delivered-To: apmail-kylin-dev-archive@kylin.apache.org Received: (qmail 66853 invoked by uid 500); 1 Dec 2015 08:13:23 -0000 Mailing-List: contact dev-help@kylin.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@kylin.incubator.apache.org Delivered-To: mailing list dev@kylin.incubator.apache.org Received: (qmail 66833 invoked by uid 99); 1 Dec 2015 08:13:23 -0000 Received: from Unknown (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 01 Dec 2015 08:13:23 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 22F26C0FF5 for ; Tue, 1 Dec 2015 08:13:23 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.126 X-Spam-Level: *** X-Spam-Status: No, score=3.126 tagged_above=-999 required=6.31 tests=[HK_RANDOM_ENVFROM=0.626, RDNS_NONE=2.5, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Received: from mx1-us-east.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id UMyeJiggNKcu for ; Tue, 1 Dec 2015 08:13:13 +0000 (UTC) Received: from smtp.jd.com (unknown [58.83.206.59]) by mx1-us-east.apache.org (ASF Mail Server at mx1-us-east.apache.org) with ESMTPS id 8444442BC2 for ; Tue, 1 Dec 2015 08:13:09 +0000 (UTC) Received: from [10.12.167.41] (118.144.128.52) by HUB03.360buyAD.local (172.17.27.17) with Microsoft SMTP Server (TLS) id 14.3.123.3; Tue, 1 Dec 2015 16:12:02 +0800 Subject: Re: How to support Avro Complex Type on Kylin To: References: <565D4A46.3060600@jd.com> <565D4D05.6060607@jd.com> From: Xiaoyu Wang Message-ID: <565D565C.3060205@jd.com> Date: Tue, 1 Dec 2015 16:12:12 +0800 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset="iso-2022-jp" Content-Transfer-Encoding: 7bit X-Originating-IP: [118.144.128.52] Yes agree,the jira:https://issues.apache.org/jira/browse/KYLIN-1111 I will try do it and submit patch. 在 2015年12月01日 16:07, Shi, Shaofeng 写道: > Kylin should automatically skip these complex columns, instead of blocking > user from import the table, what do you think? > > On 12/1/15, 3:32 PM, "Xiaoyu Wang" wrote: > >> Yes You can create a hive view to remove the datatype array,map column. >> >> 在 2015年12月01日 15:26, Yiming Liu 写道: >>> Thanks Xiaoyu, for the quick response. >>> >>> >>> Currently, there is no way to remove those fields. The error happens on >>> the first step "Sync Hive tables" when designing cube. >>> >>> >>> I will redesign my original tables to fit the datatype requirement. >>> >>> >>> ------------------ Original ------------------ >>> From: "Xiaoyu Wang";; >>> Date: Tue, Dec 1, 2015 03:20 PM >>> To: "dev"; >>> >>> Subject: Re: How to support Avro Complex Type on Kylin >>> >>> >>> >>> Kylin does not support datatype like "array" "map". >>> Can't set the array,map datatype column as dimension. >>> You can remove the array,map column from cube design, and retry . >>> >>> 在 2015年12月01日 15:05, Yiming Liu 写道: >>>> Hi Kylin expert, >>>> >>>> I have a table with avro encoding. It has map, array field type. I >>>> could query the table on Hive. >>>> >>>> When I sync the table into Kylin, the Kylin says: >>>> "bad data type -- array<string>, does not match >>>> (any|char|varchar|boolean|binary|integer|tinyint|smallint|bigint|decimal >>>> |numeric|float|real|double|date|time|datetime|timestamp|byte|int|short|l >>>> ong|string|hllc|_literal_type|_derived_type)\s*(?:[(]([\d\s,]+)[)])?" >>>> >>>> So it seems Kylin does not support the avro complex type, is it right? >>>> Do you have any suggestion how to process the complex data type. >>>> >>>> SerDe Library: org.apache.hadoop.hive.serde2.avro.AvroSerDe >>>> >>>> InputFormat: org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat >>>> >>>> OutputFormat: org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputForma >>>> t >>>> >>>> Following is my table schema: >>>> 0 sessionid string >>>> 1 userid string >>>> 2 hosts array >>>> 3 domain string >>>> 4 visittimes int >>>> 5 firsttimestamp bigint >>>> 6 lasttimestamp bigint >>>> 7 sessiontimestamp bigint >>>> 8 useragent map >>>> 9 srcaddrunsignedint bigint >>>> 10 srcaddrstr string >>>> 11 srcaddrcity map >>>> 12 srcaddrlocation map >>>> 13 destaddrunsignedint bigint >>>> 14 destaddrstr string >>>> 15 destaddrcity map >>>> 16 destaddrlocation map >>>> 17 keywords map> >>>> 18 topics map >>>> 19 cookies map >>>> 20 urls array >>>> 21 year int >>>> 22 month int >>>> 23 day int >>>> 24 hour int >