Mailing-List: contact dev-help@kylin.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@kylin.incubator.apache.org
MIME-Version: 1.0
In-Reply-To: <D28375F9.10849%shaoshi@ebay.com>
References: <tencent_59E9478D036030600D6258E2@qq.com>
 <565D4A46.3060600@jd.com>
 <tencent_54F17B89619A0C5E45BD1AC6@qq.com> <565D4D05.6060607@jd.com>
 <D28375F9.10849%shaoshi@ebay.com>
From: hongbin ma <mahongbin@apache.org>
Date: Tue, 1 Dec 2015 16:12:43 +0800
Message-ID: 
 <CAF7etTk8CA5qsLZ5XEXydcZDRTb1pcoq09wdBZZQykTGEnY=ag@mail.gmail.com>
Subject: Re: How to support Avro Complex Type on Kylin
To: dev <dev@kylin.incubator.apache.org>
Content-Type: multipart/alternative; boundary=001a113f8fca0e699f0525d1bafa

--001a113f8fca0e699f0525d1bafa
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

=E2=80=8B@shaofeng, it looks like a nice feature.=E2=80=8B

On Tue, Dec 1, 2015 at 4:07 PM, Shi, Shaofeng <shaoshi@ebay.com> wrote:

> Kylin should automatically skip these complex columns, instead of blockin=
g
> user from import the table, what do you think?
>
> On 12/1/15, 3:32 PM, "Xiaoyu Wang" <wangxiaoyu1@jd.com> wrote:
>
> >Yes You can create a hive view to remove the datatype array,map column.
> >
> >=E5=9C=A8 2015=E5=B9=B412=E6=9C=8801=E6=97=A5 15:26, Yiming Liu =E5=86=
=99=E9=81=93:
> >> Thanks Xiaoyu, for the quick response.
> >>
> >>
> >> Currently, there is no way to remove those fields. The error happens o=
n
> >>the first step "Sync Hive tables" when designing cube.
> >>
> >>
> >> I will redesign my original tables to fit the datatype requirement.
> >>
> >>
> >> ------------------ Original ------------------
> >> From:  "Xiaoyu Wang";<wangxiaoyu1@jd.com>;
> >> Date:  Tue, Dec 1, 2015 03:20 PM
> >> To:  "dev"<dev@kylin.incubator.apache.org>;
> >>
> >> Subject:  Re: How to support Avro Complex Type on Kylin
> >>
> >>
> >>
> >> Kylin does not support datatype like "array" "map".
> >> Can't set the array,map datatype column as dimension.
> >> You can remove the array,map column from cube design, and retry .
> >>
> >> =E5=9C=A8 2015=E5=B9=B412=E6=9C=8801=E6=97=A5 15:05, Yiming Liu =E5=86=
=99=E9=81=93:
> >>> Hi Kylin expert,
> >>>
> >>> I have a table with avro encoding. It has map, array field type. I
> >>>could query the table on Hive.
> >>>
> >>> When I sync the table into Kylin, the Kylin says:
> >>> "bad data type -- array&lt;string&gt;, does not match
> >>>(any|char|varchar|boolean|binary|integer|tinyint|smallint|bigint|decim=
al
> >>>|numeric|float|real|double|date|time|datetime|timestamp|byte|int|short=
|l
> >>>ong|string|hllc|_literal_type|_derived_type)\s*(?:[(]([\d\s,]+)[)])?"
> >>>
> >>> So it seems Kylin does not support the avro complex type, is it right=
?
> >>>Do you have any suggestion how to process the complex data type.
> >>>
> >>> SerDe Library:      org.apache.hadoop.hive.serde2.avro.AvroSerDe
> >>>
> >>>InputFormat: org.apache.hadoop.hive.ql.io.avro.AvroContainerInputForma=
t
> >>>
> >>>OutputFormat:
> org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputForma
> >>>t
> >>>
> >>> Following is my table schema:
> >>> 0           sessionid       string
> >>> 1           userid  string
> >>> 2           hosts   array<string>
> >>> 3           domain  string
> >>> 4           visittimes      int
> >>> 5           firsttimestamp  bigint
> >>> 6           lasttimestamp   bigint
> >>> 7           sessiontimestamp        bigint
> >>> 8           useragent       map<string,string>
> >>> 9           srcaddrunsignedint      bigint
> >>> 10          srcaddrstr      string
> >>> 11          srcaddrcity     map<string,string>
> >>> 12          srcaddrlocation map<string,string>
> >>> 13          destaddrunsignedint     bigint
> >>> 14          destaddrstr     string
> >>> 15          destaddrcity    map<string,string>
> >>> 16          destaddrlocation        map<string,string>
> >>> 17          keywords        map<string,array<string>>
> >>> 18          topics  map<string,double>
> >>> 19          cookies map<string,string>
> >>> 20          urls    array<string>
> >>> 21          year    int
> >>> 22          month   int
> >>> 23          day     int
> >>> 24          hour    int
>
>


--=20
Regards,

*Bin Mahone | =E9=A9=AC=E6=B4=AA=E5=AE=BE*
Apache Kylin: http://kylin.io
Github: https://github.com/binmahone

--001a113f8fca0e699f0525d1bafa--