From user-return-3008-archive-asf-public=cust-asf.ponee.io@kylin.apache.org Sun Feb 18 03:23:26 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id D4D3B180657 for ; Sun, 18 Feb 2018 03:23:24 +0100 (CET) Received: (qmail 23150 invoked by uid 500); 18 Feb 2018 02:23:23 -0000 Mailing-List: contact user-help@kylin.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@kylin.apache.org Delivered-To: mailing list user@kylin.apache.org Received: (qmail 23141 invoked by uid 99); 18 Feb 2018 02:23:23 -0000 Received: from mail-relay.apache.org (HELO mailrelay1-lw-us.apache.org) (207.244.88.152) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 18 Feb 2018 02:23:23 +0000 Received: from mail-pg0-f41.google.com (mail-pg0-f41.google.com [74.125.83.41]) by mailrelay1-lw-us.apache.org (ASF Mail Server at mailrelay1-lw-us.apache.org) with ESMTPSA id 430B8623 for ; Sun, 18 Feb 2018 02:23:21 +0000 (UTC) Received: by mail-pg0-f41.google.com with SMTP id f6so4571893pgs.10 for ; Sat, 17 Feb 2018 18:23:21 -0800 (PST) X-Gm-Message-State: APf1xPB0bnGQOefdoACFPQzmZKUUwFwMe9nwb1lp3tDoBv4/Cv7K5UkI emIB2SbvvIajWIQ37xGGLSxOpkNx35+B9negi8c= X-Google-Smtp-Source: AH8x226rxtrMYxyDG7RUvXmup/4x9nS1LOk7T8X6L+jvtTIl1DzavHMsD0/XoQAlSaGtyIMrATxi8MUapSgyFxW0E6c= X-Received: by 10.99.55.65 with SMTP id g1mr8902298pgn.284.1518920600353; Sat, 17 Feb 2018 18:23:20 -0800 (PST) MIME-Version: 1.0 Received: by 10.100.160.240 with HTTP; Sat, 17 Feb 2018 18:22:39 -0800 (PST) In-Reply-To: <8527708fb5a54531b067170446db344d@SNP02SEM02.bureau.si.interne> References: <8527708fb5a54531b067170446db344d@SNP02SEM02.bureau.si.interne> From: ShaoFeng Shi Date: Sun, 18 Feb 2018 10:22:39 +0800 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: usage of Web inteface Kylin an performances To: user Content-Type: multipart/alternative; boundary="94eb2c0bef54d383cf0565734239" --94eb2c0bef54d383cf0565734239 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi Jean-luc, Most of the Kylin developers are in the new year holiday, so there might be some delay. Here are some comments from my side: 1. I presume that the whole .json files are stored, is it right ? yes 2. Do these kinds of tables contain the cube data ? yes; cube are stored in HBase with "KYLINL_" as prefix 3. So I am wondering if it is the good method the "compression" in Tomcat/conf/server.xml has nothing with cube build. To enable compression for cube, you need to configure that in your Hadoop configurations like mapred-site.xml, hbase-site.xml or kylin/conf/kylin_job_conf.xml. 4. How is it possible to optimize cube size to keep good performance ? https://kylin.apache.org/docs21/howto/howto_optimize_cubes.html 5. Is it through the =E2=80=98rowkeys=E2=80=99 in the advanced settings wh= en you build the cube ? yes, exactly; putting the most used filtering column to the heading position on the rowkey can get better performance. 6. What shall we put exactly in the =E2=80=98Rowkeys=E2=80=99 section ? All dimensions (excluding 'derived' dimensions) need be on rowkey; If you see too many columns in the agg. group, remove some dimensions from your cube. 7. Are the aggregation groups used for speed of the queries. The agg. group is used to optimize the dimension combinations. For a N dimension cube, by default it will have 2^N combinations (we called cuboid). If you can divide N dimensions to several groups, the combination numbers can be greatly reduced, so the cube build will be much easier and taking much less space. How to define the agg. group? You can do that with your business query patterns. 2018-02-14 1:49 GMT+08:00 BELLIER Jean-luc : > Hello, > > > > I have several questions on Kylin, especially about performances and how > to manage them. I would like to understand precisely how it works to see = if > I can use it in my business context. > > > > I come from the relational database world, so as far as I understand on > OLAP, the searches are performed on the values of primary keys in > dimensions. These subsets are then joined to get the corresponding lines = on > the facts table. As the dimensions tables are much smaller than the facts > table, the queries run faster > > > > *1. **Questions on performances* > > =C2=B7 the raw data are stored in Hive, and the models and struct= ures > (cubes) are stored in HBase; I presume that the whole .json files are > stored, is it right ? > > =C2=B7 Where are the cube results stores (I mean after a build, a > refresh or an append action). Is it also in HBase ? I can see in HBase > tables like "KYLIN_FF46WDAAGH". Do these kinds of tables contain the cube > data ? > > =C2=B7 I noticed that when I build the =E2=80=98sample_cube=E2=80= =99, the volume of > data was very important compared to the size of the original files. Is > there a way to reduce it (I saw a attribute in the $KYLIN_HOME/tomcat/con= f/server.xml > file, called =E2=80=98compression=E2=80=99 for the connector on port 7070= , but I do not > know if it is related to the cube size). I tried to change this parameter > to =E2=80=98yes=E2=80=99, but I noticed a huge increase of the duration o= f cube generation. > So I am wondering if it is the good method. > > =C2=B7 How is it possible to optimize cube size to keep good > performance ? > > =C2=B7 In Hive, putting indexes is not recommended. So how Kylin = is > ensuring good performance when querying high volumes of data ? Is it > through the =E2=80=98rowkeys=E2=80=99 in the advanced settings when you b= uild the cube ? > > Or is the answer elsewhere ? > > > > *2. **Questions on cube building* > > =C2=B7 By the way, the =E2=80=98Advanced settings=E2=80=99 step i= s still unclear for > me. I tried to build a cube from scratch using the tables provided in the > sample project. But I do not know very much what to put in this section. > > =C2=B7 My goal is to define groups of data on YEAR_BEG_DT, > QTR_BEG_DT,MONTH_BEG_DT. > > =C2=B7 I do not understand very well why the aggregation group > contains so many columns. I tried to remove as many as possible, but when= I > tried to set up the joins, but some fields were missing so the saving of > the cube failed. > > =C2=B7 What shall we put exactly in the =E2=80=98Rowkeys=E2=80=99= section ? I > understand that this is used to define data encoding (for speed access ? > ).Am I right ? > > =C2=B7 Are the aggregation groups used for speed of the queries. = I > assume it is the case, because it represents the most commonly used > associations of columns for the cube. > > > > Thank you in advance for your help. > > > > Best regards, > > Jean-Luc. > > > > > > > > > "Ce message est destin=C3=A9 exclusivement aux personnes ou entit=C3=A9s = auxquelles > il est adress=C3=A9 et peut contenir des informations privil=C3=A9gi=C3= =A9es ou > confidentielles. Si vous avez re=C3=A7u ce document par erreur, merci de = nous > l'indiquer par retour, de ne pas le transmettre et de proc=C3=A9der =C3= =A0 sa > destruction. > > This message is solely intended for the use of the individual or entity t= o > which it is addressed and may contain information that is privileged or > confidential. If you have received this communication by error, please > notify us immediately by electronic mail, do not disclose it and delete t= he > original message." > --=20 Best regards, Shaofeng Shi =E5=8F=B2=E5=B0=91=E9=94=8B --94eb2c0bef54d383cf0565734239 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi Jean-luc,

Most of the Kylin develope= rs are in the new year holiday, so there might be some delay. Here are some= comments from my side:

1.=C2=A0=C2=A0I presume that the whole .json files are = stored, is it right ?
yes=
2.=C2=A0Do these kinds of tables contain the cube data ?
yes; cube are stored in HBa= se with "KYLINL_" as prefix
3. So I am wondering if it is= the good method
t= he "compression" in Tomcat/conf/server.xml=C2=A0has nothing with = cube build. To enable compression for cube, you need to configure that in y= our Hadoop configurations like mapred-site.xml, hbase-site.xml or kylin/con= f/kylin_job_conf.xml.
4.= =C2=A0How is it possible to optimize cube = size to keep good performance ?=C2=A0
5.=C2=A0=C2=A0Is it through the =E2=80= =98rowkeys=E2=80=99 in the advanced settings when you build the cube ?
yes, exactly; puttin= g the most used filtering column to the heading position on the rowkey can = get better performance.
6.=C2=A0What shall we put exactly in the =E2=80=98Rowke= ys=E2=80=99 section ?
All dime= nsions (excluding 'derived' dimensions) need be on rowkey; If you s= ee too many columns in the agg. group, remove some dimensions from your cub= e.
7.=C2=A0=C2=A0Are the aggregation groups used for speed of the queries.=C2=A0
The agg. group is used to optimize the dimension combination= s. For a N dimension cube, by default it will have 2^N combinations (we cal= led cuboid). If you can divide N dimensions to several groups, the combinat= ion numbers can be greatly reduced, so the cube build will be much easier a= nd taking much less space. How to define the agg. group? You can do that wi= th your business query patterns.=C2=A0
=C2=A0


2018-02-14 1:49 GMT+08:00 BELLIER Jean-luc <= ;jean-= luc.bellier@rte-france.com>:

Hello,

=C2=A0

I have = several questions on Kylin, especially about performances and how to manage= them. I would like to understand precisely how it works to see if I can us= e it in my business context.

= =C2=A0

I come = from the relational database world, so as far as I understand on OLAP, the = searches are performed on the values of primary keys in dimensions. These s= ubsets are then joined to get the corresponding lines on the facts table. As the dimensions tables are much smaller than t= he facts table, the queries run faster

= =C2=A0

1.=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Questions on performances

=C2=B7=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 th= e raw data are stored in Hive, and the models and structures (cubes) are st= ored in HBase; I presume that the whole .json files are stored, is it right= ?

=C2=B7=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 Wh= ere are the cube results stores (I mean after a build, a refresh or an appe= nd action). Is it also in HBase ? I can see in HBase tables like "KYLI= N_FF46WDAAGH". Do these kinds of tables contain the cube data ?

=C2=B7=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 I = noticed that when I build the =E2=80=98sample_cube=E2=80=99, the volume of = data was very important compared to the size of the original files. Is ther= e a way to reduce it (I saw a attribute in the $KYLIN_HOME/tomcat/conf/server.xml file, called =E2=80=98compression=E2=80=99 for the connector on port 7070,= but I do not know if it is related to the cube size). I tried to change th= is parameter to =E2=80=98yes=E2=80=99, but I noticed a huge increase of the= duration of cube generation. So I am wondering if it is the good method.

=C2=B7=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 Ho= w is it possible to optimize cube size to keep good performance ? =C2=A0=

=C2=B7=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 In= Hive, putting indexes is not recommended. So how Kylin is ensuring good pe= rformance when querying high volumes of data=C2=A0 ? Is it through the =E2= =80=98rowkeys=E2=80=99 in the advanced settings when you build the cube ?

Or is t= he answer elsewhere ?

= =C2=A0

2.=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 Questions on cube building

=C2=B7=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 By= the way, the =E2=80=98Advanced settings=E2=80=99 step is still unclear for= me. I tried to build a cube from scratch using the tables provided in the = sample project. But I do not know very much what to put in this section.

=C2=B7=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 My= goal is to define groups of data on YEAR_BEG_DT, QTR_BEG_DT,MONTH_BEG_DT.

=C2=B7=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 I = do not understand very well why the aggregation group contains so many colu= mns. I tried to remove as many as possible, but when I tried to set up the = joins, but some fields were missing so the saving of the cube failed.

=C2=B7=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 Wh= at shall we put exactly in the =E2=80=98Rowkeys=E2=80=99 section ? I unders= tand that this is used to define data encoding (for speed access ? ).Am I r= ight ?

=C2=B7=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 Ar= e the aggregation groups used for speed of the queries. I assume it is the = case, because it represents the most commonly used associations of columns = for the cube.

= =C2=A0

Thank y= ou in advance for your help.

= =C2=A0

Best re= gards,

Jean-Lu= c.

=C2=A0

= =C2=A0

= =C2=A0



"Ce message est dest= in=C3=A9 exclusivement aux personnes ou entit=C3=A9s auxquelles il est adre= ss=C3=A9 et peut contenir des informations privil=C3=A9gi=C3=A9es ou confid= entielles. Si vous avez re=C3=A7u ce document par erreur, merci de nous l&#= 39;indiquer par retour, de ne pas le transmettre et de proc=C3=A9der =C3=A0 sa destruc= tion.

This message is solely intended for the use of the individual or entity to = which it is addressed and may contain information that is privileged or con= fidential. If you have received this communication by error, please notify = us immediately by electronic mail, do not disclose it and delete the original message."



--
Best regards,

Shaofeng Shi =E5=8F=B2= =E5=B0=91=E9=94=8B

--94eb2c0bef54d383cf0565734239--