kylin-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <roberto.tar...@stratebi.com>
Subject RE: Segment overlap issues when refreshing a cube segment
Date Tue, 28 Aug 2018 08:36:11 GMT
Many thanks for the advice. Do you plan to release version 2.4.1 in September?

 

 

From: ShaoFeng Shi [mailto:shaofengshi@apache.org] 
Sent: viernes, 24 de agosto de 2018 5:01
To: user <user@kylin.apache.org>
Subject: Re: Segment overlap issues when refreshing a cube segment

 

Kylin 2.4.1 will be kicked off soon, which includes several bugfixes in 2.4.0. You can directly
upgrade to 2.4.1.

 

2018-08-23 19:07 GMT+08:00 <roberto.tardio@stratebi.com <mailto:roberto.tardio@stratebi.com>
>:

Shaofeng, thank you very much for the help. I will share if version 2.4 solves our segment
overlap issues.

Roberto Tardío Olmos

Head of Big Data Analytics

Avenida de Brasil, 17 <https://maps.google.com/?q=Avenida+de+Brasil,+17&entry=gmail&source=g>
, Planta 16.28020 Madrid

Fijo: 91.788.34.10




 

 <http://bigdata.stratebi.com/> http://bigdata.stratebi.com/ 

 

 <http://www.stratebi.com/> http://www.stratebi.com 

 

From: ShaoFeng Shi [mailto:shaofengshi@apache.org <mailto:shaofengshi@apache.org> ]

Sent: miércoles, 22 de agosto de 2018 15:21


To: user <user@kylin.apache.org <mailto:user@kylin.apache.org> >
Subject: Re: Segment overlap issues when refreshing a cube segment

 

Yes, KYLIN-3311 introduces a lock mechanism when updating metadata, that can avoid concurrent
updates on the same entity, so to avoid two duplicated segments be inserted. 

 

2018-08-22 19:52 GMT+08:00 <roberto.tardio@stratebi.com <mailto:roberto.tardio@stratebi.com>
>:

Many thanks ShaoFeng!

 

Yes, I could see the new button because we also have a test installation of Kylin 2.4 and
we are thinking about to update kylin of the production environment from 2.2 to 2.4.

 

As for the API, I was aware of its existence. However, in the case I'm telling you we can't
delete it because the segment is in the middle of other segments and in NEW state, duplicated.
That's why I was talking about the JIRA advocacy https://issues.apache.org/jira/browse/KYLIN-3311
 .  Do you think that the error causing the segment overlap is related to this JIRA?

 

Kind Regards,

 

Roberto

 

From: ShaoFeng Shi [mailto:shaofengshi@apache.org <mailto:shaofengshi@apache.org> ]

Sent: martes, 21 de agosto de 2018 14:54
To: user <user@kylin.apache.org <mailto:user@kylin.apache.org> >
Subject: Re: Segment overlap issues when refreshing a cube segment

 

Hi Roberto, in the latest Kylin (I'm running), on the web GUI, there is a "Delete segment"
button, which allows you to delete a segment easily. 

 

You can try to use the Rest API to delete a segment, this API should have existed for a long
time: https://kylin.apache.org/docs/howto/howto_use_restapi.html#delete-segment

 

I

 

2018-08-21 6:15 GMT+08:00 <roberto.tardio@stratebi.com <mailto:roberto.tardio@stratebi.com>
>:

Hi, 

 

I am using Kylin 2.2, where I have several OLAP cubes in which I am constantly adding and
refreshing segments. I am faced with a problem that has generated two segments for the same
period of time and therefore an error of the "segments overlap" type. Kylin is also cluster
mounted, with one machine for cube building and one for querying. Here is the log of the query
machine that seems to reflect the error (cube segment building process was successfully completed):

 

2018-08-16 09:19:35,833 INFO  [http-bio-7070-exec-93] cube.CubeManager:799 : Reloaded cube
trafico_cubo_v8 being CUBE[name=trafico_cubo_v8] having 20 segments

2018-08-16 09:19:35,833 DEBUG [http-bio-7070-exec-93] cachesync.Broadcaster:256 : BroadcastingUPDATE,
project_data, My_Cube_Trafico

2018-08-16 09:19:35,834 INFO  [http-bio-7070-exec-93] service.CacheService:143 : removeOLAPDataSource
is called for project My_Cube_Trafico

2018-08-16 09:19:35,834 INFO  [http-bio-7070-exec-93] service.CacheService:125 : cleaning
cache for project My_Cube_Trafico (currently remove all entries)

2018-08-16 09:19:35,834 DEBUG [http-bio-7070-exec-93] cachesync.Broadcaster:290 : Done broadcastingUPDATE,
project_data, My_Cube_Trafico

2018-08-16 09:19:35,834 DEBUG [http-bio-7070-exec-93] cachesync.Broadcaster:290 : Done broadcastingUPDATE,
cube, trafico_cubo_v8

2018-08-16 09:19:36,817 INFO  [http-bio-7070-exec-104] cube.CubeManager:358 : Updating cube
instance 'plataforma_cubo_mes_v2'

2018-08-16 09:19:36,817 ERROR [http-bio-7070-exec-104] controller.CubeController:337 : Segments
overlap: plataforma_cubo_mes_v2[20180801000000_20180901000000] and plataforma_cubo_mes_v2[20180801000000_20180901000000]

2018-08-16 09:19:36,818 ERROR [http-bio-7070-exec-104] controller.BasicController:57 : 

2018-08-16 09:19:37,264 DEBUG [http-bio-7070-exec-93] cachesync.Broadcaster:256 : BroadcastingUPDATE,
cube, trafico_cubo_v8

2018-08-16 09:19:37,265 INFO  [http-bio-7070-exec-93] cube.CubeManager:799 : Reloaded cube
trafico_cubo_v8 being CUBE[name=trafico_cubo_v8] having 20 segments

 

Could this error be related to the JIRA entry https://issues.apache.org/jira/browse/KYLIN-3311?
In that case, I see that it has been fixed in version 2.4 of Kylin.

 

Also, to fix the error, we had to stop Kylin, remove the duplicate segment from the metadata,
restart Kylin and then refresh the segment. As far as I can read on the website https://issues.apache.org/jira/browse/KYLIN-2849
, from version 2.3 onwards you can delete a segment in any position from the UI or the Kylin
API after disabling the cube. Is that correct? This would help us mitigate the problem more
quickly.

 

Thanks in advance!

Roberto Tardío Olmos

Head of Big Data Analytics

Avenida de Brasil, 17 <https://maps.google.com/?q=Avenida+de+Brasil,+17&entry=gmail&source=g>
, Planta 16.28020 Madrid

Fijo: 91.788.34.10




 

http://bigdata.stratebi.com/ 

 

http://www.stratebi.com <http://www.stratebi.com/>  

 





 

-- 

Best regards,

 

Shaofeng Shi 史少锋

 





 

-- 

Best regards,

 

Shaofeng Shi 史少锋

 





 

-- 

Best regards,

 

Shaofeng Shi 史少锋

 


Mime
View raw message