kylin-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sonny Heer <sonnyh...@gmail.com>
Subject Re: #3 Step Name: Extract Fact Table Distinct Columns (slow)
Date Wed, 14 Mar 2018 21:53:49 GMT
Maybe this is fixed in 2.0.0?  See
https://issues.apache.org/jira/browse/KYLIN-2135

Is this the same issue?  It should split dims to higher reducer it seems...


Although we are on 1.6.  If this will fix our issue - can i apply this
patch?  I may look at what is all there vs 1.6.  Any ideas?

On Wed, Mar 14, 2018 at 1:44 PM, Alberto Ramón <a.ramonportoles@gmail.com>
wrote:

> You can monitoring your yarn in step 3
> In any case, step 3 is a sample of Fat table to estimate number of keys
> for each dim
> If this step takes a lot of time, you will need review your cube design
>
> Alb
>
> On 14 March 2018 at 16:54, Sonny Heer <sonnyheer@gmail.com> wrote:
>
>> 8 YARN nodes with 11 slots each.  each slot is configured to ~2gb.  Step
>> #3 in Kylin is launching 19 mappers and 5 reducers.  5 reducers when there
>> are 88 slots.
>>
>> btw: kylin version is 1.6
>>
>> On Wed, Mar 14, 2018 at 9:48 AM, Sonny Heer <sonnyheer@gmail.com> wrote:
>>
>>> YARN is properly configured.  we use many other m/r and spark programs
>>> that utilize the full slots.  It's only when building cubes.
>>>
>>> On Wed, Mar 14, 2018 at 9:46 AM, Alberto Ramón <
>>> a.ramonportoles@gmail.com> wrote:
>>>
>>>> You need  check your yarn configuration first
>>>>
>>>> On Wed, 14 Mar 2018, 14:58 Sonny Heer, <sonnyheer@gmail.com> wrote:
>>>>
>>>>> Step 3 isn't using our full cluster.  How can i increase the
>>>>> mappers/reducers to use all the slots?  Any config to look at in kylin?
>>>>>
>>>>> Thanks
>>>>>
>>>>
>>>
>>
>

Mime
View raw message