kylin-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Adunuthula, Seshu" <sadunuth...@ebay.com>
Subject Re: Tez
Date Mon, 02 Mar 2015 18:13:55 GMT
Thanks Jakob,

Would be fantastic if you could dig in and submit a patch…


On 3/2/15, 10:08 AM, "Jakob Stengård" <jakob.stengard@gmail.com> wrote:

>I'll file a bug in jira.
>For your information i also happen to know that tez boosted map-reduce
>doesn't work either, but there, the problem seems to have to do with the
>map reduce jobs them selves failing.
>
>In this case however, from what i've read on git hub about the "Can't get
>cube source record size",
>error", it could possibly be a problem with that tez doesn't output a
>string on the form:
>"HDFS Read: (\d+) HDFS Write"
>
>However, it could be something else, it could also be my logging
>configuration for tez.
>
>/J
>
>
>
>On Mon, Mar 2, 2015 at 6:54 PM, Adunuthula, Seshu <sadunuthula@ebay.com>
>wrote:
>
>> Did not read your logs fully. The culprit seems to be
>>
>>
>> 1. [QuartzScheduler_Worker-7]:[2015-03-02
>> 
>>16:09:24,737][ERROR][com.kylinolap.job.flow.JobFlowListener.handleExcepti
>>on
>> (JobFlowListener.java:312)] - Can't get cube source record size.
>>
>> source_records_size may have to be computed differently when using Tez
>>as
>> the Hive engine. I will let Yang/Luke answer this when they come back
>> onlineŠ
>>
>>
>>
>>
>> On 3/2/15, 9:30 AM, "Adunuthula, Seshu" <sadunuthula@ebay.com> wrote:
>>
>> >Jakob,
>> >
>> >Could you go ahead and file a JIRA with the stack trace at
>> >https://issues.apache.org/jira/browse/KYLIN/
>> >
>> >Also include the App Master and Map/Reduce Task logs for the Tasks that
>> >failed.
>> >
>> >Regards
>> >Seshu
>> >
>> >On 3/2/15, 8:12 AM, "Jakob Stengård" <jakob.stengard@gmail.com> wrote:
>> >
>> >>Correction: That, isn't the last job, it's step 9.
>> >>
>> >>/J
>> >>
>> >>On Mon, Mar 2, 2015 at 5:10 PM, Jakob Stengård
>><jakob.stengard@gmail.com
>> >
>> >>wrote:
>> >>
>> >>> Well, this only happens with
>> >>> <property>
>> >>>   <name>hive.execution.engine</name>
>> >>>   <value>tez</value>
>> >>> </property>
>> >>>
>> >>> If i use mr, it works fine.
>> >>>
>> >>>
>> >>> From what i can see form the job log, none of the previous
>>map-reduce
>> >>>jobs
>> >>> fail, but the build fails at the last step.
>> >>> Log from the last mr job for example:
>> >>> http://pastebin.com/jNWSRKLi
>> >>>
>> >>>
>> >>> Is tez supposed to work? Because in that case, maybe it's just my
>> >>>setup.
>> >>>
>> >>> /Jakob
>> >>>
>> >>>
>> >>>
>> >>> On Mon, Mar 2, 2015 at 4:34 PM, Santosh Akhilesh <
>> >>> santoshakhilesh@gmail.com> wrote:
>> >>>
>> >>>> This doesn't seems to be complete kylin.log. Do you have any
>>exception
>> >>>> related to MR counter in map reduce logs? If yes then this needs
>>to be
>> >>>> checked because kylin will need the MR counter to calculate the
>>cube
>> >>>>size.
>> >>>> On Mon, 2 Mar 2015 at 8:49 pm, Jakob Stengård
>> >>>><jakob.stengard@gmail.com>
>> >>>> wrote:
>> >>>>
>> >>>> > It breaks on the last step: "Load HFile to HBase Table":
>> >>>> >
>> >>>> > Kylin.log:
>> >>>> > http://pastebin.com/3kAQmJ3u
>> >>>> >
>> >>>> > On Mon, Mar 2, 2015 at 2:27 PM, Adunuthula, Seshu
>> >>>><sadunuthula@ebay.com
>> >>>> >
>> >>>> > wrote:
>> >>>> >
>> >>>> > > Jakob,
>> >>>> > >
>> >>>> > > Could you please send out the complete stack trace with
Tez
>> >>>>enabled?
>> >>>> > >
>> >>>> > > Regards
>> >>>> > > Seshu
>> >>>> > >
>> >>>> > > On 3/2/15, 4:51 AM, "Jakob Stengård" <jakob.stengard@gmail.com>
>> >>>> wrote:
>> >>>> > >
>> >>>> > > >Hi
>> >>>> > > >Will kylin suport hive+tez in v0.7.x?
>> >>>> > > >
>> >>>> > > >I've tried using tez with hive, but currently the
cube build
>> >>>>fails
>> >>>> with
>> >>>> > > >"Can't get cube source record size", because kylin
expects
>> >>>>certain
>> >>>> > console
>> >>>> > > >output from map-reduce, and tez doesn't print this.
>> >>>> > > >
>> >>>> > > >BG.
>> >>>> > > >Jakob Stengard
>> >>>> > >
>> >>>> > >
>> >>>> >
>> >>>>
>> >>>
>> >>>
>> >
>>
>>

Mime
View raw message