kylin-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jakob Stengård <jakob.steng...@gmail.com>
Subject Re: Tez
Date Mon, 02 Mar 2015 18:38:22 GMT
I looked at the source code, but the fix wasn't obvious to me.

Maybe, if i have time, i'll look into it further.

/J


On Mon, Mar 2, 2015 at 7:13 PM, Adunuthula, Seshu <sadunuthula@ebay.com>
wrote:

> Thanks Jakob,
>
> Would be fantastic if you could dig in and submit a patch…
>
>
> On 3/2/15, 10:08 AM, "Jakob Stengård" <jakob.stengard@gmail.com> wrote:
>
> >I'll file a bug in jira.
> >For your information i also happen to know that tez boosted map-reduce
> >doesn't work either, but there, the problem seems to have to do with the
> >map reduce jobs them selves failing.
> >
> >In this case however, from what i've read on git hub about the "Can't get
> >cube source record size",
> >error", it could possibly be a problem with that tez doesn't output a
> >string on the form:
> >"HDFS Read: (\d+) HDFS Write"
> >
> >However, it could be something else, it could also be my logging
> >configuration for tez.
> >
> >/J
> >
> >
> >
> >On Mon, Mar 2, 2015 at 6:54 PM, Adunuthula, Seshu <sadunuthula@ebay.com>
> >wrote:
> >
> >> Did not read your logs fully. The culprit seems to be
> >>
> >>
> >> 1. [QuartzScheduler_Worker-7]:[2015-03-02
> >>
> >>16:09:24,737][ERROR][com.kylinolap.job.flow.JobFlowListener.handleExcepti
> >>on
> >> (JobFlowListener.java:312)] - Can't get cube source record size.
> >>
> >> source_records_size may have to be computed differently when using Tez
> >>as
> >> the Hive engine. I will let Yang/Luke answer this when they come back
> >> onlineŠ
> >>
> >>
> >>
> >>
> >> On 3/2/15, 9:30 AM, "Adunuthula, Seshu" <sadunuthula@ebay.com> wrote:
> >>
> >> >Jakob,
> >> >
> >> >Could you go ahead and file a JIRA with the stack trace at
> >> >https://issues.apache.org/jira/browse/KYLIN/
> >> >
> >> >Also include the App Master and Map/Reduce Task logs for the Tasks that
> >> >failed.
> >> >
> >> >Regards
> >> >Seshu
> >> >
> >> >On 3/2/15, 8:12 AM, "Jakob Stengård" <jakob.stengard@gmail.com> wrote:
> >> >
> >> >>Correction: That, isn't the last job, it's step 9.
> >> >>
> >> >>/J
> >> >>
> >> >>On Mon, Mar 2, 2015 at 5:10 PM, Jakob Stengård
> >><jakob.stengard@gmail.com
> >> >
> >> >>wrote:
> >> >>
> >> >>> Well, this only happens with
> >> >>> <property>
> >> >>>   <name>hive.execution.engine</name>
> >> >>>   <value>tez</value>
> >> >>> </property>
> >> >>>
> >> >>> If i use mr, it works fine.
> >> >>>
> >> >>>
> >> >>> From what i can see form the job log, none of the previous
> >>map-reduce
> >> >>>jobs
> >> >>> fail, but the build fails at the last step.
> >> >>> Log from the last mr job for example:
> >> >>> http://pastebin.com/jNWSRKLi
> >> >>>
> >> >>>
> >> >>> Is tez supposed to work? Because in that case, maybe it's just
my
> >> >>>setup.
> >> >>>
> >> >>> /Jakob
> >> >>>
> >> >>>
> >> >>>
> >> >>> On Mon, Mar 2, 2015 at 4:34 PM, Santosh Akhilesh <
> >> >>> santoshakhilesh@gmail.com> wrote:
> >> >>>
> >> >>>> This doesn't seems to be complete kylin.log. Do you have any
> >>exception
> >> >>>> related to MR counter in map reduce logs? If yes then this
needs
> >>to be
> >> >>>> checked because kylin will need the MR counter to calculate
the
> >>cube
> >> >>>>size.
> >> >>>> On Mon, 2 Mar 2015 at 8:49 pm, Jakob Stengård
> >> >>>><jakob.stengard@gmail.com>
> >> >>>> wrote:
> >> >>>>
> >> >>>> > It breaks on the last step: "Load HFile to HBase Table":
> >> >>>> >
> >> >>>> > Kylin.log:
> >> >>>> > http://pastebin.com/3kAQmJ3u
> >> >>>> >
> >> >>>> > On Mon, Mar 2, 2015 at 2:27 PM, Adunuthula, Seshu
> >> >>>><sadunuthula@ebay.com
> >> >>>> >
> >> >>>> > wrote:
> >> >>>> >
> >> >>>> > > Jakob,
> >> >>>> > >
> >> >>>> > > Could you please send out the complete stack trace
with Tez
> >> >>>>enabled?
> >> >>>> > >
> >> >>>> > > Regards
> >> >>>> > > Seshu
> >> >>>> > >
> >> >>>> > > On 3/2/15, 4:51 AM, "Jakob Stengård" <jakob.stengard@gmail.com
> >
> >> >>>> wrote:
> >> >>>> > >
> >> >>>> > > >Hi
> >> >>>> > > >Will kylin suport hive+tez in v0.7.x?
> >> >>>> > > >
> >> >>>> > > >I've tried using tez with hive, but currently
the cube build
> >> >>>>fails
> >> >>>> with
> >> >>>> > > >"Can't get cube source record size", because
kylin expects
> >> >>>>certain
> >> >>>> > console
> >> >>>> > > >output from map-reduce, and tez doesn't print
this.
> >> >>>> > > >
> >> >>>> > > >BG.
> >> >>>> > > >Jakob Stengard
> >> >>>> > >
> >> >>>> > >
> >> >>>> >
> >> >>>>
> >> >>>
> >> >>>
> >> >
> >>
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message