kylin-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jakob Stengård <jakob.steng...@gmail.com>
Subject Re: Tez
Date Wed, 04 Mar 2015 09:07:27 GMT
I tried installing 0.7.1, but i get "The 'partition_date_column' format is
invalid: , it should be {db}.{table}.{column}."
when trying to set up a cube.

Dunno if it's my tables that it doesn't like or if it's some deprecated
settings from 0.6.5 that causes this.
Tried deleting all Hbase tables, but i still get the error after restarting
Kylin and re-syncing the tables.

/J

On Tue, Mar 3, 2015 at 9:24 AM, Li Yang <liyang@apache.org> wrote:

> I briefly looked into the code and seems the issue has been fixed in 0.7.x
> (the current inverted-index branch) already. Basically, it does not rely on
> counters of the hive step any more.
>
> However I don't have the env to verify this. @Jakob, may try this latest
> version http://kylin.incubator.apache.org/download/
>
> On Tue, Mar 3, 2015 at 6:59 AM, Jakob Stengård <jakob.stengard@gmail.com>
> wrote:
>
> > Yeah, i know that.
> > Here is the jira bug i filed:
> > https://issues.apache.org/jira/browse/KYLIN-619
> >
> > On Mon, Mar 2, 2015 at 11:47 PM, Luke Han <luke.hq@gmail.com> wrote:
> >
> > > Hi Jakob, did you create JIRA for this already?
> > >
> > > Tez itself should not the problem but just some statistics information
> > > Kylin can't parse as you figure out already.
> > >
> > > Please let me know the JIRA number and we will enhance that to avoid
> such
> > > issue again.
> > >
> > > Thanks.
> > > Luke
> > >
> > >
> > > Best Regards!
> > > ---------------------
> > >
> > > Luke Han
> > >
> > > 2015-03-03 2:38 GMT+08:00 Jakob Stengård <jakob.stengard@gmail.com>:
> > >
> > > > I looked at the source code, but the fix wasn't obvious to me.
> > > >
> > > > Maybe, if i have time, i'll look into it further.
> > > >
> > > > /J
> > > >
> > > >
> > > > On Mon, Mar 2, 2015 at 7:13 PM, Adunuthula, Seshu <
> > sadunuthula@ebay.com>
> > > > wrote:
> > > >
> > > > > Thanks Jakob,
> > > > >
> > > > > Would be fantastic if you could dig in and submit a patch…
> > > > >
> > > > >
> > > > > On 3/2/15, 10:08 AM, "Jakob Stengård" <jakob.stengard@gmail.com>
> > > wrote:
> > > > >
> > > > > >I'll file a bug in jira.
> > > > > >For your information i also happen to know that tez boosted
> > map-reduce
> > > > > >doesn't work either, but there, the problem seems to have to
do
> with
> > > the
> > > > > >map reduce jobs them selves failing.
> > > > > >
> > > > > >In this case however, from what i've read on git hub about the
> > "Can't
> > > > get
> > > > > >cube source record size",
> > > > > >error", it could possibly be a problem with that tez doesn't
> output
> > a
> > > > > >string on the form:
> > > > > >"HDFS Read: (\d+) HDFS Write"
> > > > > >
> > > > > >However, it could be something else, it could also be my logging
> > > > > >configuration for tez.
> > > > > >
> > > > > >/J
> > > > > >
> > > > > >
> > > > > >
> > > > > >On Mon, Mar 2, 2015 at 6:54 PM, Adunuthula, Seshu <
> > > sadunuthula@ebay.com
> > > > >
> > > > > >wrote:
> > > > > >
> > > > > >> Did not read your logs fully. The culprit seems to be
> > > > > >>
> > > > > >>
> > > > > >> 1. [QuartzScheduler_Worker-7]:[2015-03-02
> > > > > >>
> > > > >
> > > >
> > >
> >
> >>16:09:24,737][ERROR][com.kylinolap.job.flow.JobFlowListener.handleExcepti
> > > > > >>on
> > > > > >> (JobFlowListener.java:312)] - Can't get cube source record
size.
> > > > > >>
> > > > > >> source_records_size may have to be computed differently
when
> using
> > > Tez
> > > > > >>as
> > > > > >> the Hive engine. I will let Yang/Luke answer this when they
come
> > > back
> > > > > >> onlineŠ
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> On 3/2/15, 9:30 AM, "Adunuthula, Seshu" <sadunuthula@ebay.com>
> > > wrote:
> > > > > >>
> > > > > >> >Jakob,
> > > > > >> >
> > > > > >> >Could you go ahead and file a JIRA with the stack trace
at
> > > > > >> >https://issues.apache.org/jira/browse/KYLIN/
> > > > > >> >
> > > > > >> >Also include the App Master and Map/Reduce Task logs
for the
> > Tasks
> > > > that
> > > > > >> >failed.
> > > > > >> >
> > > > > >> >Regards
> > > > > >> >Seshu
> > > > > >> >
> > > > > >> >On 3/2/15, 8:12 AM, "Jakob Stengård" <jakob.stengard@gmail.com
> >
> > > > wrote:
> > > > > >> >
> > > > > >> >>Correction: That, isn't the last job, it's step
9.
> > > > > >> >>
> > > > > >> >>/J
> > > > > >> >>
> > > > > >> >>On Mon, Mar 2, 2015 at 5:10 PM, Jakob Stengård
> > > > > >><jakob.stengard@gmail.com
> > > > > >> >
> > > > > >> >>wrote:
> > > > > >> >>
> > > > > >> >>> Well, this only happens with
> > > > > >> >>> <property>
> > > > > >> >>>   <name>hive.execution.engine</name>
> > > > > >> >>>   <value>tez</value>
> > > > > >> >>> </property>
> > > > > >> >>>
> > > > > >> >>> If i use mr, it works fine.
> > > > > >> >>>
> > > > > >> >>>
> > > > > >> >>> From what i can see form the job log, none
of the previous
> > > > > >>map-reduce
> > > > > >> >>>jobs
> > > > > >> >>> fail, but the build fails at the last step.
> > > > > >> >>> Log from the last mr job for example:
> > > > > >> >>> http://pastebin.com/jNWSRKLi
> > > > > >> >>>
> > > > > >> >>>
> > > > > >> >>> Is tez supposed to work? Because in that case,
maybe it's
> just
> > > my
> > > > > >> >>>setup.
> > > > > >> >>>
> > > > > >> >>> /Jakob
> > > > > >> >>>
> > > > > >> >>>
> > > > > >> >>>
> > > > > >> >>> On Mon, Mar 2, 2015 at 4:34 PM, Santosh Akhilesh
<
> > > > > >> >>> santoshakhilesh@gmail.com> wrote:
> > > > > >> >>>
> > > > > >> >>>> This doesn't seems to be complete kylin.log.
Do you have
> any
> > > > > >>exception
> > > > > >> >>>> related to MR counter in map reduce logs?
If yes then this
> > > needs
> > > > > >>to be
> > > > > >> >>>> checked because kylin will need the MR
counter to calculate
> > the
> > > > > >>cube
> > > > > >> >>>>size.
> > > > > >> >>>> On Mon, 2 Mar 2015 at 8:49 pm, Jakob Stengård
> > > > > >> >>>><jakob.stengard@gmail.com>
> > > > > >> >>>> wrote:
> > > > > >> >>>>
> > > > > >> >>>> > It breaks on the last step: "Load
HFile to HBase Table":
> > > > > >> >>>> >
> > > > > >> >>>> > Kylin.log:
> > > > > >> >>>> > http://pastebin.com/3kAQmJ3u
> > > > > >> >>>> >
> > > > > >> >>>> > On Mon, Mar 2, 2015 at 2:27 PM, Adunuthula,
Seshu
> > > > > >> >>>><sadunuthula@ebay.com
> > > > > >> >>>> >
> > > > > >> >>>> > wrote:
> > > > > >> >>>> >
> > > > > >> >>>> > > Jakob,
> > > > > >> >>>> > >
> > > > > >> >>>> > > Could you please send out the
complete stack trace with
> > Tez
> > > > > >> >>>>enabled?
> > > > > >> >>>> > >
> > > > > >> >>>> > > Regards
> > > > > >> >>>> > > Seshu
> > > > > >> >>>> > >
> > > > > >> >>>> > > On 3/2/15, 4:51 AM, "Jakob Stengård"
<
> > > > jakob.stengard@gmail.com
> > > > > >
> > > > > >> >>>> wrote:
> > > > > >> >>>> > >
> > > > > >> >>>> > > >Hi
> > > > > >> >>>> > > >Will kylin suport hive+tez
in v0.7.x?
> > > > > >> >>>> > > >
> > > > > >> >>>> > > >I've tried using tez with
hive, but currently the cube
> > > build
> > > > > >> >>>>fails
> > > > > >> >>>> with
> > > > > >> >>>> > > >"Can't get cube source record
size", because kylin
> > expects
> > > > > >> >>>>certain
> > > > > >> >>>> > console
> > > > > >> >>>> > > >output from map-reduce, and
tez doesn't print this.
> > > > > >> >>>> > > >
> > > > > >> >>>> > > >BG.
> > > > > >> >>>> > > >Jakob Stengard
> > > > > >> >>>> > >
> > > > > >> >>>> > >
> > > > > >> >>>> >
> > > > > >> >>>>
> > > > > >> >>>
> > > > > >> >>>
> > > > > >> >
> > > > > >>
> > > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>



-- 

Med vänlig hälsning (Best Regards)
*Jakob Stengård*

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message