hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Teoh <chris.t...@gmail.com>
Subject Re: TBLPROPERTIES appears to be ignored by custom inputformats
Date Mon, 19 Dec 2016 10:58:34 GMT
Thanks Jorn.

I do see the inputformat logging the settings in HiveServer2 logs when I do
SELECT * . I don't see anything in the HiveServer2 log when I do SELECT
count(*). I assume it must be somewhere in the lower levels or because the
count function does some other thing. I also tried writing my own UDAF
count function and it also did not show the inputformat logs. I'm not using
any custom serde, all I want is the multi line data to be returned as one
row but in a configurable manner.

Kind Regards
Chris

On Mon, 19 Dec 2016 at 19:06 Jörn Franke <jornfranke@gmail.com> wrote:

> I assume it is related to lazy evaluation by the serde but this would
> require investigation of the source code and log files.
> If there is no exception in the log files then you forgot to log them
> (bad!). Use sonarqube or similar to check your source code related to those
> kind of mistakes - saves debug and fixing time.
> At the same time, your inputformat should have either default values for
> configuration or where not possible throw an exception if expected
> configuration is  it there.
>
> > On 19 Dec 2016, at 07:53, Chris Teoh <chris.teoh@gmail.com> wrote:
> >
> > Thanks Jorn.
> >
> > I don't understand how my select * is correctly reading my table property
> > then if I'm just using default serde.
> >> On Mon., 19 Dec. 2016 at 5:36 pm, Jörn Franke <jornfranke@gmail.com>
> wrote:
> >>
> >> You have to write a custom hiveserde format to pass tblproperties as
> >> inputformat properties, but check the source code of the serde you used.
> >>
> >>> On 19 Dec 2016, at 07:22, Chris Teoh <chris.teoh@gmail.com> wrote:
> >>>
> >>> rows.
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message