hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Evan Pollan <>
Subject Re: Custom SerDe -- tracking down stack trace
Date Wed, 22 Feb 2012 20:08:23 GMT
So, I tracked down the problem.  But, I'm curious as to why I got such different behavior when
selecting directly from the partition vs. selecting from all partitions.

Context:  my custom deserializer was returning null when it encountered an unintelligible
line (I saw this pattern in the contrib RegexSerDe and reused it).  This was apparently causing
the LazySimpleSerDe.serialize() operation to NPE as the CLI driver was fetching the results
when selecting directly from the partition with the bad line:

2012-02-21 22:55:14,166 ERROR CliDriver ( - Failed with
exception java.lang.NullPointerException
        at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(
        at org.apache.hadoop.hive.ql.Driver.getResults(
        at org.apache.hadoop.hive.cli.CliDriver.processCmd(
        at org.apache.hadoop.hive.cli.CliDriver.processLine(
        at org.apache.hadoop.hive.cli.CliDriver.main(
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(
        at java.lang.reflect.Method.invoke(
        at org.apache.hadoop.util.RunJar.main(
Caused by: java.lang.NullPointerException
        at java.util.ArrayList.addAll(
        at org.apache.hadoop.hive.serde2.objectinspector.UnionStructObjectInspector.getStructFieldsDataAsList(
        at org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe.serialize(
        at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(
        ... 9 more

However, when I queried across the entire data set (eliminating the partition predicate),
there query returns without any errors.  Does the CLI behave differently based on the query

From: Evan Pollan <<>>
Reply-To: <<>>
Date: Wed, 22 Feb 2012 12:28:26 +0000
To: "<>" <<>>
Subject: Re: Custom SerDe -- tracking down stack trace

/tmp it is!  My bad — it was the one obvious place I omitted from my find/grep statement.

From: Matthew Byrd <<>>
Reply-To: <<>>
Date: Wed, 22 Feb 2012 11:54:09 +0000
To: <<>>
Subject: Re: Custom SerDe -- tracking down stack trace

Hi Evan,
Did you look in your hive.log file?
Mine is found in /tmp/$USER/ ... usually where stack traces from hive cli show up If I'm not
Have you tried hooking up a debugger to hive yet also? I'm guessing this is how you knew the
null pointer was being thrown on deserialize?
what is actually null?

On Tue, Feb 21, 2012 at 11:01 PM, Evan Pollan <<>>
One more data point:  I can read data from this partition as long as I don't reference the
partition explicitly…

E.g., I my partition column is "ArrivalDate", and I have several different partitions:  "2012-02-01"…,
and a partition with my test data with ArrivalDate="test".

This works:  'select * from table where <some constraint such that I only get results from
the "test" partition>'.

And this works:  'select * from table where ArrivalDate="2012-02-01"'

But, this fails:  'select * from table where ArrivalDate="test"'

Does this make sense to anybody?

From: Evan Pollan <<>>
Reply-To: <<>>
Date: Tue, 21 Feb 2012 20:56:07 +0000
To: "<>" <<>>
Subject: Custom SerDe -- tracking down stack trace

I have a custom SerDe that's initializing properly and works on one data set.  I built it
to adapt to a couple of different data formats, though, and it's choking on a different data
set (different partitions in the same table).

A null pointer exception is being thrown on deserialize, that's being wrapped by an IOException
somewhere up the stack.  The exception is showing up in the hive output ("Failed with exception"), but I can't find the stack trace in
any logs.

It's worth noting that I'm running hive via the cli on a machine external to the cluster,
and the query doesn't get far enough to create any M/R tasks.  I looked in all log files in
/var/log on the hive client machine, and in all userlogs on each cluster instance.  I also
looked in derby.log (I'm using the embedded metastore) and in /var/lib/hive/metastore on the
hive client machine.

I'm sure I'm missing something obvious…  Any ideas?

View raw message