hadoop-hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Min Zhou <coderp...@gmail.com>
Subject Re: [VOTE] hive release candidate 0.4.1-rc0
Date Mon, 02 Nov 2009 06:49:14 GMT
I think there may be a bug still in this release.

hive>select stuff_status from auctions where auction_id='2591238417'
and pt='20091027';

auctions is a table partitioned by date, it stored as a textfile w/o
compression. The query above should return 0 rows.
but when hive.exec.compress.output=true,  hive will crash with a
StackOverflowError

java.lang.StackOverflowError
        at java.lang.ref.FinalReference.<init>(FinalReference.java:16)
        at java.lang.ref.Finalizer.<init>(Finalizer.java:66)
        at java.lang.ref.Finalizer.register(Finalizer.java:72)
        at java.lang.Object.<init>(Object.java:20)
        at java.net.SocketImpl.<init>(SocketImpl.java:27)
        at java.net.PlainSocketImpl.<init>(PlainSocketImpl.java:90)
        at java.net.SocksSocketImpl.<init>(SocksSocketImpl.java:33)
        at java.net.Socket.setImpl(Socket.java:434)
        at java.net.Socket.<init>(Socket.java:68)
        at sun.nio.ch.SocketAdaptor.<init>(SocketAdaptor.java:50)
        at sun.nio.ch.SocketAdaptor.create(SocketAdaptor.java:55)
        at sun.nio.ch.SocketChannelImpl.socket(SocketChannelImpl.java:105)
        at org.apache.hadoop.net.StandardSocketFactory.createSocket(StandardSocketFactory.java:58)
        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1540)
        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1662)
        at java.io.DataInputStream.read(DataInputStream.java:132)
        at org.apache.hadoop.io.compress.DecompressorStream.getCompressedData(DecompressorStream.java:96)
        at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:86)
        at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:74)
        at java.io.InputStream.read(InputStream.java:85)
        at org.apache.hadoop.util.LineReader.backfill(LineReader.java:82)
        at org.apache.hadoop.util.LineReader.readLine(LineReader.java:112)
        at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:134)
        at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:39)
        at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:256)
        at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:272)

Each mapper will produce a 8 bytes deflate file on hdfs(we set
hive.merge.mapfiles=false), their hex representation  is like below:

78 9C 03 00 00 00 00 01

This is the reason why FetchOperator:272 is called recursively, and
caused a stack overflow error.

Regards,
Min


On Mon, Nov 2, 2009 at 6:34 AM, Zheng Shao <zshao9@gmail.com> wrote:
> I have made a release candidate 0.4.1-rc0.
>
> We've fixed several critical bugs to hive release 0.4.0. We need hive
> release 0.4.1 out asap.
>
> Here are the list of changes:
>
>    HIVE-884. Metastore Server should call System.exit() on error.
>    (Zheng Shao via pchakka)
>
>    HIVE-864. Fix map-join memory-leak.
>    (Namit Jain via zshao)
>
>    HIVE-878. Update the hash table entry before flushing in Group By
>    hash aggregation (Zheng Shao via namit)
>
>    HIVE-882. Create a new directory every time for scratch.
>    (Namit Jain via zshao)
>
>    HIVE-890. Fix cli.sh for detecting Hadoop versions. (Paul Huff via zshao)
>
>    HIVE-892. Hive to kill hadoop jobs using POST. (Dhruba Borthakur via zshao)
>
>    HIVE-883. URISyntaxException when partition value contains special chars.
>    (Zheng Shao via namit)
>
>
> Please vote.
>
> --
> Yours,
> Zheng
>



-- 
My research interests are distributed systems, parallel computing and
bytecode based virtual machine.

My profile:
http://www.linkedin.com/in/coderplay
My blog:
http://coderplay.javaeye.com

Mime
View raw message