hadoop-hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Min Zhou <coderp...@gmail.com>
Subject Re: [VOTE] hive release candidate 0.4.1-rc0
Date Mon, 02 Nov 2009 07:07:43 GMT
we use zip codec in default.
Some of the same lines were omitted from the error stack:
at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:272)


Thanks,
Min

On Mon, Nov 2, 2009 at 2:57 PM, Zheng Shao <zshao9@gmail.com> wrote:
> Min, can you check the default compression codec in your hadoop conf?
> The 8-byte file must be a compressed file using the codec which
> represents 0-length file.
>
> It seems that codec was not able to decompress the stream.
>
> Zheng
>
> On Sun, Nov 1, 2009 at 10:49 PM, Min Zhou <coderplay@gmail.com> wrote:
>> I think there may be a bug still in this release.
>>
>> hive>select stuff_status from auctions where auction_id='2591238417'
>> and pt='20091027';
>>
>> auctions is a table partitioned by date, it stored as a textfile w/o
>> compression. The query above should return 0 rows.
>> but when hive.exec.compress.output=true,  hive will crash with a
>> StackOverflowError
>>
>> java.lang.StackOverflowError
>>        at java.lang.ref.FinalReference.<init>(FinalReference.java:16)
>>        at java.lang.ref.Finalizer.<init>(Finalizer.java:66)
>>        at java.lang.ref.Finalizer.register(Finalizer.java:72)
>>        at java.lang.Object.<init>(Object.java:20)
>>        at java.net.SocketImpl.<init>(SocketImpl.java:27)
>>        at java.net.PlainSocketImpl.<init>(PlainSocketImpl.java:90)
>>        at java.net.SocksSocketImpl.<init>(SocksSocketImpl.java:33)
>>        at java.net.Socket.setImpl(Socket.java:434)
>>        at java.net.Socket.<init>(Socket.java:68)
>>        at sun.nio.ch.SocketAdaptor.<init>(SocketAdaptor.java:50)
>>        at sun.nio.ch.SocketAdaptor.create(SocketAdaptor.java:55)
>>        at sun.nio.ch.SocketChannelImpl.socket(SocketChannelImpl.java:105)
>>        at org.apache.hadoop.net.StandardSocketFactory.createSocket(StandardSocketFactory.java:58)
>>        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1540)
>>        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1662)
>>        at java.io.DataInputStream.read(DataInputStream.java:132)
>>        at org.apache.hadoop.io.compress.DecompressorStream.getCompressedData(DecompressorStream.java:96)
>>        at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:86)
>>        at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:74)
>>        at java.io.InputStream.read(InputStream.java:85)
>>        at org.apache.hadoop.util.LineReader.backfill(LineReader.java:82)
>>        at org.apache.hadoop.util.LineReader.readLine(LineReader.java:112)
>>        at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:134)
>>        at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:39)
>>        at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:256)
>>        at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:272)
>>
>> Each mapper will produce a 8 bytes deflate file on hdfs(we set
>> hive.merge.mapfiles=false), their hex representation  is like below:
>>
>> 78 9C 03 00 00 00 00 01
>>
>> This is the reason why FetchOperator:272 is called recursively, and
>> caused a stack overflow error.
>>
>> Regards,
>> Min
>>
>>
>> On Mon, Nov 2, 2009 at 6:34 AM, Zheng Shao <zshao9@gmail.com> wrote:
>>> I have made a release candidate 0.4.1-rc0.
>>>
>>> We've fixed several critical bugs to hive release 0.4.0. We need hive
>>> release 0.4.1 out asap.
>>>
>>> Here are the list of changes:
>>>
>>>    HIVE-884. Metastore Server should call System.exit() on error.
>>>    (Zheng Shao via pchakka)
>>>
>>>    HIVE-864. Fix map-join memory-leak.
>>>    (Namit Jain via zshao)
>>>
>>>    HIVE-878. Update the hash table entry before flushing in Group By
>>>    hash aggregation (Zheng Shao via namit)
>>>
>>>    HIVE-882. Create a new directory every time for scratch.
>>>    (Namit Jain via zshao)
>>>
>>>    HIVE-890. Fix cli.sh for detecting Hadoop versions. (Paul Huff via zshao)
>>>
>>>    HIVE-892. Hive to kill hadoop jobs using POST. (Dhruba Borthakur via zshao)
>>>
>>>    HIVE-883. URISyntaxException when partition value contains special chars.
>>>    (Zheng Shao via namit)
>>>
>>>
>>> Please vote.
>>>
>>> --
>>> Yours,
>>> Zheng
>>>
>>
>>
>>
>> --
>> My research interests are distributed systems, parallel computing and
>> bytecode based virtual machine.
>>
>> My profile:
>> http://www.linkedin.com/in/coderplay
>> My blog:
>> http://coderplay.javaeye.com
>>
>
>
>
> --
> Yours,
> Zheng
>



-- 
My research interests are distributed systems, parallel computing and
bytecode based virtual machine.

My profile:
http://www.linkedin.com/in/coderplay
My blog:
http://coderplay.javaeye.com

Mime
View raw message