hadoop-hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Min Zhou <coderp...@gmail.com>
Subject Re: [VOTE] hive release candidate 0.4.1-rc0
Date Mon, 02 Nov 2009 09:06:27 GMT
No, it's zip.

On Mon, Nov 2, 2009 at 4:03 PM, Zheng Shao <zshao9@gmail.com> wrote:
> Do you mean gzip codec?
> I think an empty gzip file should be 20 bytes. There might be some
> problem with the gzip codec (or native gzip codec) on your cluster.
> Can you check the log message of the map tasks whether it has a line
> called "Successfully loaded native gzip lib"?
>
> You can try any query that produces empty results - it should go
> through the same code path.
>
> Zheng
>
> On Sun, Nov 1, 2009 at 11:07 PM, Min Zhou <coderplay@gmail.com> wrote:
>> we use zip codec in default.
>> Some of the same lines were omitted from the error stack:
>> at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:272)
>>
>>
>> Thanks,
>> Min
>>
>> On Mon, Nov 2, 2009 at 2:57 PM, Zheng Shao <zshao9@gmail.com> wrote:
>>> Min, can you check the default compression codec in your hadoop conf?
>>> The 8-byte file must be a compressed file using the codec which
>>> represents 0-length file.
>>>
>>> It seems that codec was not able to decompress the stream.
>>>
>>> Zheng
>>>
>>> On Sun, Nov 1, 2009 at 10:49 PM, Min Zhou <coderplay@gmail.com> wrote:
>>>> I think there may be a bug still in this release.
>>>>
>>>> hive>select stuff_status from auctions where auction_id='2591238417'
>>>> and pt='20091027';
>>>>
>>>> auctions is a table partitioned by date, it stored as a textfile w/o
>>>> compression. The query above should return 0 rows.
>>>> but when hive.exec.compress.output=true,  hive will crash with a
>>>> StackOverflowError
>>>>
>>>> java.lang.StackOverflowError
>>>>        at java.lang.ref.FinalReference.<init>(FinalReference.java:16)
>>>>        at java.lang.ref.Finalizer.<init>(Finalizer.java:66)
>>>>        at java.lang.ref.Finalizer.register(Finalizer.java:72)
>>>>        at java.lang.Object.<init>(Object.java:20)
>>>>        at java.net.SocketImpl.<init>(SocketImpl.java:27)
>>>>        at java.net.PlainSocketImpl.<init>(PlainSocketImpl.java:90)
>>>>        at java.net.SocksSocketImpl.<init>(SocksSocketImpl.java:33)
>>>>        at java.net.Socket.setImpl(Socket.java:434)
>>>>        at java.net.Socket.<init>(Socket.java:68)
>>>>        at sun.nio.ch.SocketAdaptor.<init>(SocketAdaptor.java:50)
>>>>        at sun.nio.ch.SocketAdaptor.create(SocketAdaptor.java:55)
>>>>        at sun.nio.ch.SocketChannelImpl.socket(SocketChannelImpl.java:105)
>>>>        at org.apache.hadoop.net.StandardSocketFactory.createSocket(StandardSocketFactory.java:58)
>>>>        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1540)
>>>>        at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1662)
>>>>        at java.io.DataInputStream.read(DataInputStream.java:132)
>>>>        at org.apache.hadoop.io.compress.DecompressorStream.getCompressedData(DecompressorStream.java:96)
>>>>        at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:86)
>>>>        at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:74)
>>>>        at java.io.InputStream.read(InputStream.java:85)
>>>>        at org.apache.hadoop.util.LineReader.backfill(LineReader.java:82)
>>>>        at org.apache.hadoop.util.LineReader.readLine(LineReader.java:112)
>>>>        at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:134)
>>>>        at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:39)
>>>>        at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:256)
>>>>        at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:272)
>>>>
>>>> Each mapper will produce a 8 bytes deflate file on hdfs(we set
>>>> hive.merge.mapfiles=false), their hex representation  is like below:
>>>>
>>>> 78 9C 03 00 00 00 00 01
>>>>
>>>> This is the reason why FetchOperator:272 is called recursively, and
>>>> caused a stack overflow error.
>>>>
>>>> Regards,
>>>> Min
>>>>
>>>>
>>>> On Mon, Nov 2, 2009 at 6:34 AM, Zheng Shao <zshao9@gmail.com> wrote:
>>>>> I have made a release candidate 0.4.1-rc0.
>>>>>
>>>>> We've fixed several critical bugs to hive release 0.4.0. We need hive
>>>>> release 0.4.1 out asap.
>>>>>
>>>>> Here are the list of changes:
>>>>>
>>>>>    HIVE-884. Metastore Server should call System.exit() on error.
>>>>>    (Zheng Shao via pchakka)
>>>>>
>>>>>    HIVE-864. Fix map-join memory-leak.
>>>>>    (Namit Jain via zshao)
>>>>>
>>>>>    HIVE-878. Update the hash table entry before flushing in Group By
>>>>>    hash aggregation (Zheng Shao via namit)
>>>>>
>>>>>    HIVE-882. Create a new directory every time for scratch.
>>>>>    (Namit Jain via zshao)
>>>>>
>>>>>    HIVE-890. Fix cli.sh for detecting Hadoop versions. (Paul Huff via
zshao)
>>>>>
>>>>>    HIVE-892. Hive to kill hadoop jobs using POST. (Dhruba Borthakur
via zshao)
>>>>>
>>>>>    HIVE-883. URISyntaxException when partition value contains special
chars.
>>>>>    (Zheng Shao via namit)
>>>>>
>>>>>
>>>>> Please vote.
>>>>>
>>>>> --
>>>>> Yours,
>>>>> Zheng
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> My research interests are distributed systems, parallel computing and
>>>> bytecode based virtual machine.
>>>>
>>>> My profile:
>>>> http://www.linkedin.com/in/coderplay
>>>> My blog:
>>>> http://coderplay.javaeye.com
>>>>
>>>
>>>
>>>
>>> --
>>> Yours,
>>> Zheng
>>>
>>
>>
>>
>> --
>> My research interests are distributed systems, parallel computing and
>> bytecode based virtual machine.
>>
>> My profile:
>> http://www.linkedin.com/in/coderplay
>> My blog:
>> http://coderplay.javaeye.com
>>
>
>
>
> --
> Yours,
> Zheng
>



-- 
My research interests are distributed systems, parallel computing and
bytecode based virtual machine.

My profile:
http://www.linkedin.com/in/coderplay
My blog:
http://coderplay.javaeye.com

Mime
View raw message