incubator-hcatalog-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From agateaaa <agate...@gmail.com>
Subject Re: Out of Memory Error - Plz suggest if a JIRA needs to be created
Date Fri, 14 Sep 2012 21:32:43 GMT
Hi Rajesh

I have been using HCatalog Version 0.4.1, Hive 0.9 (patched HIVE-3008) with
Thrift 0.7 (patched for THRIFT-1438)

Its been running ok so far  - but  then again our current use case
exercises HCatalog very minimally

Any reason you decided to move to libthrift-0.8 ?

~Agatea



On Mon, Sep 3, 2012 at 2:23 AM, Rajesh Balamohan <rajesh.balamohan@gmail.com
> wrote:

> After applying the patch and with Hcatalog 0.4.1 + libthrift-0.8.0, the
> memory issue got resolved. However, we get the following exception
>
> Any pointers would be of great help (we tried  applying
> https://issues.apache.org/jira/browse/THRIFT-1447 which doesn't work with
> Hive)
>
> 2012-09-03 00:01:56,395 ERROR server.TThreadPoolServer
> (TThreadPoolServer.java:run(182)) - Error occurred during processing of
> message.
> java.lang.NullPointerException
>         at
> org.apache.thrift.ProcessFunction.process(ProcessFunction.java:34)
>         at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
>         at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge20S.java:519)
>         at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge20S.java:516)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:396)
>         at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
>         at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge20S.java:516)
>         at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:176)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:662)
>
>
> 2012-09-03 02:17:46,051 ERROR server.TThreadPoolServer
> (TThreadPoolServer.java:run(182)) - Error occurred during processing of
> message.
> java.lang.RuntimeException: org.apache.thrift.transport.TTransportException
>         at
> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
>         at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge20S.java:561)
>         at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge20S.java:559)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:337)
>         at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1159)
>         at
> org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge20S.java:559)
>         at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:170)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:662)
> Caused by: org.apache.thrift.transport.TTransportException
>         at
> org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
>         at
> org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
>         at
> org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178)
>         at
> org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
>         at
> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253)
>         at
> org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
>         at
> org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
>         ... 10 more
>
> ~Rajesh.B
>
>
>
> On Tue, Aug 14, 2012 at 5:54 AM, Rajesh Balamohan <
> rajesh.balamohan@gmail.com> wrote:
>
>> Thanks Rohini, we are planning to apply this patch on same lines.
>>
>> ~Rajesh.B
>>
>>
>> On Tue, Aug 14, 2012 at 12:24 AM, Rohini Palaniswamy <
>> rohini.aditya@gmail.com> wrote:
>>
>>> Internally in Yahoo we use a patched library of thrift in our
>>> deployments to solve this issue. Need to get hive to move to the libthrift
>>> 0.9 that has the fix for THRIFT-1468<https://issues.apache.org/jira/browse/THRIFT-1468>
>>>
>>> Regards,
>>> Rohini
>>>
>>>
>>> On Wed, Aug 8, 2012 at 8:21 PM, Rajesh Balamohan <
>>> rajesh.balamohan@gmail.com> wrote:
>>>
>>>> Thanks for the pointers Alan. I believe I need to apply both Hive-3008
>>>> and the thrift-1468.
>>>>
>>>> Analyzing heapdump points 2 GB of space occupied by
>>>> TSaslServerTransport. I do not see TUGIContainingTransport occupying
>>>> memory.
>>>>
>>>> ~Rajesh.B
>>>>
>>>>
>>>> On Thu, Aug 9, 2012 at 12:25 AM, Alan Gates <gates@hortonworks.com>wrote:
>>>>
>>>>> This looks like it may be related to HIVE-3008.  I've tested back
>>>>> porting this to Hive 0.9.  I don't know how easy it would be to backport
it
>>>>> to Hive 0.8.
>>>>>
>>>>> Alan.
>>>>>
>>>>> On Aug 8, 2012, at 7:33 AM, Rajesh Balamohan wrote:
>>>>>
>>>>> > I took the heapdump and analyzed it. It stems from
>>>>> TsaslServerTransport WeakHahsMap.
>>>>> >
>>>>> > I gues, its related to
>>>>> https://issues.apache.org/jira/browse/THRIFT-1468.
>>>>> >
>>>>> > Are there any fixes or workarounds for this issue in hive 0.8 itself
>>>>> (as we wouldn't be able to move to Hive 0.9 immediately).
>>>>> >
>>>>> > Any pointers would be of great help.
>>>>> >
>>>>> > ~Rajesh.B
>>>>> >
>>>>> > On Wed, Aug 8, 2012 at 7:56 PM, Rajesh Balamohan <
>>>>> rajesh.balamohan@gmail.com> wrote:
>>>>> > Hi All,
>>>>> >
>>>>> > I am currently using HCatalog 0.4 with PIG 0.9.2.
>>>>> >
>>>>> > After 3 or 4 days, I encountered PIG exceptions and digging deep
I
>>>>> found that it tries to connect to HCatalog server and gets socket read
>>>>> timeout.
>>>>> >
>>>>> > Looking at the JVM process of Hcatalog server suggest OOM. I have
>>>>> attached the jstat details for further reference.
>>>>> >
>>>>> > At this point in time, I am not sure which usecase causes this OOM.
>>>>> I would post more details as and when I find. Should I raise a JIRA for
>>>>> this?. Plz suggest.
>>>>> >
>>>>> > [rajesh@machineXYZ hcatalog]$ sudo  /usr/java/jdk1.6.0_25/bin/jstat
>>>>> -gcutil 7279 250 9
>>>>> >   S0     S1     E      O      P     YGC     YGCT    FGC    FGCT
>>>>> GCT
>>>>> >   0.00   0.00  71.38  99.99  59.76  33781  478.507  6095 2358.672
>>>>> 2837.179
>>>>> >   0.00   0.00  71.38  99.99  59.76  33781  478.507  6095 2358.672
>>>>> 2837.179
>>>>> >   0.00   0.00  71.38  99.99  59.76  33781  478.507  6095 2358.672
>>>>> 2837.179
>>>>> >   0.00   0.00  71.38  99.99  59.76  33781  478.507  6095 2358.672
>>>>> 2837.179
>>>>> >   0.00   0.00  71.38  99.99  59.76  33781  478.507  6095 2358.672
>>>>> 2837.179
>>>>> >   0.00   0.00  71.38  99.99  59.76  33781  478.507  6095 2358.672
>>>>> 2837.179
>>>>> >   0.00   0.00  71.64  99.99  59.76  33781  478.507  6095 2358.672
>>>>> 2837.179
>>>>> >   0.00   0.00  71.64  99.99  59.76  33781  478.507  6096 2358.712
>>>>> 2837.219
>>>>> >   0.00   0.00  71.64  99.99  59.76  33781  478.507  6097 2358.743
>>>>> 2837.249
>>>>> >
>>>>> > --
>>>>> > ~Rajesh.B
>>>>> >
>>>>> >
>>>>> >
>>>>> > --
>>>>> > ~Rajesh.B
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> ~Rajesh.B
>>>>
>>>
>>>
>>
>>
>> --
>> ~Rajesh.B
>>
>
>
>
> --
> ~Rajesh.B
>

Mime
View raw message