incubator-hcatalog-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rajesh Balamohan <rajesh.balamo...@gmail.com>
Subject Re: Out of Memory Error - Plz suggest if a JIRA needs to be created
Date Mon, 03 Sep 2012 09:23:34 GMT
After applying the patch and with Hcatalog 0.4.1 + libthrift-0.8.0, the
memory issue got resolved. However, we get the following exception

Any pointers would be of great help (we tried  applying
https://issues.apache.org/jira/browse/THRIFT-1447 which doesn't work with
Hive)

2012-09-03 00:01:56,395 ERROR server.TThreadPoolServer
(TThreadPoolServer.java:run(182)) - Error occurred during processing of
message.
java.lang.NullPointerException
        at
org.apache.thrift.ProcessFunction.process(ProcessFunction.java:34)
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
        at
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge20S.java:519)
        at
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge20S.java:516)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
        at
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge20S.java:516)
        at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:176)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)


2012-09-03 02:17:46,051 ERROR server.TThreadPoolServer
(TThreadPoolServer.java:run(182)) - Error occurred during processing of
message.
java.lang.RuntimeException: org.apache.thrift.transport.TTransportException
        at
org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)
        at
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge20S.java:561)
        at
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge20S.java:559)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:337)
        at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1159)
        at
org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge20S.java:559)
        at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:170)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.thrift.transport.TTransportException
        at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
        at
org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
        at
org.apache.thrift.transport.TSaslTransport.receiveSaslMessage(TSaslTransport.java:178)
        at
org.apache.thrift.transport.TSaslServerTransport.handleSaslStartMessage(TSaslServerTransport.java:125)
        at
org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:253)
        at
org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41)
        at
org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216)
        ... 10 more

~Rajesh.B



On Tue, Aug 14, 2012 at 5:54 AM, Rajesh Balamohan <
rajesh.balamohan@gmail.com> wrote:

> Thanks Rohini, we are planning to apply this patch on same lines.
>
> ~Rajesh.B
>
>
> On Tue, Aug 14, 2012 at 12:24 AM, Rohini Palaniswamy <
> rohini.aditya@gmail.com> wrote:
>
>> Internally in Yahoo we use a patched library of thrift in our deployments
>> to solve this issue. Need to get hive to move to the libthrift 0.9 that has
>> the fix for THRIFT-1468<https://issues.apache.org/jira/browse/THRIFT-1468>
>>
>> Regards,
>> Rohini
>>
>>
>> On Wed, Aug 8, 2012 at 8:21 PM, Rajesh Balamohan <
>> rajesh.balamohan@gmail.com> wrote:
>>
>>> Thanks for the pointers Alan. I believe I need to apply both Hive-3008
>>> and the thrift-1468.
>>>
>>> Analyzing heapdump points 2 GB of space occupied by
>>> TSaslServerTransport. I do not see TUGIContainingTransport occupying
>>> memory.
>>>
>>> ~Rajesh.B
>>>
>>>
>>> On Thu, Aug 9, 2012 at 12:25 AM, Alan Gates <gates@hortonworks.com>wrote:
>>>
>>>> This looks like it may be related to HIVE-3008.  I've tested back
>>>> porting this to Hive 0.9.  I don't know how easy it would be to backport
it
>>>> to Hive 0.8.
>>>>
>>>> Alan.
>>>>
>>>> On Aug 8, 2012, at 7:33 AM, Rajesh Balamohan wrote:
>>>>
>>>> > I took the heapdump and analyzed it. It stems from
>>>> TsaslServerTransport WeakHahsMap.
>>>> >
>>>> > I gues, its related to
>>>> https://issues.apache.org/jira/browse/THRIFT-1468.
>>>> >
>>>> > Are there any fixes or workarounds for this issue in hive 0.8 itself
>>>> (as we wouldn't be able to move to Hive 0.9 immediately).
>>>> >
>>>> > Any pointers would be of great help.
>>>> >
>>>> > ~Rajesh.B
>>>> >
>>>> > On Wed, Aug 8, 2012 at 7:56 PM, Rajesh Balamohan <
>>>> rajesh.balamohan@gmail.com> wrote:
>>>> > Hi All,
>>>> >
>>>> > I am currently using HCatalog 0.4 with PIG 0.9.2.
>>>> >
>>>> > After 3 or 4 days, I encountered PIG exceptions and digging deep I
>>>> found that it tries to connect to HCatalog server and gets socket read
>>>> timeout.
>>>> >
>>>> > Looking at the JVM process of Hcatalog server suggest OOM. I have
>>>> attached the jstat details for further reference.
>>>> >
>>>> > At this point in time, I am not sure which usecase causes this OOM.
I
>>>> would post more details as and when I find. Should I raise a JIRA for
>>>> this?. Plz suggest.
>>>> >
>>>> > [rajesh@machineXYZ hcatalog]$ sudo  /usr/java/jdk1.6.0_25/bin/jstat
>>>> -gcutil 7279 250 9
>>>> >   S0     S1     E      O      P     YGC     YGCT    FGC    FGCT
>>>> GCT
>>>> >   0.00   0.00  71.38  99.99  59.76  33781  478.507  6095 2358.672
>>>> 2837.179
>>>> >   0.00   0.00  71.38  99.99  59.76  33781  478.507  6095 2358.672
>>>> 2837.179
>>>> >   0.00   0.00  71.38  99.99  59.76  33781  478.507  6095 2358.672
>>>> 2837.179
>>>> >   0.00   0.00  71.38  99.99  59.76  33781  478.507  6095 2358.672
>>>> 2837.179
>>>> >   0.00   0.00  71.38  99.99  59.76  33781  478.507  6095 2358.672
>>>> 2837.179
>>>> >   0.00   0.00  71.38  99.99  59.76  33781  478.507  6095 2358.672
>>>> 2837.179
>>>> >   0.00   0.00  71.64  99.99  59.76  33781  478.507  6095 2358.672
>>>> 2837.179
>>>> >   0.00   0.00  71.64  99.99  59.76  33781  478.507  6096 2358.712
>>>> 2837.219
>>>> >   0.00   0.00  71.64  99.99  59.76  33781  478.507  6097 2358.743
>>>> 2837.249
>>>> >
>>>> > --
>>>> > ~Rajesh.B
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > ~Rajesh.B
>>>>
>>>>
>>>
>>>
>>> --
>>> ~Rajesh.B
>>>
>>
>>
>
>
> --
> ~Rajesh.B
>



-- 
~Rajesh.B

Mime
View raw message