hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Viral Bajaria <viral.baja...@gmail.com>
Subject Re: Hive Server - Transport error occurred during acceptance of message
Date Sat, 12 Feb 2011 00:36:35 GMT
We use hive through the thrift api too. I have seen the above error when
someone accidentally connects to the thrift server port through multiple
instances of our hive-wrapper code. We open a bunch of hive thrift ports to
achieve the automation of running multiple hive queries in parallel i.e.
continuous aggregations.

you need to make sure you are not facing a similar issue because the hive
thrift server as far as i know is single threaded up until hive 0.5.0 (the
version that i use). Not too sure if that's been changed in 0.6.0 or higher.

-Viral

On Fri, Feb 11, 2011 at 7:06 AM, Ajo Fod <ajo.fod@gmail.com> wrote:

> Are you using hive 0.6? ... may be fixed in the latest version.
>
> Also I wonder why these thrift libraries are being used ... is this normal
> hive operation, or can you do something to avoid using thrift?
>
> -Ajo
>
>
> On Fri, Feb 11, 2011 at 12:05 AM, vaibhav negi <sssssssenator@gmail.com>wrote:
>
>>
>> Hi all,
>>
>> I am loading data into hive tables by connecting to hiveserver through
>> thrift api using "load data local inpath ... " query .
>> Hive server is running as a background process for days . After some 4-5
>> days hiveserver throws this error :
>>
>> 11/02/11 11:20:02 WARN server.TThreadPoolServer: Transport error occurred
>> during acceptance of message.
>> org.apache.thrift.transport.TTransportException: java.net.SocketException:
>> Too many open files
>>         at
>> org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:124)
>>         at
>> org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:35)
>>         at
>> org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31)
>>         at
>> org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:184)
>>         at
>> org.apache.hadoop.hive.service.HiveServer.main(HiveServer.java:388)
>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>         at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>         at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>         at java.lang.reflect.Method.invoke(Method.java:597)
>>         at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>> Caused by: java.net.SocketException: Too many open files
>>         at java.net.PlainSocketImpl.socketAccept(Native Method)
>>         at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>>         at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>>         at java.net.ServerSocket.accept(ServerSocket.java:421)
>>         at
>> org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:119)
>>         ... 9 more
>>
>> I checked network connections using netstat and it show about 1000 active
>> network connections, most of them to hadoop datanode . Output is attached as
>> netstat.out
>>
>> after i restarted hiveserver network connections falls to about 20 .
>> output is attached as netstat_after_HS_restart.out
>>
>> Is this a problem with hive or hadoop configuration ?
>>
>> Vaibhav Negi
>>
>
>

Mime
View raw message