hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ajo Fod <ajo....@gmail.com>
Subject Re: Hive Server - Transport error occurred during acceptance of message
Date Tue, 15 Feb 2011 13:06:22 GMT
You could try to save the information in a tab separated file and try to
load that into hive.

-Ajo

On Tue, Feb 15, 2011 at 1:24 AM, vaibhav negi <sssssssenator@gmail.com>wrote:

> Hi ,
>
> I am using hive version 0.7 ( as suggested by lib/hive-hwi-0.7.0.war) . I
> checked out code from svn two weeks back . I use thrift api's to load data
> into hive programmatically .
>
> Vaibhav Negi
>
>
>
> On Sat, Feb 12, 2011 at 6:06 AM, Viral Bajaria <viral.bajaria@gmail.com>wrote:
>
>> We use hive through the thrift api too. I have seen the above error when
>> someone accidentally connects to the thrift server port through multiple
>> instances of our hive-wrapper code. We open a bunch of hive thrift ports to
>> achieve the automation of running multiple hive queries in parallel i.e.
>> continuous aggregations.
>>
>> you need to make sure you are not facing a similar issue because the hive
>> thrift server as far as i know is single threaded up until hive 0.5.0 (the
>> version that i use). Not too sure if that's been changed in 0.6.0 or higher.
>>
>> -Viral
>>
>> On Fri, Feb 11, 2011 at 7:06 AM, Ajo Fod <ajo.fod@gmail.com> wrote:
>>
>>> Are you using hive 0.6? ... may be fixed in the latest version.
>>>
>>> Also I wonder why these thrift libraries are being used ... is this
>>> normal hive operation, or can you do something to avoid using thrift?
>>>
>>> -Ajo
>>>
>>>
>>> On Fri, Feb 11, 2011 at 12:05 AM, vaibhav negi <sssssssenator@gmail.com>wrote:
>>>
>>>>
>>>> Hi all,
>>>>
>>>> I am loading data into hive tables by connecting to hiveserver through
>>>> thrift api using "load data local inpath ... " query .
>>>> Hive server is running as a background process for days . After some 4-5
>>>> days hiveserver throws this error :
>>>>
>>>> 11/02/11 11:20:02 WARN server.TThreadPoolServer: Transport error
>>>> occurred during acceptance of message.
>>>> org.apache.thrift.transport.TTransportException:
>>>> java.net.SocketException: Too many open files
>>>>         at
>>>> org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:124)
>>>>         at
>>>> org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:35)
>>>>         at
>>>> org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31)
>>>>         at
>>>> org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:184)
>>>>         at
>>>> org.apache.hadoop.hive.service.HiveServer.main(HiveServer.java:388)
>>>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>         at
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>         at
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>         at java.lang.reflect.Method.invoke(Method.java:597)
>>>>         at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>>>> Caused by: java.net.SocketException: Too many open files
>>>>         at java.net.PlainSocketImpl.socketAccept(Native Method)
>>>>         at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
>>>>         at java.net.ServerSocket.implAccept(ServerSocket.java:453)
>>>>         at java.net.ServerSocket.accept(ServerSocket.java:421)
>>>>         at
>>>> org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:119)
>>>>         ... 9 more
>>>>
>>>> I checked network connections using netstat and it show about 1000
>>>> active network connections, most of them to hadoop datanode . Output is
>>>> attached as netstat.out
>>>>
>>>> after i restarted hiveserver network connections falls to about 20 .
>>>> output is attached as netstat_after_HS_restart.out
>>>>
>>>> Is this a problem with hive or hadoop configuration ?
>>>>
>>>> Vaibhav Negi
>>>>
>>>
>>>
>>
>

Mime
View raw message