hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aggarwal, Vaibhav" <vagg...@amazon.com>
Subject RE: Hive session locking up after 4 queries using S3
Date Wed, 06 Jul 2011 17:39:57 GMT
Could you please tell us which Hadoop and Hive version are you using?
Looks like you might be using an older version of Hadoop (more specifically one which ships
with old version of jets3t).

Thanks
Vaibhav

From: Wouter de Bie [mailto:wouter@spotify.com]
Sent: Wednesday, July 06, 2011 9:07 AM
To: user@hive.apache.org
Subject: Hive session locking up after 4 queries using S3

Hi all,

I'm using Hive with the s3native FS. Today, I noticed that hive locks up after 4 queries that
directly access S3 (select * from mytable limit 10). With debug logging on, I get the following
output:

2011-07-06 15:54:31,459 DEBUG s3native.NativeS3FileSystem (NativeS3FileSystem.java:getFileStatus(393))
- getFileStatus retrieving metadata for key 'tmp/hive-mapred/hive_2011-07-06_15-54-29_881_4253697128840334916/-mr-10000'
2011-07-06 15:54:31,459 DEBUG httpclient.RestS3Service (RestS3Service.java:getObjectImpl(1511))
- Retrieving Head information for bucket XXXXXXXX and object tmp/hive-mapred/hive_2011-07-06_15-54-29_881_4253697128840334916/-mr-10000
2011-07-06 15:54:31,460 DEBUG service.Jets3tProperties (Jets3tProperties.java:getBoolProperty(314))
- s3service.disable-dns-buckets=false
2011-07-06 15:54:31,460 DEBUG httpclient.RestS3Service (RestS3Service.java:setupConnection(811))
- S3 URL: https://XXXXXXXX:443/tmp%2Fhive-mapred%2Fhive_2011-07-06_15-54-29_881_4253697128840334916%2F-mr-10000
2011-07-06 15:54:31,460 DEBUG httpclient.RestS3Service (RestS3Service.java:performRequest(334))
- Performing HEAD request for 'https://XXXXXXXXXXXXXX/tmp%2Fhive-mapred%2Fhive_2011-07-06_15-54-29_881_4253697128840334916%2F-mr-10000',
expecting response code 200
2011-07-06 15:54:31,461 DEBUG httpclient.RestS3Service (RestS3Service.java:buildAuthorizationString(872))
- Adding authorization for AWS Access Key 'XXXXXXXXXXXXX'.
2011-07-06 15:54:31,461 DEBUG httpclient.RestS3Service (RestS3Service.java:buildAuthorizationString(922))
- Canonical string ('|' is a newline): HEAD|||Wed, 06 Jul 2011 15:54:31 GMT|/XXXXXXXXX/tmp%2Fhive-mapred%2Fhive_2011-07-06_15-54-29_881_4253697128840334916%2F-mr-10000
2011-07-06 15:54:31,461 DEBUG httpclient.HttpClient (HttpClient.java:executeMethod(322)) -
enter HttpClient.executeMethod(HttpMethod)
2011-07-06 15:54:31,462 DEBUG httpclient.HttpClient (HttpClient.java:executeMethod(373)) -
enter HttpClient.executeMethod(HostConfiguration,HttpMethod,HttpState)
2011-07-06 15:54:31,462 DEBUG httpclient.MultiThreadedHttpConnectionManager (MultiThreadedHttpConnectionManager.java:getConnectionWithTimeout(383))
- enter HttpConnectionManager.getConnectionWithTimeout(HostConfiguration, long)
2011-07-06 15:54:31,462 DEBUG httpclient.MultiThreadedHttpConnectionManager (MultiThreadedHttpConnectionManager.java:getConnectionWithTimeout(390))
- HttpConnectionManager.getConnection:  config = HostConfiguration[host=https://XXXXXXXXX.s3.amazonaws.com],
timeout = 0
2011-07-06 15:54:31,462 DEBUG httpclient.MultiThreadedHttpConnectionManager (MultiThreadedHttpConnectionManager.java:getHostPool(775))
- enter HttpConnectionManager.ConnectionPool.getHostPool(HostConfiguration)
2011-07-06 15:54:31,463 DEBUG httpclient.MultiThreadedHttpConnectionManager (MultiThreadedHttpConnectionManager.java:doGetConnection(494))
- Unable to get a connection, waiting..., hostConfig=HostConfiguration[host=https://XXXXXXXXXX.s3.amazonaws.com]

Does anyone know if I can do anything to prevent this? It looks like connections are not returned
correctly to the pool..

// Wouter

Mime
View raw message