hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HADOOP-12753) S3A jUnit tests failing if using HTTP proxy
Date Fri, 05 Feb 2016 21:39:39 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-12753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Steve Loughran updated HADOOP-12753:
------------------------------------
    Status: Patch Available  (was: Open)

> S3A jUnit tests failing if using HTTP proxy
> -------------------------------------------
>
>                 Key: HADOOP-12753
>                 URL: https://issues.apache.org/jira/browse/HADOOP-12753
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 2.7.2
>         Environment: Test hardware:
> * issue identified while running jUnit tests on MacBook Pro
> PREREQUISITE:
> * have a functional HTTP proxy running (ie. SQUID HTTP proxy @ http://127.0.0.1:3128)
> * S3A test configuration (relevant options):
>     <!-- NOTE: by not specifying endpoint, we'll use the default AWS S3 -->
>     <property>
> 	<name>fs.s3a.endpoint</name>
> 	<value></value>
>     </property>
>     <!-- NOTE: please configure functional HTTP proxy with fs.s3a.proxy.{host,port}
-->
>     <property>
> 	<name>fs.s3a.proxy.host</name>
> 	<value>127.0.0.1</value>
>     </property>
>     <property>
> 	<name>fs.s3a.proxy.port</name>
> 	<value>3128</value>
>     </property>
>     <property>
> 	<name>fs.s3a.access.key</name>
> 	<value>_REAL_AWS_S3_ACCESS_KEY</value>
>     </property>
>     <property>
> 	<name>fs.s3a.secret.key</name>
> 	<value>_REAL_AWS_S3_SECRET_KEY</value>
>     </property>
>            Reporter: Zoran Rajic
>            Priority: Minor
>         Attachments: HADOOP-12753-s3a-jUnits.diff
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> Some companies have restricted firewalls and don't allow the "direct access" to Web-pages
or Internet, so their employees have to use the SOCKS or the HTTP proxy.
> This also means that the engineers who develop and test the Hadoop code and s3a:// -filesystem
inside the company firewalls, need to access the "external" Amazon S3 service via the HTTP
proxy.
> Unfortunately, there are S3A jUnit test failures when run with the HTTP proxy (see Environment
field above).
> --
> Details:
> * steps to reproduce:
>     cd hadoop_git/hadoop-tools/hadoop-aws
>     mvn clean test -Dtest=TestS3AConfiguration
>     [...]
>     Running org.apache.hadoop.fs.s3a.TestS3AConfiguration
>     Tests run: 5, Failures: 1, Errors: 1, Skipped: 0, Time elapsed: 2.737 sec <<<
FAILURE! - in org.apache.hadoop.fs.s3a.TestS3AConfiguration
>     TestAutomaticProxyPortSelection(org.apache.hadoop.fs.s3a.TestS3AConfiguration)  Time
elapsed: 0.737 sec  <<< FAILURE!
>     java.lang.AssertionError: Expected a connection error for proxy server
> 	    at org.junit.Assert.fail(Assert.java:88)
> 	    at org.apache.hadoop.fs.s3a.TestS3AConfiguration.TestAutomaticProxyPortSelection(TestS3AConfiguration.java:130)
>     TestProxyPortWithoutHost(org.apache.hadoop.fs.s3a.TestS3AConfiguration)  Time elapsed:
0.624 sec  <<< ERROR!
>     com.amazonaws.AmazonClientException: Unable to execute HTTP request: Connection to
http://127.0.0.1:1 refused
> 	    at java.net.PlainSocketImpl.socketConnect(Native Method)
> 	    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
> 	    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
> 	    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
> 	    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> 	    at java.net.Socket.connect(Socket.java:589)
> 	    at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
> 	    at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
> 	    at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
> 	    at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:643)
> 	    at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:479)
> 	    at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
> 	    at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
> 	    at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:384)
> 	    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232)
> 	    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528)
> 	    at com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1031)
> 	    at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:994)
> 	    at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:297)
> 	    at org.apache.hadoop.fs.s3a.S3ATestUtils.createTestFileSystem(S3ATestUtils.java:51)
> 	    at org.apache.hadoop.fs.s3a.TestS3AConfiguration.TestProxyPortWithoutHost(TestS3AConfiguration.java:111)
>     Results :
>     Failed tests:
>     TestS3AConfiguration.TestAutomaticProxyPortSelection:130 Expected a connection error
for proxy server
>     Tests in error:
>     TestS3AConfiguration.TestProxyPortWithoutHost:111 ยป AmazonClient Unable to exe...
>     Tests run: 5, Failures: 1, Errors: 1, Skipped: 0



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message