hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jagane Sundar (Updated) (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HADOOP-8136) Enhance hadoop to use a newer version (0.8.1) of the jets3t library
Date Tue, 06 Mar 2012 05:33:02 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-8136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jagane Sundar updated HADOOP-8136:
----------------------------------

    Target Version/s: 1.0.2
              Status: Patch Available  (was: Open)

Here's my first cut at a patch for this jira. Some notes:
1. Most of the changes are on account of the jets3t-0.8.1 library throwing a ServiceException,
instead of a S3ServiceException.
2. There is also a piece of brittle code that was comparing an exception string with 'ResponseCode=404'
to decide whether the S3 server threw a 404 error. I replaced that with a more rational test
of response code.
3. I turned down the logging by setting
org.apache.hadoop.fs.s3native=ERROR
org.jets3t.service.impl.rest.httpclient.RestStorageService=ERROR
in log4j.properties
                
> Enhance hadoop to use a newer version (0.8.1) of the jets3t library
> -------------------------------------------------------------------
>
>                 Key: HADOOP-8136
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8136
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs/s3
>    Affects Versions: 1.0.0, 0.22.0, 0.23.3
>         Environment: Ubuntu 11.04, 64 bit, JDK 1.6.0_30
>            Reporter: Jagane Sundar
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> Hadoop is built against, and includes, an older version of the Jets3t library - version
0.6.1.
> The current version of the Jets3t library(as of March 2012) is 0.8.1. This new version
includes many improvements such as support for "Requester-Pays" buckets.
> Since hadoop includes a copy of the version 0.6.1 jets3t library, and since this version
ends up early in the CLASSPATH, any Map Reduce application that wants to use the jets3t library
ends up getting version 0.6.1 of the jets3t library. The MR application fails, usually with
an error stating that the method signature of some method in the Jets3t library does not match.
> It would be useful to enhance Jets3tNativeFileSystemStore.java to use the API published
by the 0.8.1 version of the jets3t library.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message