hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun C Murthy (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-538) Implement a nio's 'direct buffer' based wrapper over zlib to improve performance of java.util.zip.{De|In}flater as a 'custom codec'
Date Wed, 08 Nov 2006 04:38:53 GMT
    [ http://issues.apache.org/jira/browse/HADOOP-538?page=comments#action_12448027 ] 
Arun C Murthy commented on HADOOP-538:

Wrt ZlibFactory, I agree it looks a little ugly for now - and above direction.

However I feel we can do sligthly better by getting ZlibFactory to return the 'Class' instead
of the instance i.e.
public Class getZlibCompressorClass() 
public Class getZlibDecompressorClass() 

(This of course is Owen's original suggestion...)

This will require that ReflectionUtils has a new method:

    public static Object newInstance(Class theClass, Object[] arguments, 
        Configuration conf) {
        // Constructor signature
        Class[] constructorSignature = new Class[arguments.length];
        for (int i=0; i < arguments.length; ++i) {
          constructorSignature[i] = arguments[i].getClass();
        Object result;
        try {
            Constructor meth = theClass.getConstructor(constructorSignature);
            result = meth.newInstance(arguments);
        } catch (Exception e) {
            throw new RuntimeException(e);
        if (conf != null) {
            if (result instanceof Configurable) {
                ((Configurable) result).setConf(conf);
            if (conf instanceof JobConf && 
                    result instanceof JobConfigurable) {
                ((JobConfigurable)result).configure((JobConf) conf);
        return result;

which can then be used to call constructors with arguments (ala http://java.sun.com/docs/books/tutorial/reflect/object/arg.html).

This is necessary since I will need to call Zlib{Com|Decom}pressor with arguments to make
it 'behave' like a Gzip {com|decom}pressor. 

The api can probably be useful elsewhere too...


> Implement a nio's 'direct buffer' based wrapper over zlib to improve performance of java.util.zip.{De|In}flater
as a 'custom codec'
> -----------------------------------------------------------------------------------------------------------------------------------
>                 Key: HADOOP-538
>                 URL: http://issues.apache.org/jira/browse/HADOOP-538
>             Project: Hadoop
>          Issue Type: Improvement
>    Affects Versions: 0.6.1
>            Reporter: Arun C Murthy
>         Assigned To: Arun C Murthy
>             Fix For: 0.9.0
>         Attachments: HADOOP-538.patch, HADOOP-538_20061005.tgz, HADOOP-538_20061011.tgz,
HADOOP-538_20061026.tgz, HADOOP-538_20061030.tgz, HADOOP-538_20061107.tgz, HADOOP-538_benchmarks.tgz
> There has been more than one instance where java.util.zip's {De|In}flater classes perform
unreliably, a simple wrapper over zlib-1.2.3 (latest stable) using java.nio.ByteBuffer (i.e.
direct buffers) should go a long way in alleviating these woes.

This message is automatically generated by JIRA.
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message