hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Nauroth (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-5541) LIBHDFS questions and performance suggestions
Date Fri, 22 Nov 2013 18:00:39 GMT

    [ https://issues.apache.org/jira/browse/HDFS-5541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13830164#comment-13830164
] 

Chris Nauroth commented on HDFS-5541:
-------------------------------------

Hi, [~stevebovy].  Thank you for reporting this.  I suggest splitting this into 2 separate
jiras: one for cross-platform compatibility and another for performance enhancements.  We
can probably act quickly on compatibility.  The performance questions might require more discussion.
 Splitting out separate patches also helps make code reviews easier.

Regarding compatibility, we've done a lot of similar work on the JNI extensions in hadoop-common
for C89 compatibility.  The work hasn't been done for libhdfs, but I expect it will be similar
(declare variables at the top, no // comments, avoid Linux-specific functions or put them
behind an {{#ifdef}}, etc.).

bq. >> I just spent a week white-washing the code back to nornal C standards so that
it could compile and build accross a wide range of platforms <<

Would you be interested in contributing this as a patch?  We'd definitely appreciate it!

bq. 5) FINALLY Windows Compatibility

Assuming all of the compatibility issues mentioned above are addressed (standard C89 code
+ no Linux extras), is there some additional Windows compatibility problem that you encountered,
or this comment just stating the end goal that you want to achieve?

> LIBHDFS questions and performance suggestions
> ---------------------------------------------
>
>                 Key: HDFS-5541
>                 URL: https://issues.apache.org/jira/browse/HDFS-5541
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: hdfs-client
>            Reporter: Stephen Bovy
>            Priority: Minor
>
> Since libhdfs is a "client" interface",  and esspecially because it is a "C" interface
, it should be assumed that the code will be used accross many different platforms, and many
different compilers.
> 1) The code should be cross platform ( no Linux extras )
> 2) The code should compile on standard c89 compilers, the
> >>>  {least common denominator rule applies here} !! <<  
> C  code with  "c"   extension should follow the rules of the c standard  
> All variables must be declared at the begining of scope , and no (//) comments allowed

> >> I just spent a week white-washing the code back to nornal C standards so that
it could compile and build accross a wide range of platforms << 
> Now on-to  performance questions 
> 1) If threads are not used why do a thread attach ( when threads are not used all the
thread attach nonesense is a waste of time and a performance killer ) 
> 2) The JVM  init  code should not be imbedded within the context of every function call
  .  The  JVM init code should be in a stand-alone  LIBINIT function that is only invoked
once.   The JVM * and the JNI * should be global variables for use when no threads are utilized.
 
> 3) When threads are utilized the attach fucntion can use the GLOBAL  jvm * created by
the LIBINIT  { WHICH IS INVOKED ONLY ONCE } and thus safely outside the scope of any LOOP
that is using the functions 
> 4) Hash Table and Locking  Why ?????
> When threads are used the hash table locking is going to hurt perfromance .  Why not
use thread local storage for the hash table,that way no locking is required either with or
without threads.   
>  
> 5) FINALLY Windows  Compatibility 
> Do not use posix features if they cannot easilly be replaced on other platforms   !!



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message