hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stephen Bovy (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-573) Porting libhdfs to Windows
Date Thu, 21 Nov 2013 03:00:36 GMT

    [ https://issues.apache.org/jira/browse/HDFS-573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13828437#comment-13828437

Stephen Bovy commented on HDFS-573:

Darn,  I just  re-did  all the work you guys did !!

Is their anyone out there still interested in this ??

What hashing fucntions did you use ??

I usesd ut-hash 

Any suggestions would be appreciated 

It is a pain in the but  making  "c"  code look like  C code is supposed to look  !!

I worked off  the 2.0 branch  and unfortunately  more  linux only anachronisisms  were added

Hard-coded linux only  err-no  yikes !!

I have some serious performance questions about  the hashing functions 

Why not use  Thread Local storage for the hashing table  ,  then we can avoid the performance
from locks we do not need to use ???  

The jvm init code should not be  imbedded into every  function call   many of these functions
will be used in loops 

The jvm init  should  be in a stand alone  function that is called only once  to init the

Then we can use  a global variable for the  jvm *  and the fucntion to discover the count
jvm's is not needed .   

The   jvm attach functionallity  should  only be invoked  if threads  are used   otherwise
it is another 
useless perfromance killer   


> Porting libhdfs to Windows
> --------------------------
>                 Key: HDFS-573
>                 URL: https://issues.apache.org/jira/browse/HDFS-573
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: hdfs-client
>         Environment: Windows, Visual Studio 2008
>            Reporter: Ziliang Guo
>   Original Estimate: 336h
>  Remaining Estimate: 336h
> The current C code in libhdfs is written using C99 conventions and also uses a few POSIX
specific functions such as hcreate, hsearch, and pthread mutex locks.  To compile it using
Visual Studio would require a conversion of the code in hdfsJniHelper.c and hdfs.c to C89
and replacement/reimplementation of the POSIX functions.  The code also uses the stdint.h
header, which is not part of the original C89, but there exists what appears to be a BSD licensed
reimplementation written to be compatible with MSVC floating around.  I have already done
the other necessary conversions, as well as created a simplistic hash bucket for use with
hcreate and hsearch and successfully built a DLL of libhdfs.  Further testing is needed to
see if it is usable by other programs to actually access hdfs, which will likely happen in
the next few weeks as the Condor Project continues with its file transfer work.
> In the process, I've removed a few what I believe are extraneous consts and also fixed
an incorrect array initialization where someone was attempting to initialize with something
like this: JavaVMOption options[noArgs]; where noArgs was being incremented in the code above.
 This was in the hdfsJniHelper.c file, in the getJNIEnv function.

This message was sent by Atlassian JIRA

View raw message