hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stephen Bovy (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-5541) LIBHDFS questions and performance suggestions
Date Tue, 26 Nov 2013 01:50:35 GMT

    [ https://issues.apache.org/jira/browse/HDFS-5541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13832179#comment-13832179
] 

Stephen Bovy commented on HDFS-5541:
------------------------------------

Here are the traces from  the   OPS  test  ( almost 100% )  

I am getting a strange error  on writing a file in append mode  

>>  And is  getStatus  a new method ??? <<   

Some new methods are  missing ( thus cannot be used for backwards comaptibility )

Usage: hadoop [--config confdir] COMMAND
where COMMAND is one of:
  namenode -format     format the DFS filesystem
  secondarynamenode    run the DFS secondary namenode
  namenode             run the DFS namenode
  datanode             run a DFS datanode
  dfsadmin             run a DFS admin client
  mradmin              run a Map-Reduce admin client
  fsck                 run a DFS filesystem checking utility
  fs                   run a generic filesystem user client
  balancer             run a cluster balancing utility
  snapshotDiff         diff two snapshots of a directory or diff the
                       current directory contents with a snapshot
  lsSnapshottableDir   list all snapshottable dirs owned by the current user
  oiv                  apply the offline fsimage viewer to an fsimage
  fetchdt              fetch a delegation token from the NameNode
  jobtracker           run the MapReduce job Tracker node
  pipes                run a Pipes job
  tasktracker          run a MapReduce task Tracker node
  historyserver        run job history servers as a standalone daemon
  job                  manipulate MapReduce jobs
  queue                get information regarding JobQueues
  version              print the version
  jar <jar>            run a jar file

  distcp <srcurl> <desturl> copy file or directories recursively
  distcp2 <srcurl> <desturl> DistCp version 2
  archive -archiveName NAME <src>* <dest> create a hadoop archive
  daemonlog            get/set the log level for each daemon
 or
  CLASSNAME            run the class named CLASSNAME
Most commands print help when invoked w/o parameters.

C:\hdp\hadoop\hadoop-1.2.0.1.3.0.0-0380>z:

Z:\>cd d\dclibhdfs

Z:\D\dclibhdfs>dir
 Volume in drive Z is Shared Folders
 Volume Serial Number is 0000-0064

 Directory of Z:\D\dclibhdfs

11/25/2013  05:33 PM    <DIR>          .
11/25/2013  04:29 PM            56,832 dclibhdfs.dll
11/25/2013  05:33 PM            17,408 TstOpsHdfs.exe
11/22/2013  09:28 PM             7,680 TstReadHdfs.exe
11/22/2013  09:29 PM             7,680 TstWriteHdfs.exe
09/09/2013  03:14 PM         4,961,800 vc2008_SP1_redist_x64.exe
09/09/2013  03:16 PM         1,821,192 vc2008_SP1_redist_x86.exe
09/09/2013  03:02 PM         7,185,000 vc2012_Update3_redist_x64.exe
09/09/2013  03:02 PM         6,552,288 vc2012_Update3_redist_x86.exe
               8 File(s)     20,613,976 bytes
               1 Dir(s)  272,641,044,480 bytes free

Z:\D\dclibhdfs>TstOpsHdfs.exe
dll attached
dll: tls1=1
Get Global JNI
load jvm
dll: get proc addresses
dll: thread attach
dll: thread attach
dll: thread attach
dll: thread attach
dll: thread attach
dll: thread attach
dll: thread attach
dll: thread attach
dll: thread attach
dll: thread attach
dll: jvm created
dll: thread attach
dll: save environment
dll: thread attach
dll: thread attach
dll: thread attach
dll: thread attach
dll: thread attach
Opened /tmp/testfile.txt for writing successfully...
dll: thread attach
Wrote 14 bytes
Current position: 14
Flushed /tmp/testfile.txt successfully!
dll: thread attach
dll: detach thread
dll: detach thread
could not find method read from class org/apache/hadoop/fs/FSDataInputStream wit
h signature (Ljava/nio/ByteBuffer;)I
readDirect: FSDataInputStream#read error:
java.lang.NoSuchMethodError: read
hdfsOpenFile(/tmp/testfile.txt): WARN: Unexpected error 255 when testing for dir
ect read compatibility
hdfsAvailable: 14
Current position: 1
Direct read support not detected for HDFS filesystem
Read following 13 bytes:
ello, World!
Read following 14 bytes:
Hello, World!
Test Local File System C:\tmp\testfile.txt
13/11/25 17:40:41 WARN util.NativeCodeLoader: Unable to load native-hadoop libra
ry for your platform... using builtin-java classes where applicable
dll: thread attach
dll: detach thread
could not find method read from class org/apache/hadoop/fs/FSDataInputStream wit
h signature (Ljava/nio/ByteBuffer;)I
readDirect: FSDataInputStream#read error:
java.lang.NoSuchMethodError: read
hdfsOpenFile(C:\tmp\testfile.txt): WARN: Unexpected error 255 when testing for d
irect read compatibility
dll: thread attach
dll: detach thread
hdfsCopy(remote-local): Success!
dll: thread attach
dll: thread attach
dll: detach thread
dll: detach thread
hdfsCopy(remote-remote): Success!
dll: thread attach
dll: detach thread
hdfsMove(local-local): Success!
dll: thread attach
dll: detach thread
hdfsMove(remote-local): Success!
hdfsRename: Success!
dll: thread attach
dll: thread attach
dll: detach thread
dll: detach thread
hdfsCopy(remote-remote): Success!
hdfsCreateDirectory: Success!
hdfsSetReplication: Success!
hdfsGetWorkingDirectory: hdfs://bovy2008.td.teradata.com:8020/user/sb186007
hdfsSetWorkingDirectory: Success!
hdfsGetWorkingDirectory: /tmp
hdfsGetDefaultBlockSize: 67108864
could not find method getStatus from class org/apache/hadoop/fs/FileSystem with
signature ()Lorg/apache/hadoop/fs/FsStatus;
hdfsGetCapacity: FileSystem#getStatus error:
java.lang.NoSuchMethodError: getStatus
hdfsGetCapacity: -1
could not find method getStatus from class org/apache/hadoop/fs/FileSystem with
signature ()Lorg/apache/hadoop/fs/FsStatus;
hdfsGetUsed: FileSystem#getStatus error:
java.lang.NoSuchMethodError: getStatus
hdfsGetUsed: -1
hdfsGetPathInfo - SUCCESS!
Name: hdfs://bovy2008.td.teradata.com:8020/tmp, Type: D, Replication: 0, BlockSi
ze: 0, Size: 0, LastMod: Mon Nov 25 17:40:41 2013
Owner: sb186007, Group: supergroup, Permissions: 511 (rwxrwxrwx)
Name: hdfs://bovy2008.td.teradata.com:8020/tmp/appends, Type: F, Replication: 1,
 BlockSize: 67108864, Size: 6, LastMod: Mon Nov 25 17:35:37 2013
Owner: sb186007, Group: supergroup, Permissions: 420 (rw-r--r--)
Name: hdfs://bovy2008.td.teradata.com:8020/tmp/newdir, Type: D, Replication: 0,
BlockSize: 0, Size: 0, LastMod: Mon Nov 25 17:40:41 2013
Owner: sb186007, Group: supergroup, Permissions: 493 (rwxr-xr-x)
Name: hdfs://bovy2008.td.teradata.com:8020/tmp/testfile.txt, Type: F, Replicatio
n: 2, BlockSize: 67108864, Size: 14, LastMod: Mon Nov 25 17:40:41 2013
Owner: sb186007, Group: supergroup, Permissions: 420 (rw-r--r--)
Name: hdfs://bovy2008.td.teradata.com:8020/tmp/testfile2.txt, Type: F, Replicati
on: 1, BlockSize: 67108864, Size: 14, LastMod: Mon Nov 25 17:40:41 2013
Owner: sb186007, Group: supergroup, Permissions: 420 (rw-r--r--)
Name: hdfs://bovy2008.td.teradata.com:8020/tmp/usertestfile.txt, Type: F, Replic
ation: 1, BlockSize: 67108864, Size: 14, LastMod: Mon Nov 25 17:35:37 2013
Owner: nobody, Group: supergroup, Permissions: 420 (rw-r--r--)
hdfsGetHosts - SUCCESS! ...
        hosts[0][0] - bovy2008.td.teradata.com
hdfsChown:Group: Success!
hdfsChown:User: Success!
hdfsChmod: Success!
hdfsUtime: Success!
hdfsChown read: Success!
hdfsChmod read: Success!
hdfsChmod: Success!
newMTime=1385430041
curMTime=1385430041
hdfsUtime read (mtime): Success!
hdfsDelete: Success!
hdfsDelete: Success!
hdfsDelete: Success!
hdfsDelete: Success!
hdfsExists: Success!
Opened /tmp/appends for writing successfully...
dll: thread attach
Wrote 6 bytes
Flushed /tmp/appends successfully!
dll: thread attach
dll: detach thread
dll: detach thread
Open file in append mode:/tmp/appends
hdfsOpenFile(/tmp/appends): FileSystem#append((Lorg/apache/hadoop/fs/Path;)Lorg/
apache/hadoop/fs/FSDataOutputStream;) error:
org.apache.hadoop.ipc.RemoteException: java.io.IOException: Append is not suppor
ted. Please see the dfs.support.append configuration parameter
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.appendFile(FSName
system.java:1840)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.append(NameNode.java:
768)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1456)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1452)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInforma
tion.java:1233)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1450)

        at org.apache.hadoop.ipc.Client.call(Client.java:1118)
        at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
        at $Proxy1.append(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryI
nvocationHandler.java:85)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocat
ionHandler.java:62)
        at $Proxy1.append(Unknown Source)
        at org.apache.hadoop.hdfs.DFSClient.append(DFSClient.java:993)
        at org.apache.hadoop.hdfs.DFSClient.append(DFSClient.java:982)
        at org.apache.hadoop.hdfs.DistributedFileSystem.append(DistributedFileSy
stem.java:199)
        at org.apache.hadoop.fs.FileSystem.append(FileSystem.java:650)
Failed to open /tmp/appends for Append/writing!
dll: detach thread
Connect As User:nobody
dll: thread attach
dll: thread attach
Opened /tmp/usertestfile.txt for writing successfully...
dll: thread attach
Wrote 14 bytes
Flushed /tmp/usertestfile.txt successfully!
dll: thread attach
dll: detach thread
dll: detach thread
hdfs new file user is correct: Success!
dll: detach thread
dll: detach process
dll: invoke thread destructor
dll detached

Z:\D\dclibhdfs> 

And How do I get the NativeCodeLoader to work ??

13/11/25 17:40:41 WARN util.NativeCodeLoader: Unable to load native-hadoop libra
ry for your platform... using builtin-java classes where applicable



> LIBHDFS questions and performance suggestions
> ---------------------------------------------
>
>                 Key: HDFS-5541
>                 URL: https://issues.apache.org/jira/browse/HDFS-5541
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: hdfs-client
>            Reporter: Stephen Bovy
>            Priority: Minor
>         Attachments: pdclibhdfs.zip
>
>
> Since libhdfs is a "client" interface",  and esspecially because it is a "C" interface
, it should be assumed that the code will be used accross many different platforms, and many
different compilers.
> 1) The code should be cross platform ( no Linux extras )
> 2) The code should compile on standard c89 compilers, the
> >>>  {least common denominator rule applies here} !! <<  
> C  code with  "c"   extension should follow the rules of the c standard  
> All variables must be declared at the begining of scope , and no (//) comments allowed

> >> I just spent a week white-washing the code back to nornal C standards so that
it could compile and build accross a wide range of platforms << 
> Now on-to  performance questions 
> 1) If threads are not used why do a thread attach ( when threads are not used all the
thread attach nonesense is a waste of time and a performance killer ) 
> 2) The JVM  init  code should not be imbedded within the context of every function call
  .  The  JVM init code should be in a stand-alone  LIBINIT function that is only invoked
once.   The JVM * and the JNI * should be global variables for use when no threads are utilized.
 
> 3) When threads are utilized the attach fucntion can use the GLOBAL  jvm * created by
the LIBINIT  { WHICH IS INVOKED ONLY ONCE } and thus safely outside the scope of any LOOP
that is using the functions 
> 4) Hash Table and Locking  Why ?????
> When threads are used the hash table locking is going to hurt perfromance .  Why not
use thread local storage for the hash table,that way no locking is required either with or
without threads.   
>  
> 5) FINALLY Windows  Compatibility 
> Do not use posix features if they cannot easilly be replaced on other platforms   !!



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message