hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yi Liu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-10734) Implementation of true secure random with high performance using hardware random number generator.
Date Thu, 03 Jul 2014 14:26:27 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-10734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14051500#comment-14051500

Yi Liu commented on HADOOP-10734:

[~cmccabe], thanks for the review :-). 

I actually have the same problem with the scheme here: JNI calls are expensive... do we know
how many random bits the API user is getting at a time? If that number is small, we might
want to implement batching. 
In most cases, we use it to generate key(16bytes, 32bytes, 128bytes, 256bytes), IV(16 bytes),
long (8 bytes).
Furthermore, to make the random bytes good enough, we can’t avoid JNI, even {{java.security.SecureRandom}}
also uses JNI.

I also think we should consider using ByteBuffer rather than byte[] array, if performance
is the primary goal.
I suppose you mean direct ByteBuffer. Per my understanding, merit of direct ByteBuffer is
to avoid bytes copy.  But {{SecureRandom#nextBytes}} will accept an pre-allocated byte[] array,
if we use direct ByteBuffer for JNI, then there is additional copy in java layer, so the performance
is the same, and we need to manage the direct ByteBuffer.

+  final protected int next(int numBits) {
Should be private
OK, I will update it.

+  public long nextLong() {
+    return ((long)(next(32)) << 32) + next(32);
+  }
Why use addition rather than bitwise OR here?
Bitwise OR is also OK. Actually {{nextLong}}, {{nextFloat}} and {{nextDouble}} are copied
from implementations in {{java.security.SecureRandom}}

This is not correct. The type of {{pthread_t}} is not known. If you want a numeric thread
ID, you could try gettid on Linux.
Can you explain a bit more, I’m not sure I get your meaning. Per my understanding, {{pthread_t}}
is defined in {{/usr/include/bits/pthreadtypes.h}} as
typedef unsigned long int pthread_t;
And this patch is compiled and run successfully on my Linux server.

> Implementation of true secure random with high performance using hardware random number
> --------------------------------------------------------------------------------------------------
>                 Key: HADOOP-10734
>                 URL: https://issues.apache.org/jira/browse/HADOOP-10734
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: security
>    Affects Versions: fs-encryption (HADOOP-10150 and HDFS-6134)
>            Reporter: Yi Liu
>            Assignee: Yi Liu
>             Fix For: fs-encryption (HADOOP-10150 and HDFS-6134)
>         Attachments: HADOOP-10734.patch
> This JIRA is to implement Secure random using JNI to OpenSSL, and implementation should
be thread-safe.
> Utilize RdRand to return random numbers from hardware random number generator. It's TRNG(True
Random Number generators) having much higher performance than {{java.security.SecureRandom}}.

> https://wiki.openssl.org/index.php/Random_Numbers
> http://en.wikipedia.org/wiki/RdRand
> https://software.intel.com/en-us/articles/performance-impact-of-intel-secure-key-on-openssl

This message was sent by Atlassian JIRA

View raw message