hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Patrick McCabe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-10734) Implementation of true secure random with high performance using hardware random number generator.
Date Thu, 03 Jul 2014 23:43:34 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-10734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14052008#comment-14052008
] 

Colin Patrick McCabe commented on HADOOP-10734:
-----------------------------------------------

bq. \[discussion of randomness methods\]

I don't think {{/dev/random}} is a practical choice, due to the blocking issue.  From what
I've heard, {{/dev/urandom}} can often be a good choice.  I'm tempted to try a super-simple
piece of code that just periodically fills a big buffer from {{/dev/urandom}} and see if that
performs well.  I have a hunch that it would (and it would also use RDRAND on supported platforms.)
 But I think it's fine if you want to provide an option to go through openssl as well... we
already have a dependency on that library.

bq. And we can also add enable flag in configuration and user can disable it.

I agree.  I think we should have a configuration option like {{encryption.random.number.generator}}
which specifies a comma-separated list of class names to try to use.  That way a user could
specify the openssl one plus a fallback to the standard java one if they so chose.  Or alternately,
the user could enable just the java one (and configure it to use /dev/urandom) to get something
which used RDRAND plus some additional randomness.  If you want to do this in a follow-on
JIRA, that's OK too.

I think it's confusing that we have both {{org.apache.hadoop.crypto.random.SecureRandom}}
and {{java.security.SecureRandom}}.  Maybe a better name for this new class would be {{OpenSslSecureRandom}}
or something like that, to emphasize that it is using OpenSSL to get random bits.

{code}
+/**
+ * Utilize RdRand to return random numbers from hardware random number 
+ * generator. It's TRNG(True Random Number generators) having high performance. 
+ * https://wiki.openssl.org/index.php/Random_Numbers#Hardware
+ * http://en.wikipedia.org/wiki/RdRand
+ */
+static ENGINE * rdrand_init(JNIEnv *env)
{code}

I think the comment is a bit misleading here.  Openssl compiles on a lot of platforms that
don't have RDRAND.  So all we really know here is that we're using openssl, not that we're
using RDRAND.  I think it's appropriate to have a comment saying, "if you are using an Intel
chipset with RDRAND, the high-performance random number generator will be used", or something
like that.  But it's platform specific and we may be compiling on another platform.

{code}
+  \@Test(timeout=120000)
+  public void testRandomInt() throws Exception {
+    SecureRandom random = new SecureRandom();
+    
+    int rand1 = random.nextInt();
+    int rand2 = random.nextInt();
+    Assert.assertFalse(rand1 == rand2);
+  }
{code}

It's definitely difficult to test something which is returning true random numbers.  It requires
a lot of mathematics.  So I see why you did it this way.  Just one comment... maybe I'm being
overly paranoid here, but can we loop until rand2 is not equal to rand1?

bq. I suppose you mean direct ByteBuffer. Per my understanding, merit of direct ByteBuffer
is to avoid bytes copy. But SecureRandom#nextBytes will accept an pre-allocated byte[] array,
if we use direct ByteBuffer for JNI, then there is additional copy in java layer, so the performance
is the same, and we need to manage the direct ByteBuffer.

OK.

bq. Can you explain a bit more, I’m not sure I get your meaning. Per my understanding, pthread_t
is defined in /usr/include/bits/pthreadtypes.h as

The stuff in {{/usr/include/bits}} is not public; it is an implementation detail that could
change at any time.

from {{man pthread_self}}:
bq. POSIX.1 allows an implementation wide freedom in choosing the type used to represent a
thread ID; for example, representation using either an arithmetic  type  or a structure is
permitted.  Therefore, variables of type pthread_t can't portably be compared using the C
equality operator (==); use pthread_equal(3) instead.  Thread identifiers should be considered
opaque: any attempt to use a thread ID other than in pthreads calls is nonportable and can
lead to  unspecified results.

> Implementation of true secure random with high performance using hardware random number
generator.
> --------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-10734
>                 URL: https://issues.apache.org/jira/browse/HADOOP-10734
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: security
>    Affects Versions: fs-encryption (HADOOP-10150 and HDFS-6134)
>            Reporter: Yi Liu
>            Assignee: Yi Liu
>             Fix For: fs-encryption (HADOOP-10150 and HDFS-6134)
>
>         Attachments: HADOOP-10734.patch
>
>
> This JIRA is to implement Secure random using JNI to OpenSSL, and implementation should
be thread-safe.
> Utilize RdRand to return random numbers from hardware random number generator. It's TRNG(True
Random Number generators) having much higher performance than {{java.security.SecureRandom}}.

> https://wiki.openssl.org/index.php/Random_Numbers
> http://en.wikipedia.org/wiki/RdRand
> https://software.intel.com/en-us/articles/performance-impact-of-intel-secure-key-on-openssl



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message