nifi-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (NIFI-1257) Provide additional KDFs for EncryptContent
Date Tue, 02 Feb 2016 01:41:40 GMT

    [ https://issues.apache.org/jira/browse/NIFI-1257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15127422#comment-15127422
] 

ASF GitHub Bot commented on NIFI-1257:
--------------------------------------

GitHub user alopresto opened a pull request:

    https://github.com/apache/nifi/pull/201

    NIFI-1257 and 1259

    NIFI-1257
    NIFI-1259
    
    Added the mechanics for additional strong `Key Derivation Functions` (KDF) including `PBKDF2`,
`Bcrypt`, and `Scrypt` as well as `keyed ciphers` (raw key; no KDF). Integrated with `EncryptContent`
processor and added new property and property descriptors. Dependent properties not available
until NIFI-1121 is implemented. 
    
    Explanation of KDFs available here: [Apache NiFi Wiki](https://cwiki.apache.org/confluence/display/NIFI/Key+Derivation+Function+Explanations)

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/alopresto/nifi NIFI-1257-cherrypick

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/nifi/pull/201.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #201
    
----
commit ab566da624914d553188495a906a523ce7cbe209
Author: Andy LoPresto <alopresto.apache@gmail.com>
Date:   2015-12-22T18:12:02Z

    NIFI-1257:
    
    Added first skeleton of KeyDeriver functionality but re-evaluating to handle OpenSSL and
NiFi legacy PBE consistently with modern KDFs.

commit aaef4ec6e5551392fdf441fe2bca5f6a924206ea
Author: Andy LoPresto <alopresto.apache@gmail.com>
Date:   2015-12-22T19:02:27Z

    NIFI-1257 Skeleton of key deriver mechanics and moving existing classes into crypto package.
    
    Added variable substitution with defaults to logback-test.xml. (+11 squashed commits)
    Squashed commits:
    [5aa2ae2] NIFI-1257:
    
    Minor formatting changes in logback-test.xml.
    [af12eb5] NIFI-1257:
    
    Added CipherFactory.
    [3e04f5b] NIFI-1257:
    
    Improved Javadoc in NiFi legacy cipher provider and OpenSSL cipher provider.
    [bfdc1d7] NIFI-1257:
    
    Added PBKDF2 cipher provider implementation and Java & Groovy unit test skeletons.
    [20adeef] NIFI-1257:
    
    Added KeyedCipherProvider interface.
    [302a864] NIFI-1257:
    
    Moved OpenPGP encryptor unit tests to proper package.
    [ce5c231] NIFI-1257:
    
    Added test logback file with variable substitution expressions.
    [f2475eb] NIFI-1257:
    
    Added Groovy unit test support to allow faster development cycle.
    [4004cfd] NIFI-1257:
    
    Added OpenSSL PKCS#5 v1.5 EVP_BytesToKey cipher provider and unit test.
    [332b9dc] NIFI-1257:
    
    Moved crypto classes into new package.
    Began refactor from KeyDeriver to CipherProvider.
    [5277efc] NIFI-1257:
    
    Added variables with default value to logback-test.xml.

commit 07c1e1219153ee3ab5a030f68f0647ad41bdd3b9
Author: Andy LoPresto <alopresto.apache@gmail.com>
Date:   2016-01-13T20:25:34Z

    NIFI-1257 Committed state before refactor to inherit from shared CipherProvider interface.
(+5 squashed commits)
    Squashed commits:
    [634802c] NIFI-1257 Added new keyed encryption methods and added boolean field for compatibility
with new KDFs.
    [b74d23c] NIFI-1257 Added parseKeyLengthFromAlgorithm utility method.
    Added unit tests.
    [f3cd3b6] NIFI-1257 Moved cipher algorithm parsing utility method to common util class.
    [b0e785e] NIFI-1257 Applied contrib-check fixes for line length.
    [94a6ecf] NIFI-1257 Added Apache License to various tests.

commit 106de28eeda3838413226185dfe52e0c83bbf04a
Author: Andy LoPresto <alopresto.apache@gmail.com>
Date:   2016-01-14T02:00:34Z

    NIFI-1257 Implemented PBKDF2 cipher provider.
    
    Added default constructor with strong choices for PBKDF2 cipher provider. (+12 squashed
commits)
    Squashed commits:
    [19f94a0] NIFI-1257 Implemented NiFiLegacyCipherProvider and added unit tests.
    [efcb5f6] NIFI-1257 Added convenience methods in OpenSSLPKCS5 provider that ignore desired
key lengths (all OpenSSL algorithms specify key length in algorithm).
    [0381a06] NIFI-1257 Moved BC provider loading in test to single static invocation (decreases
test time by ~40%).
    [b5a8682] NIFI-1257 Added ASF license to openssl_pbkdf2.rb.
    [87cfca0] NIFI-1257 Test cleanup.
    [cb3598b] NIFI-1257 Added key length parameter to PBKDF2 cipher provider.
    [aa50006] NIFI-1257 Added PRF resolution to PBKDF2 cipher provider.
    [9dfb6b0] NIFI-1257 Changed PBKDF2 and OpenSSLPKCS5 implementations to accept EncryptionMethod
instead of individual algorithm and provider combinations.
    [d6156bd] NIFI-1257 Added test resource to generate PBKDF2-derived key and encrypt content
using OpenSSL via Ruby.
    [6cee174] NIFI-1257 Removed duplicated test.
    [5ff87a4] NIFI-1257 Temporarily commented out other implementations while interface changes.
    [2f1ac0c] NIFI-1257 Added RandomIVPBECipherProvider to allow for non-deterministic IVs.

commit 3a835c4c9a8390a41090b7a66e905a3985b39977
Author: Andy LoPresto <alopresto.apache@gmail.com>
Date:   2016-01-17T06:43:25Z

    NIFI-1257 Added Bcrypt implementation.
    Added unit tests.
    Added jBcrypt dependency to pom because BC Bcrypt is OpenBSD-based, not compatible with
standard Bcrypt as defined by the Niels Provos paper.
    Added Ruby script in resources for external compatibility testing.

commit a7c9a4a63facb82d5037c47332230c8b9200ca40
Author: Andy LoPresto <alopresto.apache@gmail.com>
Date:   2016-01-19T03:52:18Z

    NIFI-1257 Added Javadoc, unit tests, and utility methods.
    
    Updated PBKDF2 IV check to invalid length, rather than just non-zero check. (+15 squashed
commits)
    Squashed commits:
    [0c66002] NIFI-1257 Added isKeyedCipher boolean method for EncryptionMethod.
    [5d64dfa] NIFI-1257 Added TODO for invalid IV length check.
    [5c73364] NIFI-1257 Added test for empty PRF in PBKDF2 constructor.
    Added key length check to PBKDF2 cipher provider.
    [57ade68] NIFI-1257 Ignored tests for weak iteration count/work factor but added warning
message in constructors.
    Changed default PRF to SHA-512.
    Added salt and key length check to PBKDF2 cipher provider.
    [362828f] NIFI-1257 Added pass-through of IllegalArgumentExceptions in cipher initialization.
    [9f019ed] NIFI-1257 OpenSSL cipher provider ignores requested key length.
    [6db634b] NIFI-1257 Added check for salt length and encryption method presence.
    [e6426a9] NIFI-1257 Added utility method for algorithm-specific key length check.
    Added unit tests.
    [3503b35] NIFI-1257 Added unit test validating salt format for Bcrypt.
    [9ffd358] NIFI-1257 Implemented key length check in Bcrypt cipher provider.
    Added unit tests.
    [5dc8801] NIFI-1257 Improved Javadoc on CipherUtility.
    [a3eb290] NIFI-1257 Added utility method to check key length validity for cipher families.
    Added unit tests.
    [51ce74f] NIFI-1257 Added unit tests to enforce strong default values for Bcrypt work
factor and PBKDF2 iteration counts.
    [b8dfd4e] NIFI-1257 Corrected license language for jBcrypt in nifi-standard-nar and nifi-assembly
LICENSE files.
    [8a85b33] NIFI-1257 Added license info for jBcrypt into nifi-standard-nar and nifi-assembly
LICENSE files.

commit 28243e9663e53fc8232bff21a65f1bd7654e32e0
Author: Andy LoPresto <alopresto.apache@gmail.com>
Date:   2016-01-21T03:48:45Z

    NIFI-1257 Reduced code duplication by invoking KeyedCipherProvider from BcryptCipherProvider
after key derivation. (+4 squashed commits)
    Squashed commits:
    [1c2f06f] NIFI-1257 Fixed bug in Base64 section of Bcrypt salt regular expression.
    [632104a] NIFI-1257 Added unit test for generateSalt.
    [8cc990f] NIFI-1257 Reduced code duplication by invoking KeyedCipherProvider from PBKDF2CipherProvider
after key derivation.
    Added unit test for invalid IV.
    [75c4690] NIFI-1257 Defined interface for KeyedCipherProvider.
    Implemented AES implementation for KeyedCipherProvider.
    Added unit tests.
    Added Ruby script to test/resources for external compatibility check.

commit 3f0b3ca30423000c50075414d4ca02bb21bc69cd
Author: Andy LoPresto <alopresto.apache@gmail.com>
Date:   2016-01-22T03:35:43Z

    NIFI-1257 Implemented Scrypt cipher provider.
    
    Refactored Scrypt internals and added unit tests.
    
    Added unit test for Scrypt check with correct and incorrect passwords.
    Changed check to remove hard-coded derived key length. (+3 squashed commits)
    Squashed commits:
    [c1d7637] NIFI-1257 Added unit test for Scrypt check with invalid passwords and hashes.
    [f89b2fb] NIFI-1257 Implemented minimum safe parameters calculator and unit test.
    Updated salt format regex.
    [f553bce] NIFI-1257 Moved memory calculation helper to Scrypt. (+2 squashed commits)
    Squashed commits:
    [13c1b52] NIFI-1257 Cleaned up unit tests and javadoc.
    
    Resolved failing PBKDF2 tests and copied multiple run average logic to Bcrypt tests. (+6
squashed commits)
    Squashed commits:
    [b16a5b3] NIFI-1257 Updated default iteration count for PBKDF2 cipher provider.
    [b083b66] NIFI-1257 Refactored parameter encoding to separate method.
    [5369fbb] NIFI-1257 Updated Javadoc.
    [90feb2a] NIFI-1257 Added check for all-zero IV (common security vulnerability) in AES
cipher provider.
    Added unit test.
    [7cf6c78] NIFI-1257 Updated Javadoc to be consistent across implementations.
    [c0424d6] NIFI-1257 Deprecated no-salt cipher generator for Bcrypt to be consistent with
Scrypt implementation.
    Updated unit tests.
    [06bed9a] NIFI-1257 Implemented Scrypt cipher provider.
    
    Added unit tests for empty salt. (+7 squashed commits)
    Squashed commits:
    [130b477] NIFI-1257 Added salt translator from mcrypt format to Java format.
    Added unit tests for salt formatting and validation.
    [c43ba44] NIFI-1257 Implemented ScryptCipherProvider and added unit test skeleton with
internal consistency and external compatibility tests.
    [9db678f] NIFI-1257 Updated Bcrypt unit test to accept new interface method contract.
    [897ad03] NIFI-1257 Added sanity test for Scrypt primitive key derivation compatibility
with external Ruby library.
    [4adc3fe] NIFI-1257 Exposed Scrypt parseSalt method as public to allow ScryptCipherProvider
to accept formatted salt externally.
    [391cd8b] NIFI-1257 Updated scrypt Ruby script to be consistent with bcrypt and pbkdf2
scripts in key and IV data handling.
    [d167917] NIFI-1257 Exposed desired key length as parameter on Scrypt underlying implementation.
(+2 squashed commits)
    Squashed commits:
    [7f1feb5] NIFI-1257 Standardized default salt generation to byte[] and added method contracts
in top-level PBECipherProvider interface.
    [673f35f] NIFI-1257 Scrypt implementation.
    
    Added default salt length constant and accessor in Scrypt.
    Changed reference to UTF-8 to be enum instead of freeform requiring runtime resolution
and redundant exception handling. (+11 squashed commits)
    Squashed commits:
    [b48ac88] NIFI-1257 Removed salt validation check in Scrypt (spec allows for empty salt)
and replaced with logger warnings. Moved unit test to ScryptCipherProvider unit test as enforcement
will be moved higher up.
    [611abf6] NIFI-1257 Made convention & style changes in scrypt code.
    [9005573] NIFI-1257 Added salt validation and unit test for Scrypt.
    [ff12568] NIFI-1257 Added validation check for password and relevant unit test.
    [6b42e8f] NIFI-1257 Added validation check for N, r, p in Scrypt and relevant unit tests.
    [8ab7e45] NIFI-1257 Added ruby script to test external compatibility for Scrypt implementation.
    [b720b30] NIFI-1257 Combined ScryptUtil (application-facing interface) into Scrypt class
to reduce unnecessary complexity.
    Updated unit test references.
    [7739349] NIFI-1257 Moved high-memory test vector into separate unit test with Assume
statement limiting execution depending on heap size to allow faster execution.
    [badd2ab] NIFI-1257 Enabled high-memory test vector for Scrypt test.
    [765cd38] NIFI-1257 Added surefire block to groovy unit test profile to enforce 3072 MB
heap for Scrypt test.
    [637f80b] NIFI-1257 Added local Java implementation of Scrypt KDF (and underlying PBKDF2
KDF) from Will Glozer.
    Added unit tests (including test vectors from paper).

commit f875ecac6c8bf2704a2d662587b7b635e2ee7e39
Author: Andy LoPresto <alopresto.apache@gmail.com>
Date:   2016-01-29T22:52:12Z

    NIFI-1257 Cleaned up interfaces and refactored shared code/unit and regression tests.
    
    Refactored shared stream handling code. (+1 squashed commit)
    Squashed commits:
    [4c6de17] NIFI-1257 Added logic to Encryptor to read and write IV to/from cipher stream.
    Added unit tests to cover all PBE KDFs. (+1 squashed commit)
    Squashed commits:
    [f333fa0] NIFI-1257 Added methods to read and write IV to/from cipher stream. (+12 squashed
commits)
    Squashed commits:
    [4ece756] NIFI-1257 Resolved all tests in nifi-standard-processors module.
    [8163f07] NIFI-1257 Refactored implementation-specific logic out of PasswordBasedEncryptor
to support all KDFs.
    Regression tests for legacy still pass.
    [a52da26] NIFI-1257 Added salt read/write logic to NifiLegacy and OpenSSL cipher providers.
    Added common salt read/write logic to RandomIVPBECipherProvider.
    Changed RandomIVPBECipherProvider from interface to abstract class.
    Updated strong KDF implementations.
    [1407afe] NIFI-1257 Added common cipher processing code to CipherUtility.
    [442495b] NIFI-1257 Added regression tests for PasswordBasedEncryptor with legacy OpenSSL
compatibility before integrating cipher provider logic.
    [d0410c0] NIFI-1257 Added regression test for PasswordBasedEncryptor before integrating
cipher provider logic.
    [e586933] NIFI-1257 Removed legacy unit test (superseded by Groovy unit test).
    [170314a] NIFI-1257 Removed unnecessary EncryptionMethod enumerations from unit test.
    [196d560] NIFI-1257 Added ASF 2.0 license to CipherProfiderFactory unit test.
    [36e95ba] NIFI-1257 Removed unnecessary KeyDeriver interface.
    [9c4eb07] NIFI-1257 Added default (NONE) KDF enum value.
    Renamed CipherFactory to CipherProviderFactory.
    Added unit test for registered KDF resolution from factory.
    [f006985] NIFI-1257 Added CipherProvider marker interface which PBE and keyed cipher providers
share.

commit d45dc69b3981ed6194a90aedbf603122d5c47c18
Author: Andy LoPresto <alopresto.apache@gmail.com>
Date:   2016-01-30T06:03:25Z

    NIFI-1257 Added IV read/write to KeyedCipherProvider and changed from interface to abstract
class. (+3 squashed commits)
    Squashed commits:
    [0ee5175] NIFI-1257 Moved null checks for encryption method and password to constructor
to de-duplicate and catch earlier.
    [6ed4142] NIFI-1257 Moved stream handling code to CipherUtility to allow reuse in KeyedCipherProvider.
    [a78b5db] NIFI-1257 Added null checks for encryption method and password.

commit 48b3463319a8fc620a9229a28253e4c99f4afc8e
Author: Andy LoPresto <alopresto.apache@gmail.com>
Date:   2016-01-30T07:27:46Z

    NIFI-1257 Added processor logic to invoke keyed cipher.
    Added unit tests. (+5 squashed commits)
    Squashed commits:
    [28b2224] NIFI-1257 Fixed contrib-check issues.
    [143b88f] NIFI-1257 Added helper method on KDF enum to indicate strength of KDF.
    Added EncryptContent processor property for raw hex key (always visible until NIFI-1121).
    Added validations for KDF (keyed and PBE) and hex key.
    Added unit tests for EncryptContent processor validations.
    [3a8367b] NIFI-1257 Added utility method to return list of valid key lengths for algorithm.
    Added unit tests.
    [04d6d16] NIFI-1257 Added description to allowable values for KDF and encryption method
in EncryptContent processor.
    [98595d8] NIFI-1257 Added KeyedEncryptor and unit tests.

commit 3c410d8f33b3ceaf5dfd79db355cf087edec80c8
Author: Andy LoPresto <alopresto.apache@gmail.com>
Date:   2016-02-02T01:25:28Z

    NIFI-1257 Fixed casing of Scrypt.java.

----


> Provide additional KDFs for EncryptContent
> ------------------------------------------
>
>                 Key: NIFI-1257
>                 URL: https://issues.apache.org/jira/browse/NIFI-1257
>             Project: Apache NiFi
>          Issue Type: Improvement
>          Components: Core Framework
>    Affects Versions: 0.4.0
>            Reporter: Andy LoPresto
>            Assignee: Andy LoPresto
>            Priority: Critical
>              Labels: encryption, security
>             Fix For: 0.5.0
>
>
> Currently, the two key derivation functions (KDF) supported are NiFi Legacy (1000 iterations
of MD5 digest over a password and optional salt) and OpenSSL PKCS#5 v1.5 (a single iteration
of MD5 digest over a password and optional salt). 
> Both of these are very weak -- they use a deprecated cryptographic hash function (CHF)
with known weakness and susceptibility to collisions (with demonstrated attacks) and a non-configurable
and tightly coupled iteration count to derive the key and IV. 
> Current best practice KDFs (with work factor recommendations) are as follows:
> * PBKDF2 with variable hash function (SHA1, SHA256, SHA384, SHA512, or ideally HMAC variants
of these functions) and variable iteration count (in the 10k - 1M range). 
> * bcrypt with work factor of 12 - 16
> * scrypt with work factor of (2^14 - 2^20, 8, 1)
> The salt and iteration count should be stored alongside the hashed record (bcrypt handles
this natively). 
> Notes:
> * http://wildlyinaccurate.com/bcrypt-choosing-a-work-factor/
> * http://blog.ircmaxell.com/2012/12/seven-ways-to-screw-up-bcrypt.html
> * http://security.stackexchange.com/questions/17207/recommended-of-rounds-for-bcrypt
> * http://security.stackexchange.com/questions/3959/recommended-of-iterations-when-using-pkbdf2-sha256/3993#3993
> * http://security.stackexchange.com/questions/4781/do-any-security-experts-recommend-bcrypt-for-password-storage/6415

> * http://web.archive.org/web/20130407190430/http://chargen.matasano.com/chargen/2007/9/7/enough-with-the-rainbow-tables-what-you-need-to-know-about-s.html
> * https://www.nccgroup.trust/us/about-us/newsroom-and-events/blog/2015/march/enough-with-the-salts-updates-on-secure-password-schemes/
> * http://www.tarsnap.com/scrypt.html
> * http://www.tarsnap.com/scrypt/scrypt.pdf



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message