nifi-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alessandro D'Armiento (JIRA)" <>
Subject [jira] [Created] (NIFI-6464) ListHDFS should support fragment attributes with strategies
Date Mon, 22 Jul 2019 12:47:00 GMT
Alessandro D'Armiento created NIFI-6464:

             Summary: ListHDFS should support fragment attributes with strategies
                 Key: NIFI-6464
             Project: Apache NiFi
          Issue Type: Improvement
          Components: Core Framework
    Affects Versions: 1.9.2
            Reporter: Alessandro D'Armiento

h2. Current Situation

ListHDFS doesn't support Fragmentation attributes

h2. Improvement Proposal

 * Since the processor works on a 1:N semantic (1 input trigger flowfile, N output flowfiles)
it would be nice to support fragmentation attributes (for example for subsequent merge operations)
 ** It would be also useful to support different fragmentation strategies, in order to support
multiple user cases. For example, it should be possible to select:
 *** A "one for all" fragmentation strategy which will create a single fragmentation group.
Therefore, all files will have the same fragment.identifier, the same fragment.count, equal
to the total number N of listed files, and fragment.index ∈ [0, N).
 *** A "per subdir" fragmentation strategy which will create different fragmentation groups,
one for each scanned subdirectory of the given path. Therefore, for each subfolder, flowfiles
will have a specific fragment.identifier, fragment.count will be, for each flowfile, equal
to the number Ni of files in the i-th directory, and fragment.index ∈ [0, Ni).

This message was sent by Atlassian JIRA

View raw message