nifi-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (NIFI-748) If unable to find a specific Provenance event, should not fail entire search
Date Sun, 15 Nov 2015 15:24:10 GMT

    [ https://issues.apache.org/jira/browse/NIFI-748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15005934#comment-15005934
] 

ASF GitHub Bot commented on NIFI-748:
-------------------------------------

Github user trkurc commented on a diff in the pull request:

    https://github.com/apache/nifi/pull/123#discussion_r44872964
  
    --- Diff: nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/lucene/DocsReader.java
---
    @@ -100,101 +96,61 @@ private ProvenanceEventRecord getRecord(final Document d, final
RecordReader rea
                 }
             }
     
    -        if ( record == null ) {
    -            throw new IOException("Failed to find Provenance Event " + d);
    -        } else {
    -            return record;
    +        if (record == null) {
    +            logger.warn("Failed to read Provenance Event for '" + d + "'. The event file
may be missing or corrupted");
             }
    -    }
     
    +        return record;
    +    }
     
         public Set<ProvenanceEventRecord> read(final List<Document> docs, final
Collection<Path> allProvenanceLogFiles,
    -        final AtomicInteger retrievalCount, final int maxResults, final int maxAttributeChars)
throws IOException {
    -        if (retrievalCount.get() >= maxResults) {
    -            return Collections.emptySet();
    -        }
    -
    -        LuceneUtil.sortDocsForRetrieval(docs);
    --- End diff --
    
    I base this on  the comment from the sort method:
     // sort by filename and then file offset so that we can retrieve the records efficiently



> If unable to find a specific Provenance event, should not fail entire search
> ----------------------------------------------------------------------------
>
>                 Key: NIFI-748
>                 URL: https://issues.apache.org/jira/browse/NIFI-748
>             Project: Apache NiFi
>          Issue Type: Improvement
>          Components: Core Framework
>            Reporter: Mark Payne
>            Assignee: Oleg Zhurakousky
>             Fix For: 0.4.0
>
>
> We have a case where running with the prov being written to a disk that can be ejected.
Disk was accidentally ejected while running. Provenance Event appears to have been indexed
but event is not in the repo.
> Specifically, we are reaching Line 104 of DocsReader:
> {code}
> throw new IOException("Failed to find Provenance Event " + d);
> {code}
> As a result, searching for a specific Component ID is returning an error, so we can't
search on that Component ID at all (unless we shrink the time range to a time when that didn't
occur).
> We should generate a warning, and notify the user that X number of events could not be
found and show what we can, rather than erroring out entirely.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message