Mailing-List: contact commits-help@nifi.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@nifi.apache.org
Date: Sun, 15 Nov 2015 15:21:10 +0000 (UTC)
From: "ASF GitHub Bot (JIRA)" <jira@apache.org>
To: commits@nifi.apache.org
Message-ID: <JIRA.12842354.1435853737000.74860.1447600870970@Atlassian.JIRA>
In-Reply-To: <JIRA.12842354.1435853737000@Atlassian.JIRA>
References: <JIRA.12842354.1435853737000@Atlassian.JIRA>
 <JIRA.12842354.1435853737533@arcas>
Subject: [jira] [Commented] (NIFI-748) If unable to find a specific
 Provenance event, should not fail entire search
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/NIFI-748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15005930#comment-15005930 ] 

ASF GitHub Bot commented on NIFI-748:
-------------------------------------

Github user trkurc commented on a diff in the pull request:

    https://github.com/apache/nifi/pull/123#discussion_r44872922
  
    --- Diff: nifi-nar-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/lucene/DocsReader.java ---
    @@ -100,101 +96,61 @@ private ProvenanceEventRecord getRecord(final Document d, final RecordReader rea
                 }
             }
     
    -        if ( record == null ) {
    -            throw new IOException("Failed to find Provenance Event " + d);
    -        } else {
    -            return record;
    +        if (record == null) {
    +            logger.warn("Failed to read Provenance Event for '" + d + "'. The event file may be missing or corrupted");
             }
    -    }
     
    +        return record;
    +    }
     
         public Set<ProvenanceEventRecord> read(final List<Document> docs, final Collection<Path> allProvenanceLogFiles,
    -        final AtomicInteger retrievalCount, final int maxResults, final int maxAttributeChars) throws IOException {
    -        if (retrievalCount.get() >= maxResults) {
    -            return Collections.emptySet();
    -        }
    -
    -        LuceneUtil.sortDocsForRetrieval(docs);
    -
    -        RecordReader reader = null;
    -        String lastStorageFilename = null;
    -        final Set<ProvenanceEventRecord> matchingRecords = new LinkedHashSet<>();
    +            final AtomicInteger retrievalCount, final int maxResults, final int maxAttributeChars) throws IOException {
     
             final long start = System.nanoTime();
    -        int logFileCount = 0;
    -
    -        final Set<String> storageFilesToSkip = new HashSet<>();
    -        int eventsReadThisFile = 0;
     
    -        try {
    -            for (final Document d : docs) {
    -                final String storageFilename = d.getField(FieldNames.STORAGE_FILENAME).stringValue();
    -                if ( storageFilesToSkip.contains(storageFilename) ) {
    -                    continue;
    -                }
    -
    -                try {
    -                    if (reader != null && storageFilename.equals(lastStorageFilename)) {
    -                        matchingRecords.add(getRecord(d, reader));
    -                        eventsReadThisFile++;
    -
    -                        if ( retrievalCount.incrementAndGet() >= maxResults ) {
    -                            break;
    -                        }
    -                    } else {
    -                        logger.debug("Opening log file {}", storageFilename);
    -
    -                        logFileCount++;
    -                        if (reader != null) {
    -                            reader.close();
    -                        }
    +        Set<ProvenanceEventRecord> matchingRecords = new LinkedHashSet<>();
    +        if (retrievalCount.get() >= maxResults) {
    +            return matchingRecords;
    --- End diff --
    
    @joewitt - if this causes GC problems we've got other _major_ issues. I was just bit surprised a change was made that in my opinion was marginally and subjectively a step backwards.


> If unable to find a specific Provenance event, should not fail entire search
> ----------------------------------------------------------------------------
>
>                 Key: NIFI-748
>                 URL: https://issues.apache.org/jira/browse/NIFI-748
>             Project: Apache NiFi
>          Issue Type: Improvement
>          Components: Core Framework
>            Reporter: Mark Payne
>            Assignee: Oleg Zhurakousky
>             Fix For: 0.4.0
>
>
> We have a case where running with the prov being written to a disk that can be ejected. Disk was accidentally ejected while running. Provenance Event appears to have been indexed but event is not in the repo.
> Specifically, we are reaching Line 104 of DocsReader:
> {code}
> throw new IOException("Failed to find Provenance Event " + d);
> {code}
> As a result, searching for a specific Component ID is returning an error, so we can't search on that Component ID at all (unless we shrink the time range to a time when that didn't occur).
> We should generate a warning, and notify the user that X number of events could not be found and show what we can, rather than erroring out entirely.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)