drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paul Rogers (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (DRILL-5504) Vector validator to diagnose offset vector issues
Date Thu, 11 May 2017 00:11:04 GMT

    [ https://issues.apache.org/jira/browse/DRILL-5504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16005685#comment-16005685
] 

Paul Rogers edited comment on DRILL-5504 at 5/11/17 12:10 AM:
--------------------------------------------------------------

Mini-design:

1. Create a BatchValidator class to iterate over vectors in a batch and validate each.
2. Create a VectorValidator class to validate each kind of vector that lends itself to validation.
3. Add a new session option, debug.validate_vectors to enable validation.
4. Modify IteratorValidatorBatchIterator to invoke the batch validator on each batch, if the
above option is set.
5. Add another new session option, debug.validate_iterators, to enable the iterator validator.
6. Modify ImplCreator to consider both assertions enabled OR the debug.validate_iterator option
to decide to inject the iterator validator.

The result will be that a production, non-debug user can enable validation for just a problematic
query, without having to restart the Drillbit with assertions enabled.


was (Author: paul-rogers):
Mini-design:

1. Create a BatchValidator class to iterate over vectors in a batch and validate each.
2. Create a VectorValidator class to validate each kind of vector that lends itself to validation.
3. Add a new session option, debug.validate-vectors to enable validation.
4. Modify IteratorValidatorBatchIterator to invoke the batch validator on each batch, if the
above option is set.
5. Add another new session option, debug.validate-iterator, to enable the iterator validator.
6. Modify ImplCreator to consider both assertions enabled OR the debug.validate-iterator option
to decide to inject the iterator validator.

The result will be that a production, non-debug user can enable validation for just a problematic
query, without having to restart the Drillbit with assertions enabled.

> Vector validator to diagnose offset vector issues
> -------------------------------------------------
>
>                 Key: DRILL-5504
>                 URL: https://issues.apache.org/jira/browse/DRILL-5504
>             Project: Apache Drill
>          Issue Type: Improvement
>    Affects Versions: 1.10.0
>            Reporter: Paul Rogers
>            Assignee: Paul Rogers
>            Priority: Minor
>             Fix For: 1.11.0
>
>
> DRILL-5470 describes a case in which an offset vector appears to have become corrupted,
yielding a bogus field-length value that is orders of magnitude larger than the vector that
contains the data.
> Debugging such cases is slow and tedious. To help, we propose to create a "vector validator"
that spins through vectors looking for problems.
> Then, to allow the validator to be used in the field, extend the "iterator validator
batch iterator" to optionally allow vector validation on each batch.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message