drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-4831) Running refresh table metadata concurrently randomly fails with JsonParseException
Date Thu, 17 Nov 2016 08:20:59 GMT

    [ https://issues.apache.org/jira/browse/DRILL-4831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15673117#comment-15673117
] 

ASF GitHub Bot commented on DRILL-4831:
---------------------------------------

Github user amansinha100 commented on a diff in the pull request:

    https://github.com/apache/drill/pull/653#discussion_r88404356
  
    --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/store/parquet/Metadata.java
---
    @@ -495,31 +499,75 @@ private ParquetFileMetadata_v3 getParquetFileMetadata_v3(ParquetTableMetadata_v3
        * @param p
        * @throws IOException
        */
    -  private void writeFile(ParquetTableMetadata_v3 parquetTableMetadata, Path p) throws
IOException {
    +  private void writeFile(ParquetTableMetadata_v3 parquetTableMetadata, String path) throws
IOException {
         JsonFactory jsonFactory = new JsonFactory();
         jsonFactory.configure(Feature.AUTO_CLOSE_TARGET, false);
         jsonFactory.configure(JsonParser.Feature.AUTO_CLOSE_SOURCE, false);
         ObjectMapper mapper = new ObjectMapper(jsonFactory);
         SimpleModule module = new SimpleModule();
         module.addSerializer(ColumnMetadata_v3.class, new ColumnMetadata_v3.Serializer());
         mapper.registerModule(module);
    -    FSDataOutputStream os = fs.create(p);
    +
    +    // If multiple clients are updating metadata cache file concurrently, the cache file
    +    // can get corrupted. To prevent this, write to a unique temporary file and then
do
    +    // atomic rename.
    +    UUID randomUUID =  UUID.randomUUID();
    --- End diff --
    
    Is there a way to get the global queryId created by the Foreman ?  The name of the temp
file would best be associated with the queryId.  If that is not possible (I am not sure if
the query context is accessible easily from here), at the very least we should re-use the
UUID for both the METADATA_FILENAME and METADATA_DIRECTORIES_FILENAME .  It is not a correctness
issue but will avoid confusion. 


> Running refresh table metadata concurrently randomly fails with JsonParseException
> ----------------------------------------------------------------------------------
>
>                 Key: DRILL-4831
>                 URL: https://issues.apache.org/jira/browse/DRILL-4831
>             Project: Apache Drill
>          Issue Type: Bug
>          Components: Metadata
>    Affects Versions: 1.8.0
>            Reporter: Rahul Challapalli
>            Assignee: Aman Sinha
>         Attachments: error.log, l_3level.tgz
>
>
> git.commit.id.abbrev=f476eb5
> Just run the below command concurrently from 10 different JDBC connections. There is
a likelihood that you might encounter the below error
> Extracts from the log
> {code}
> Caused By (java.lang.AssertionError) Internal error: Error while applying rule DrillPushProjIntoScan,
args [rel#189411:LogicalProject.NONE.ANY([]).[](input=rel#189289:Subset#3.ENUMERABLE.ANY([]).[],l_orderkey=$1,dir0=$2,dir1=$3,dir2=$4,l_shipdate=$5,l_extendedprice=$6,l_discount=$7),
rel#189233:EnumerableTableScan.ENUMERABLE.ANY([]).[](table=[dfs, metadata_caching_pp, l_3level])]
>     org.apache.calcite.util.Util.newInternal():792
>     org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch():251
> .....
> .....
>   java.lang.Thread.run():745
>   Caused By (org.apache.drill.common.exceptions.DrillRuntimeException) com.fasterxml.jackson.core.JsonParseException:
Illegal character ((CTRL-CHAR, code 0)): only regular white space (\r, \n, \t) is allowed
between tokens
>  at [Source: com.mapr.fs.MapRFsDataInputStream@57a574a8; line: 1, column: 2]
>     org.apache.drill.exec.planner.logical.DrillPushProjIntoScan.onMatch():95
> {code}  
> Attached the complete log message and the data set



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message