drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Padma Penumarthy (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-4831) Running refresh table metadata concurrently randomly fails with JsonParseException
Date Tue, 25 Oct 2016 18:17:58 GMT

    [ https://issues.apache.org/jira/browse/DRILL-4831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15606034#comment-15606034
] 

Padma Penumarthy commented on DRILL-4831:
-----------------------------------------

Current implementation of metadata cache files do not have any concurrency control in place.
 If we have multiple connections trying to read/write metadata cache files at the same time,
nothing is guaranteed. The solution using tmp files (i.e. writing to a tmp file and atomic
rename when done)  resolves the conflict between multiple writers, fixing the problem of metadata
cache files getting corrupted. The caveat is that it is dependent on atomicity guarantee by
underlying file system. MapR FS and  Mac OS Extended file system (local file system I tested
with) support atomic rename. However, reads fail with I/O Error during rename for MapR FS,
while they work fine on Mac OS. 


> Running refresh table metadata concurrently randomly fails with JsonParseException
> ----------------------------------------------------------------------------------
>
>                 Key: DRILL-4831
>                 URL: https://issues.apache.org/jira/browse/DRILL-4831
>             Project: Apache Drill
>          Issue Type: Bug
>          Components: Metadata
>    Affects Versions: 1.8.0
>            Reporter: Rahul Challapalli
>            Assignee: Padma Penumarthy
>         Attachments: error.log, l_3level.tgz
>
>
> git.commit.id.abbrev=f476eb5
> Just run the below command concurrently from 10 different JDBC connections. There is
a likelihood that you might encounter the below error
> Extracts from the log
> {code}
> Caused By (java.lang.AssertionError) Internal error: Error while applying rule DrillPushProjIntoScan,
args [rel#189411:LogicalProject.NONE.ANY([]).[](input=rel#189289:Subset#3.ENUMERABLE.ANY([]).[],l_orderkey=$1,dir0=$2,dir1=$3,dir2=$4,l_shipdate=$5,l_extendedprice=$6,l_discount=$7),
rel#189233:EnumerableTableScan.ENUMERABLE.ANY([]).[](table=[dfs, metadata_caching_pp, l_3level])]
>     org.apache.calcite.util.Util.newInternal():792
>     org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch():251
> .....
> .....
>   java.lang.Thread.run():745
>   Caused By (org.apache.drill.common.exceptions.DrillRuntimeException) com.fasterxml.jackson.core.JsonParseException:
Illegal character ((CTRL-CHAR, code 0)): only regular white space (\r, \n, \t) is allowed
between tokens
>  at [Source: com.mapr.fs.MapRFsDataInputStream@57a574a8; line: 1, column: 2]
>     org.apache.drill.exec.planner.logical.DrillPushProjIntoScan.onMatch():95
> {code}  
> Attached the complete log message and the data set



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message