drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Altekruse (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-3522) IllegalStateException from Mongo storage plugin
Date Tue, 09 Feb 2016 19:22:18 GMT

    [ https://issues.apache.org/jira/browse/DRILL-3522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15139462#comment-15139462
] 

Jason Altekruse commented on DRILL-3522:
----------------------------------------

This applied cleanly to master and did not cause any mongo test failures, I am planning to
merge it to master shortly, there should be no need for a new patch to be uploaded.

> IllegalStateException from Mongo storage plugin
> -----------------------------------------------
>
>                 Key: DRILL-3522
>                 URL: https://issues.apache.org/jira/browse/DRILL-3522
>             Project: Apache Drill
>          Issue Type: Bug
>          Components: Storage - MongoDB
>    Affects Versions: 1.1.0
>            Reporter: Adam Gilmore
>            Assignee: Adam Gilmore
>            Priority: Critical
>         Attachments: DRILL-3522.1.patch.txt
>
>
> With a Mongo storage plugin enabled, we are sporadically getting the following exception
when running queries (even not against the Mongo storage plugin):
> {code}
> SYSTEM ERROR: IllegalStateException: state should be: open
>   (org.apache.drill.exec.work.foreman.ForemanException) Unexpected exception during fragment
initialization: org.apache.drill.common.exceptions.DrillRuntimeException: state should be:
open
>     org.apache.drill.exec.work.foreman.Foreman.run():253
>     java.util.concurrent.ThreadPoolExecutor.runWorker():1145
>     java.util.concurrent.ThreadPoolExecutor$Worker.run():615
>     java.lang.Thread.run():745
>   Caused By (com.google.common.util.concurrent.UncheckedExecutionException) org.apache.drill.common.exceptions.DrillRuntimeException:
state should be: open
>     com.google.common.cache.LocalCache$Segment.get():2263
>     com.google.common.cache.LocalCache.get():4000
>     com.google.common.cache.LocalCache.getOrLoad():4004
>     com.google.common.cache.LocalCache$LocalLoadingCache.get():4874
>     org.apache.drill.exec.store.mongo.schema.MongoSchemaFactory$MongoSchema.getSubSchemaNames():172
>     org.apache.drill.exec.store.mongo.schema.MongoSchemaFactory$MongoSchema.setHolder():159
>     org.apache.drill.exec.store.mongo.schema.MongoSchemaFactory.registerSchemas():127
>     org.apache.drill.exec.store.mongo.MongoStoragePlugin.registerSchemas():86
>     org.apache.drill.exec.store.StoragePluginRegistry$DrillSchemaFactory.registerSchemas():328
>     org.apache.drill.exec.ops.QueryContext.getRootSchema():165
>     org.apache.drill.exec.ops.QueryContext.getRootSchema():154
>     org.apache.drill.exec.ops.QueryContext.getRootSchema():142
>     org.apache.drill.exec.ops.QueryContext.getNewDefaultSchema():128
>     org.apache.drill.exec.planner.sql.DrillSqlWorker.():91
>     org.apache.drill.exec.work.foreman.Foreman.runSQL():901
>     org.apache.drill.exec.work.foreman.Foreman.run():242
>     java.util.concurrent.ThreadPoolExecutor.runWorker():1145
>     java.util.concurrent.ThreadPoolExecutor$Worker.run():615
>     java.lang.Thread.run():745
>   Caused By (org.apache.drill.common.exceptions.DrillRuntimeException) state should be:
open
>     org.apache.drill.exec.store.mongo.schema.MongoSchemaFactory$DatabaseLoader.load():98
>     org.apache.drill.exec.store.mongo.schema.MongoSchemaFactory$DatabaseLoader.load():82
>     com.google.common.cache.LocalCache$LoadingValueReference.loadFuture():3599
>     com.google.common.cache.LocalCache$Segment.loadSync():2379
>     com.google.common.cache.LocalCache$Segment.lockedGetOrLoad():2342
>     com.google.common.cache.LocalCache$Segment.get():2257
>     com.google.common.cache.LocalCache.get():4000
>     com.google.common.cache.LocalCache.getOrLoad():4004
>     com.google.common.cache.LocalCache$LocalLoadingCache.get():4874
>     org.apache.drill.exec.store.mongo.schema.MongoSchemaFactory$MongoSchema.getSubSchemaNames():172
>     org.apache.drill.exec.store.mongo.schema.MongoSchemaFactory$MongoSchema.setHolder():159
>     org.apache.drill.exec.store.mongo.schema.MongoSchemaFactory.registerSchemas():127
>     org.apache.drill.exec.store.mongo.MongoStoragePlugin.registerSchemas():86
>     org.apache.drill.exec.store.StoragePluginRegistry$DrillSchemaFactory.registerSchemas():328
>     org.apache.drill.exec.ops.QueryContext.getRootSchema():165
>     org.apache.drill.exec.ops.QueryContext.getRootSchema():154
>     org.apache.drill.exec.ops.QueryContext.getRootSchema():142
>     org.apache.drill.exec.ops.QueryContext.getNewDefaultSchema():128
>     org.apache.drill.exec.planner.sql.DrillSqlWorker.():91
>     org.apache.drill.exec.work.foreman.Foreman.runSQL():901
>     org.apache.drill.exec.work.foreman.Foreman.run():242
>     java.util.concurrent.ThreadPoolExecutor.runWorker():1145
>     java.util.concurrent.ThreadPoolExecutor$Worker.run():615
>     java.lang.Thread.run():745
>   Caused By (java.lang.IllegalStateException) state should be: open
>     com.mongodb.assertions.Assertions.isTrue():70
>     com.mongodb.connection.BaseCluster.selectServer():79
>     com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.():75
>     com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.():71
>     com.mongodb.binding.ClusterBinding.getReadConnectionSource():63
>     com.mongodb.operation.OperationHelper.withConnection():166
>     com.mongodb.operation.ListDatabasesOperation.execute():100
>     com.mongodb.operation.ListDatabasesOperation.execute():52
>     com.mongodb.Mongo.execute():738
>     com.mongodb.Mongo$2.execute():725
>     com.mongodb.OperationIterable.iterator():47
>     com.mongodb.OperationIterable.forEach():66
>     com.mongodb.ListDatabasesIterableImpl.forEach():72
>     com.mongodb.MappingIterable.forEach():50
>     com.mongodb.MappingIterable.into():60
>     org.apache.drill.exec.store.mongo.schema.MongoSchemaFactory$DatabaseLoader.load():91
>     org.apache.drill.exec.store.mongo.schema.MongoSchemaFactory$DatabaseLoader.load():82
>     com.google.common.cache.LocalCache$LoadingValueReference.loadFuture():3599
>     com.google.common.cache.LocalCache$Segment.loadSync():2379
>     com.google.common.cache.LocalCache$Segment.lockedGetOrLoad():2342
>     com.google.common.cache.LocalCache$Segment.get():2257
>     com.google.common.cache.LocalCache.get():4000
>     com.google.common.cache.LocalCache.getOrLoad():4004
>     com.google.common.cache.LocalCache$LocalLoadingCache.get():4874
>     org.apache.drill.exec.store.mongo.schema.MongoSchemaFactory$MongoSchema.getSubSchemaNames():172
>     org.apache.drill.exec.store.mongo.schema.MongoSchemaFactory$MongoSchema.setHolder():159
>     org.apache.drill.exec.store.mongo.schema.MongoSchemaFactory.registerSchemas():127
>     org.apache.drill.exec.store.mongo.MongoStoragePlugin.registerSchemas():86
>     org.apache.drill.exec.store.StoragePluginRegistry$DrillSchemaFactory.registerSchemas():328
>     org.apache.drill.exec.ops.QueryContext.getRootSchema():165
>     org.apache.drill.exec.ops.QueryContext.getRootSchema():154
>     org.apache.drill.exec.ops.QueryContext.getRootSchema():142
>     org.apache.drill.exec.ops.QueryContext.getNewDefaultSchema():128
>     org.apache.drill.exec.planner.sql.DrillSqlWorker.():91
>     org.apache.drill.exec.work.foreman.Foreman.runSQL():901
>     org.apache.drill.exec.work.foreman.Foreman.run():242
>     java.util.concurrent.ThreadPoolExecutor.runWorker():1145
>     java.util.concurrent.ThreadPoolExecutor$Worker.run():615
>     java.lang.Thread.run():745
> {code}
> Upon investigation, the issue appears to be registering the Mongo schemas on connection
(because the storage plugin is enabled).
> Basically, it appears that recent changes when upgrading the Mongo driver meant that
MongoSchemaFactory indefinitely holds onto a MongoClient, even though they are part of a cache
that expire after 24 hours.  This means that MongoSchemaFactory ends up trying to use a MongoClient
that is closed, and thus the exception occurs.
> Considering we already have a cache for MongoClient, it is safe for MongoSchemaFactory
to call plugin.getClient() every time.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message