drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paul Rogers (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-5255) Drill requires that dfs.tmp be set up a start time, even if CTTAS not used
Date Sat, 11 Feb 2017 06:57:42 GMT

    [ https://issues.apache.org/jira/browse/DRILL-5255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15862268#comment-15862268
] 

Paul Rogers commented on DRILL-5255:
------------------------------------

The situation is a bit more subtle than the description suggests. The standard {{bootstrap-storage-plugins.json}}
does contain a definition for {{dfs.tmp}}.

Tracking through a bit more, it turns out that my machine has created a local storage for
plugins at: {{/tmp/drill/sys.storage_plugins/dfs.sys.drill}}. In that, the tmp directory is
defined as:

{code}
    "tmp" : {
      "location" : "/tmp",
      "writable" : false,
      "defaultInputFormat" : null
    },
{code}

This seems to be how things get persisted when the following standard test setting is configured
in the bootstrap options:

{code}
      put(ExecConstants.SYS_STORE_PROVIDER_LOCAL_ENABLE_WRITE, false);
{code}

Somehow the {{dfs.tmp}} has got to be marked writable (the default configuration) even when
the system store local write is not enabled.

Digging even deeper, it turns out that the tests themselves cause the problems:

{code}
package org.apache.drill.exec.util;

public class TestUtilities {
  ...
  /**
   * Make the dfs.tmp schema immutable, so that tests writers don't use the dfs.tmp to create
views.
   * Schema "dfs.tmp" added as part of the default bootstrap plugins file that comes with
drill-java-exec jar
   */
  public static void makeDfsTmpSchemaImmutable(final StoragePluginRegistry pluginRegistry)
throws ExecutionSetupException {
{code}

Executing this test code causes Drillbit startup to fail.

This code is called from the following test framework location:

{code}
package org.apache.drill;

public class BaseTestQuery extends ExecTest {
  private static void openClient() throws Exception {
    ..
      TestUtilities.makeDfsTmpSchemaImmutable(pluginRegistry);
{code}

How do the tests pass? Via magic at startup time:

{code}
  public static final String TEMP_SCHEMA = "dfs_test.tmp";
      ..
      put(ExecConstants.DEFAULT_TEMPORARY_WORKSPACE, TEMP_SCHEMA);
{code}

All of this seems very fragile. A much better solution is to postpone the check until executing
a query that does an actual CTTAS statement.

The workaround is to add the same setup logic to the new test framework to work around the
work around for dfs.tmp.

> Drill requires that dfs.tmp be set up a start time, even if CTTAS not used
> --------------------------------------------------------------------------
>
>                 Key: DRILL-5255
>                 URL: https://issues.apache.org/jira/browse/DRILL-5255
>             Project: Apache Drill
>          Issue Type: Bug
>    Affects Versions: 1.10
>            Reporter: Paul Rogers
>            Assignee: Arina Ielchiieva
>             Fix For: 1.10
>
>
> Drill can operate in embedded mode. In this mode, no storage plugin definitions other
than the defaults may be present. In particular, when using the Drill test framework, only
those storage plugins defined in the Drill code are available.
> Yet, Drill checks for the existence of the dfs.tmp plugin definition (as named by the
{{drill.exec.default_temporary_workspace}} parameter. Because this plugin is not defined,
an exception occurs:
> {code}
> org.apache.drill.common.exceptions.UserException: PARSE ERROR: Unable to create or drop
tables/views. Schema [dfs.tmp] is immutable.
> [Error Id: 792d4e5d-3f31-4f38-8bb4-d108f1a808f6 ]
> 	at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:544)
> 	at org.apache.drill.exec.planner.sql.SchemaUtilites.resolveToMutableDrillSchema(SchemaUtilites.java:184)
> 	at org.apache.drill.exec.planner.sql.SchemaUtilites.getTemporaryWorkspace(SchemaUtilites.java:201)
> 	at org.apache.drill.exec.server.Drillbit.validateTemporaryWorkspace(Drillbit.java:264)
> 	at org.apache.drill.exec.server.Drillbit.run(Drillbit.java:135)
> 	at org.apache.drill.test.ClusterFixture.startDrillbits(ClusterFixture.java:207)
> 	...
> {code}
> Expected that either a configuration would exist that would use the default /tmp/drill
location, or that the check for {{drill.tmp}} would be deferred until it is actually required
(such as when executing a CTTAS statement.)
> It seemed that the test framework must be altered to work around this problem by defining
the necessary workspace. Unfortunately, the Drillbit must start before we can define the workspace
needed for the Drillbit to start. So, this workaround is not possible.
> Further, users of the embedded Drillbit may not know to do this configuration.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message