drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-3535) Drop table support
Date Wed, 02 Sep 2015 16:48:46 GMT

    [ https://issues.apache.org/jira/browse/DRILL-3535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14727612#comment-14727612
] 

ASF GitHub Bot commented on DRILL-3535:
---------------------------------------

Github user mehant commented on a diff in the pull request:

    https://github.com/apache/drill/pull/140#discussion_r38555789
  
    --- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/store/dfs/WorkspaceSchemaFactory.java
---
    @@ -321,8 +327,101 @@ public DrillTable create(String key) {
           return null;
         }
     
    +    private FormatMatcher findMatcher(FileStatus file) {
    +      FormatMatcher matcher = null;
    +      try {
    +        for (FormatMatcher m : dropFileMatchers) {
    +          if (m.isFileReadable(fs, file)) {
    +            return m;
    +          }
    +        }
    +      } catch (IOException e) {
    +        logger.debug("Failed to find format matcher for file: %s", file, e);
    +      }
    +      return matcher;
    +    }
    +
         @Override
         public void destroy(DrillTable value) {
         }
    +
    +    /**
    +     * Check if the table contains homogenenous files that can be read by Drill. Eg:
parquet, json csv etc.
    +     * However if it contains more than one of these formats or a totally different file
format that Drill cannot
    +     * understand then we will raise an exception.
    +     * @param key
    +     * @return
    +     * @throws IOException
    +     */
    +    private boolean isHomogeneous(String key) throws IOException {
    +      FileSelection fileSelection = FileSelection.create(fs, config.getLocation(), key);
    +
    +      if (fileSelection == null) {
    +        throw UserException
    +            .validationError()
    +            .message(String.format("Table [%s] not found", key))
    +            .build(logger);
    +      }
    +
    +      FormatMatcher matcher = null;
    +      Queue<FileStatus> listOfFiles = new LinkedList<>();
    +      listOfFiles.addAll(fileSelection.getFileStatusList(fs));
    +
    +      while (!listOfFiles.isEmpty()) {
    +        FileStatus currentFile = listOfFiles.poll();
    +        if (currentFile.isDirectory()) {
    +          listOfFiles.addAll(fs.list(true, currentFile.getPath()));
    +        } else {
    +          if (matcher != null) {
    +            if (!matcher.isFileReadable(fs, currentFile)) {
    +              return false;
    +            }
    +          } else {
    +            matcher = findMatcher(currentFile);
    +            // Did not match any of the file patterns, exit
    +            if (matcher == null) {
    +              return false;
    +            }
    +          }
    +        }
    +      }
    +      return true;
    +    }
    +
    +    /**
    +     * We check if the table contains homogeneous file formats that Drill can read. Once
the checks are performed
    +     * we rename the file to start with an "_". After the rename we issue a recursive
delete of the directory.
    +     * @param table - Path of table to be dropped
    +     */
    +    @Override
    +    public void dropTable(String table) {
    +
    +      String[] pathSplit = table.split(Path.SEPARATOR);
    +      String dirName = DrillFileSystem.HIDDEN_FILE_PREFIX + pathSplit[pathSplit.length
- 1];
    +      int lastSlashIndex = table.lastIndexOf(Path.SEPARATOR);
    +
    +      if (lastSlashIndex != -1) {
    +        dirName = table.substring(0, lastSlashIndex + 1) + dirName;
    +      }
    +
    +      DrillFileSystem fs = getFS();
    +      String defaultLocation = getDefaultLocation();
    +      try {
    +        if (!isHomogeneous(table)) {
    +          throw UserException
    +              .validationError()
    +              .message("Table contains different file formats. \n" +
    +                  "Drop Table is only supported for directories that contain homogeneous
file formats consumable by Drill")
    +              .build(logger);
    +        }
    +        fs.rename(new Path(defaultLocation, table), new Path(defaultLocation, dirName));
    --- End diff --
    
    I will add a unique identifier in addition to the underscore prefix.
    
    I modified the exception handling below to separate out AccessControlException and the
generic IOException to handle permission errors and other errors respectively. In the case
of a race with concurrent renames the file system would throw a FileNotFoundException which
would be caught by the IOException and reported to the user.


> Drop table support
> ------------------
>
>                 Key: DRILL-3535
>                 URL: https://issues.apache.org/jira/browse/DRILL-3535
>             Project: Apache Drill
>          Issue Type: New Feature
>            Reporter: Mehant Baid
>            Assignee: Mehant Baid
>
> Umbrella JIRA to track support for "Drop table" feature.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message