impala-reviews mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "David Knupp (Code Review)" <ger...@cloudera.org>
Subject [Impala-ASF-CR] IMPALA-4482: Use ALTER TABLE / RECOVER PARTITIONS when loading tpcds.store sales
Date Fri, 27 Jan 2017 22:12:53 GMT
David Knupp has posted comments on this change.

Change subject: IMPALA-4482: Use ALTER TABLE / RECOVER PARTITIONS when loading tpcds.store_sales
......................................................................


Patch Set 7:

This is the code in question from generate-schema-statements.py.

  # The ALTER statement in hive does not accept fully qualified table names so
  # insert a use statement. The ALTER statement is skipped for HBASE as it's
  # used for adding partitions.
  # TODO: Consider splitting the ALTER subsection into specific components. At the
  # moment, it assumes we're only using ALTER for partitioning the table.
  if alter and file_format not in ("hbase", "kudu"):
    use_db = 'USE {db_name};\n'.format(db_name=db)
    if output == hive_output and codec == 'lzo':
      # Hive ALTER TABLE ADD PARTITION doesn't handle null partitions, so
      # we can't run the ALTER section in this case.
      if options.force_reload:
        # IMPALA-2278: Hive INSERT OVERWRITE won't clear out partition directories
        # that weren't already added to the table. So, for force reload, manually
        # delete the partition directories.
        output.create.append(("DFS -rm -R {data_path};").format(
          data_path=data_path));
      else:
        # If this is not a force reload use msck repair to add the partitions
        # into the table.
        output.create.append(use_db + 'msck repair table %s;' % (table_name))
    else:
      output.create.append(use_db + alter.format(table_name=table_name))

If you look at the last line, the reason we're seeing an error is that 'db_name' is not passed
in the call to alter.format().

The opening comment regarding the ALTER statement in hive not accepting fully qualified table
names  -- I presume that was true at one point? The comment is from 2013.

-- 
To view, visit http://gerrit.cloudera.org:8080/5177
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Iaae97d1d44201aeeacacdd39adbae35753512950
Gerrit-PatchSet: 7
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: David Knupp <dknupp@cloudera.com>
Gerrit-Reviewer: David Knupp <dknupp@cloudera.com>
Gerrit-Reviewer: Dimitris Tsirogiannis <dtsirogiannis@cloudera.com>
Gerrit-Reviewer: Harrison Sheinblatt <hs7@hotmail.com>
Gerrit-Reviewer: Internal Jenkins
Gerrit-Reviewer: Jim Apple <jbapple-impala@apache.org>
Gerrit-HasComments: No

Mime
View raw message