hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chinna Rao Lalam (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HIVE-1897) Alter command execution "when HDFS is down" results in holding stale data in MetaStore
Date Mon, 24 Jan 2011 13:08:44 GMT

    [ https://issues.apache.org/jira/browse/HIVE-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12985666#action_12985666
] 

Chinna Rao Lalam commented on HIVE-1897:
----------------------------------------

In HiveAlterHandler ( alterTable ) , if the HDFS operation failed then its better to rename
the table to the old table name. To avoid the inconsistency b/w Metadata and HDFS data. 

h4. Old code snippet 
h4. 
{code}
 if (success && moveData) {
        // change the file name in hdfs
        // check that src exists otherwise there is no need to copy the data
        try {
          if (srcFs.exists(srcPath)) {
            // rename the src to destination
            srcFs.rename(srcPath, destPath);
          }
        } catch (IOException e) {

		throw new InvalidOperationException("Unable to access old location "
              + srcPath + " for table " + dbname + "." + name);
	}
{code}

h4. Changes Proposed 

Before throwing the "InvalidOperationException" , rename the table name to the old table name.


{code}

      if (success && moveData) {
        // change the file name in hdfs
        // check that src exists otherwise there is no need to copy the data
        try {
          if (srcFs.exists(srcPath)) {
            // rename the src to destination
            srcFs.rename(srcPath, destPath);
          }
        } catch (IOException e) {
          boolean revertMetaDataTransaction = false;
          try
          {
              msdb.openTransaction();

              // Alter the table name to have the old table name. 
              msdb.alterTable(dbname, newt.getTableName(), oldt);
              revertMetaDataTransaction = msdb.commitTransaction();
          }
          catch (Exception e1)
          {
            LOG.error("Reverting metadata opeation failed During HDFS operation failed",e1);
            if(!revertMetaDataTransaction)
            {
               msdb.rollbackTransaction();
            }
          }
          throw new InvalidOperationException("Unable to access old location "
              + srcPath + " for table " + dbname + "." + name);
        }
      }

{code}

> Alter command execution "when HDFS is down" results in holding stale data in MetaStore

> ---------------------------------------------------------------------------------------
>
>                 Key: HIVE-1897
>                 URL: https://issues.apache.org/jira/browse/HIVE-1897
>             Project: Hive
>          Issue Type: Bug
>          Components: Metastore
>    Affects Versions: 0.6.0
>         Environment: Hadoop 0.20.1, Hive0.6.0 and SUSE Linux Enterprise Server 10 SP2
(i586) - Kernel 2.6.16.60-0.21-smp (5).
>            Reporter: Chinna Rao Lalam
>
> Lets  consider, the  *"DFS"* is down , 
> And on executing an alter query say  *"alter table firsttable rename to secondtable"*.
 
> the query execution fails with the following exception:
> {color:red} 
> InvalidOperationException(message:Unable to access old location hdfs://localhost:9000/user/hive/warehouse/firsttable
for table default.firsttable)
> {color}
> Now after starting the *DFS* and then executing the same query , the client gets the
following exception:
> {color:red}
> NoSuchObjectException(message:default.firsttable table not found)
> {color}
> h4.Root Cause
> In Alter Query execution flow, first *"MetaStore"* operation is executed successfully
and then *"DFS"* operation is started. In this scenario, *"DFS"* is down. As a result, execution
of the query failed and partial information of the operation is saved.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message