impala-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joe McDonnell (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (IMPALA-5423) test_file_modifications fails on local filesystem and isilon
Date Fri, 09 Jun 2017 16:23:18 GMT

     [ https://issues.apache.org/jira/browse/IMPALA-5423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Joe McDonnell resolved IMPALA-5423.
-----------------------------------
       Resolution: Fixed
    Fix Version/s: Impala 2.10.0

commit 5f9f704bde6816c8469e31b1d655f3df5732dac4
Author: Joe McDonnell <joemcdonnell@cloudera.com>
Date:   Tue May 30 12:59:37 2017 -0700

    IMPALA-5386: Fix ReopenCachedHdfsFileHandle failure case
    
    This fixes three issues with the file handle cache.
    
    The first issue is that ReopenCachedHdfsFileHandle can
    destroy the passed in file handle without removing the
    reference to it. The old file handle then refers to
    a piece of memory that is not a handle in the cache,
    so future use of the handle fails with an assert. The
    fix is to always overwrite the reference to the file
    handle when it has been destroyed.
    
    The second issue is that query_test/test_hdfs_fd_caching.py
    should run on anything that supports the hdfs commandline
    and tolerate query failure. It's logic is not specific to
    file handle caching, so it has been renamed to
    query_test/test_hdfs_file_mods.py.
    
    Finally, custom_cluster/test_hdfs_fd_caching.py should not
    be running on remote files (S3, ADLS, Isilon, remote
    clusters). The file handle cache semantics won't apply on
    those platforms.
    
    Change-Id: Iee982fa5e964f6c8969b2eb7e5f3eca89e793b3a
    Reviewed-on: http://gerrit.cloudera.org:8080/7020
    Reviewed-by: Joe McDonnell <joemcdonnell@cloudera.com>
    Reviewed-by: Dan Hecht <dhecht@cloudera.com>
    Tested-by: Impala Public Jenkins


> test_file_modifications fails on local filesystem and isilon
> ------------------------------------------------------------
>
>                 Key: IMPALA-5423
>                 URL: https://issues.apache.org/jira/browse/IMPALA-5423
>             Project: IMPALA
>          Issue Type: Bug
>          Components: Backend
>    Affects Versions: Impala 2.10.0
>            Reporter: Thomas Tauber-Marshall
>            Assignee: Joe McDonnell
>            Priority: Blocker
>              Labels: broken-build
>             Fix For: Impala 2.10.0
>
>
> Appears to be related to 'IMPALA-4623: Enable file handle cache'
> {noformat}
> 03:52:04  TestHdfsFdCaching.test_file_modifications[modification_type: delete_files |
exec_option: {'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold':
0, 'batch_size': 0, 'num_nodes': 0} | table_format: text/none] 
> 03:52:04 query_test/test_hdfs_fd_caching.py:102: in test_file_modifications
> 03:52:04     self.execute_query(count_query)
> 03:52:04 common/impala_test_suite.py:506: in wrapper
> 03:52:04     return function(*args, **kwargs)
> 03:52:04 common/impala_test_suite.py:531: in execute_query
> 03:52:04     return self.__execute_query(self.client, query, query_options)
> 03:52:04 common/impala_test_suite.py:598: in __execute_query
> 03:52:04     return impalad_client.execute(query, user=user)
> 03:52:04 common/impala_connection.py:160: in execute
> 03:52:04     return self.__beeswax_client.execute(sql_stmt, user=user)
> 03:52:04 beeswax/impala_beeswax.py:173: in execute
> 03:52:04     handle = self.__execute_query(query_string.strip(), user=user)
> 03:52:04 beeswax/impala_beeswax.py:339: in __execute_query
> 03:52:04     self.wait_for_completion(handle)
> 03:52:04 beeswax/impala_beeswax.py:359: in wait_for_completion
> 03:52:04     raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
> 03:52:04 E   ImpalaBeeswaxException: ImpalaBeeswaxException:
> 03:52:04 E    Query aborted:Failed to open HDFS file file:/tmp/test-warehouse/test_file_modifications_cdd1967c/data.csv
> 03:52:04 E   Error(2): No such file or directory
> 03:52:04 E   Root cause: FileNotFoundException: File file:/tmp/test-warehouse/test_file_modifications_cdd1967c/data.csv
does not exist
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message