hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Patrick McCabe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-1309) FileSystem.rename will fail silently
Date Mon, 28 Apr 2014 17:36:16 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13983259#comment-13983259
] 

Colin Patrick McCabe commented on HDFS-1309:
--------------------------------------------

Is this still an issue in branch 2?  It looks like {{DistributedFileSystem#rename}} throws
{{IOException}} if something goes wrong, and has for a while.  Unless I'm missing something,
rename is not a filesystem operation that fails silently.

> FileSystem.rename will fail silently
> ------------------------------------
>
>                 Key: HDFS-1309
>                 URL: https://issues.apache.org/jira/browse/HDFS-1309
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: hdfs-client
>    Affects Versions: 0.20.2
>         Environment: Linux version 2.6.31-302-ec2 (buildd@yellow) (gcc version 4.4.1
(Ubuntu 4.4.1-4ubuntu7) ) #7-Ubuntu SMP Tue Oct 13 19:55:22 UTC 2009
>            Reporter: Kris Nuttycombe
>            Priority: Minor
>
> Some filesystem operations (such as rename) will fail silently. In the attached example,
a failure message will be written to the hadoop log, but it would be much better if the operation
were to "fail fast" by throwing a checked exception and forcing the caller to handle the problem;
failing to do so can easily lead to inadvertent data corruption.
>   val coalesceBasePath = new Path(eventLog.basePath, coalesceTo)
>   val backupBasePath = new Path(eventLog.basePath, relocateTo)
>     eventLog.fs.mkdirs(backupBasePath)
>     for (path <- coalesced; time <- HDFSEventLog.timePart(path, eventType)) {
>       val backupPath = HDFSEventLog.path(backupBasePath, eventType, time)
>       log.info("Relocating " + path + " to " + backupPath)
>       eventLog.fs.rename(path, backupPath)
>     }
> INF [20100715-16:11:20.727] reporting: Relocating hdfs://localhost:9000/test-batchEventLog/metrics/metrics_1279226067707
to hdfs://localhost:9000/test-batchEventLog/pre-coalesce/metrics/metrics_1279226067707
> INF [20100715-16:11:20.752] reporting: Relocating hdfs://localhost:9000/test-batchEventLog/metrics/metrics_1279226077707
to hdfs://localhost:9000/test-batchEventLog/pre-coalesce/metrics/metrics_1279226077707
> INF [20100715-16:11:20.754] reporting: Relocating hdfs://localhost:9000/test-batchEventLog/metrics/metrics_1279226457707
to hdfs://localhost:9000/test-batchEventLog/pre-coalesce/metrics/metrics_1279226457707
> INF [20100715-16:11:20.757] reporting: Relocating hdfs://localhost:9000/test-batchEventLog/metrics/metrics_1279229126727
to hdfs://localhost:9000/test-batchEventLog/pre-coalesce/metrics/metrics_1279229126727
> Complete.
> [knuttycombe@floorshow reporting (reporting-coalesce)]$ hadoop fs -ls hdfs://localhost:9000/test-batchEventLog/
> Found 3 items
> drwxr-xr-x   - knuttycombe supergroup          0 2010-07-15 14:54 /test-batchEventLog/coalesced
> drwxr-xr-x   - knuttycombe supergroup          0 2010-07-15 14:35 /test-batchEventLog/metrics
> drwxr-xr-x   - knuttycombe supergroup          0 2010-07-15 16:11 /test-batchEventLog/pre-coalesce
> [knuttycombe@floorshow reporting (reporting-coalesce)]$ hadoop fs -ls hdfs://localhost:9000/test-batchEventLog/metrics
> Found 4 items
> -rw-r--r--   3 knuttycombe supergroup    2017122 2010-07-15 14:34 /test-batchEventLog/metrics/metrics_1279226067707
> -rw-r--r--   3 knuttycombe supergroup    4122951 2010-07-15 14:34 /test-batchEventLog/metrics/metrics_1279226077707
> -rw-r--r--   3 knuttycombe supergroup    5999912 2010-07-15 14:35 /test-batchEventLog/metrics/metrics_1279226457707
> -rw-r--r--   3 knuttycombe supergroup    8638301 2010-07-15 14:26 /test-batchEventLog/metrics/metrics_1279229126727
> [knuttycombe@floorshow reporting (reporting-coalesce)]$ hadoop fs -ls hdfs://localhost:9000/test-batchEventLog/pre-coalesce
> [knuttycombe@floorshow reporting (reporting-coalesce)]$



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message