hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Nauroth (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (HDFS-8761) Windows HDFS daemon - datanode.DirectoryScanner: Error compiling report (...) XXX is not a prefix of YYY
Date Tue, 21 Jul 2015 03:29:04 GMT

     [ https://issues.apache.org/jira/browse/HDFS-8761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Chris Nauroth resolved HDFS-8761.
    Resolution: Not A Problem

Hello [~odelalleau].

I answered your question on Stack Overflow.  I'm pasting the answer here too.  After using
the techniques I described to configure a path with a drive spec, I expect you won't see these
errors anymore.  In the future, the best forum for questions like this is the user@hadoop.apache.org
mailing list.

You can specify a drive spec in {{hadoop.tmp.dir}} in core-site.xml by prepending a '/' in
front of the absolute path, and using '/' as the path separator instead of '\' for all path
elements.  For example, if the desired absolute path is D:\tmp\hdp, then it would look like


The reason this works is that the default values for many of the HDFS directories are configured
to be file://${hadoop.tmp.dir}/suffix.  See the default definitions of {{dfs.namenode.name.dir}},
{{dfs.datanode.data.dir}} and {{dfs.namenode.checkpoint.dir}} here:


Substituting the above value for {{hadoop.tmp.dir}} yields a valid {{file:}} URL with a drive
spec and no authority, which satisfies the requirements for the HDFS configuration.  It's
important to use '/' instead of '\', because a bare unencoded '\' character is not valid in
URL syntax.


If you prefer not to rely on this substitution behavior, then it's also valid to override
all configuration properties that make use of {{hadoop.tmp.dir}} within your hdfs-site.xml
file.  Each value must be a full {{file:}} URL.  For example:




You might find this more readable overall.

> Windows HDFS daemon - datanode.DirectoryScanner: Error compiling report (...) XXX is
not a prefix of YYY
> --------------------------------------------------------------------------------------------------------
>                 Key: HDFS-8761
>                 URL: https://issues.apache.org/jira/browse/HDFS-8761
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: HDFS
>    Affects Versions: 2.7.1
>         Environment: Windows 7, Java SDK 1.8.0_45
>            Reporter: Olivier Delalleau
>            Priority: Minor
> I'm periodically seeing errors like the one below output by the HDFS daemon (started
with start-dfs.cmd). This is with the default settings for data location (=not specified in
my hdfs-site.xml). I assume it may be fixable by specifying a path with the drive letter in
the config file, however I haven't be able to do that (see http://stackoverflow.com/questions/31353226/setting-hadoop-tmp-dir-on-windows-gives-error-uri-has-an-authority-component).
> 15/07/11 17:29:57 ERROR datanode.DirectoryScanner: Error compiling report
> java.util.concurrent.ExecutionException: java.lang.RuntimeException: \tmp\hadoop-odelalleau\dfs\data
is not a prefix of D:\tmp\hadoop-odelalleau\dfs\data\current\BP-1474392971-\current\finalized\subdir0\subdir0\blk_1073741825
>         at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>         at java.util.concurrent.FutureTask.get(FutureTask.java:192)
>         at org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.getDiskReport(DirectoryScanner.java:566)
>         at org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.scan(DirectoryScanner.java:425)
>         at org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.reconcile(DirectoryScanner.java:406)
>         at org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.run(DirectoryScanner.java:362)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

This message was sent by Atlassian JIRA

View raw message