hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Nauroth (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (HDFS-8761) Windows HDFS daemon - datanode.DirectoryScanner: Error compiling report (...) XXX is not a prefix of YYY
Date Tue, 21 Jul 2015 03:29:04 GMT

     [ https://issues.apache.org/jira/browse/HDFS-8761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Chris Nauroth resolved HDFS-8761.
---------------------------------
    Resolution: Not A Problem

Hello [~odelalleau].

I answered your question on Stack Overflow.  I'm pasting the answer here too.  After using
the techniques I described to configure a path with a drive spec, I expect you won't see these
errors anymore.  In the future, the best forum for questions like this is the user@hadoop.apache.org
mailing list.

You can specify a drive spec in {{hadoop.tmp.dir}} in core-site.xml by prepending a '/' in
front of the absolute path, and using '/' as the path separator instead of '\' for all path
elements.  For example, if the desired absolute path is D:\tmp\hdp, then it would look like
this:

{code}
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/D:/tmp/hadoop</value>
    </property>
{code}

The reason this works is that the default values for many of the HDFS directories are configured
to be file://${hadoop.tmp.dir}/suffix.  See the default definitions of {{dfs.namenode.name.dir}},
{{dfs.datanode.data.dir}} and {{dfs.namenode.checkpoint.dir}} here:

http://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml

Substituting the above value for {{hadoop.tmp.dir}} yields a valid {{file:}} URL with a drive
spec and no authority, which satisfies the requirements for the HDFS configuration.  It's
important to use '/' instead of '\', because a bare unencoded '\' character is not valid in
URL syntax.

http://www.ietf.org/rfc/rfc1738.txt

If you prefer not to rely on this substitution behavior, then it's also valid to override
all configuration properties that make use of {{hadoop.tmp.dir}} within your hdfs-site.xml
file.  Each value must be a full {{file:}} URL.  For example:

{code}
    <property>
        <name>dfs.namenode.name.dir</name>
        <value>file:///D:/tmp/hadoop/dfs/name</value>
    </property>

    <property>
        <name>dfs.datanode.data.dir</name>
        <value>file:///D:/tmp/hadoop/dfs/data</value>
    </property>

    <property>
        <name>dfs.namenode.checkpoint.dir</name>
        <value>file:///D:/tmp/hadoop/dfs/namesecondary</value>
    </property>
{code}

You might find this more readable overall.

> Windows HDFS daemon - datanode.DirectoryScanner: Error compiling report (...) XXX is
not a prefix of YYY
> --------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-8761
>                 URL: https://issues.apache.org/jira/browse/HDFS-8761
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: HDFS
>    Affects Versions: 2.7.1
>         Environment: Windows 7, Java SDK 1.8.0_45
>            Reporter: Olivier Delalleau
>            Priority: Minor
>
> I'm periodically seeing errors like the one below output by the HDFS daemon (started
with start-dfs.cmd). This is with the default settings for data location (=not specified in
my hdfs-site.xml). I assume it may be fixable by specifying a path with the drive letter in
the config file, however I haven't be able to do that (see http://stackoverflow.com/questions/31353226/setting-hadoop-tmp-dir-on-windows-gives-error-uri-has-an-authority-component).
> 15/07/11 17:29:57 ERROR datanode.DirectoryScanner: Error compiling report
> java.util.concurrent.ExecutionException: java.lang.RuntimeException: \tmp\hadoop-odelalleau\dfs\data
is not a prefix of D:\tmp\hadoop-odelalleau\dfs\data\current\BP-1474392971-10.128.22.110-1436634926842\current\finalized\subdir0\subdir0\blk_1073741825
>         at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>         at java.util.concurrent.FutureTask.get(FutureTask.java:192)
>         at org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.getDiskReport(DirectoryScanner.java:566)
>         at org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.scan(DirectoryScanner.java:425)
>         at org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.reconcile(DirectoryScanner.java:406)
>         at org.apache.hadoop.hdfs.server.datanode.DirectoryScanner.run(DirectoryScanner.java:362)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message