crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tomáš Čechal (JIRA) <j...@apache.org>
Subject [jira] [Updated] (CRUNCH-577) NumberFormatException when parsing dfs.block.size
Date Tue, 10 Nov 2015 11:16:11 GMT

     [ https://issues.apache.org/jira/browse/CRUNCH-577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Tomáš Čechal updated CRUNCH-577:
--------------------------------
    Attachment: 0001-CRUNCH-577-Use-getLongBytes-to-correctly-parse-dfs-b.patch

One-liner that fixes the problem.

> NumberFormatException when parsing dfs.block.size
> -------------------------------------------------
>
>                 Key: CRUNCH-577
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-577
>             Project: Crunch
>          Issue Type: Bug
>          Components: IO
>    Affects Versions: 0.8.0, 0.8.1, 0.9.0, 0.8.2, 0.10.0, 0.8.3, 0.8.4, 0.11.0, 0.12.0
>            Reporter: Tomáš Čechal
>            Priority: Minor
>              Labels: patch
>         Attachments: 0001-CRUNCH-577-Use-getLongBytes-to-correctly-parse-dfs-b.patch
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> When using file size abbreviations (like "128m") for the HDFS configuration property
"dfs.block.size" the Crunch job crashes with a NumberFormatException. According to the Hadoop
documentation (https://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml),
this style of abbreviations should be supported.
> The problem occurs at line 38 in CrunchCombineFileInputFormat.java when the configuration
property is parsed using the getLong() method instead of getLongBytes() method. Furthermore,
obsolete configuration key "dfs.block.size" is used instead of "dfs.blocksize" (see https://issues.apache.org/jira/browse/HDFS-631)
which leads to a warning message being emitted when starting a MR pipeline.
> The proposed solution discussed on the crunch-users mailing list (http://mail-archives.apache.org/mod_mbox/crunch-user/201511.mbox/browser)
is to use the getLongBytes() method and the new config key "dfs.blocksize".



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message