hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tsz Wo (Nicholas), SZE (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-1425) archive throws OutOfMemoryError
Date Fri, 05 Feb 2010 18:13:28 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12830207#action_12830207
] 

Tsz Wo (Nicholas), SZE commented on MAPREDUCE-1425:
---------------------------------------------------

After the patch, there are some improvement but archive still uses double memory of ls.
- archive
{noformat}
 num     #instances         #bytes  class name
----------------------------------------------
   1:        658875       42680832  [C
   2:       1434792       34435008  java.lang.String
   3:        255174       20413920  java.net.URI
   4:        255163       16330432  org.apache.hadoop.fs.FileStatus
   5:        200001        4800024  org.apache.hadoop.fs.permission.FsPermission
   6:        255172        4082752  org.apache.hadoop.fs.Path
{noformat}

- ls
{noformat}
 num     #instances         #bytes  class name
----------------------------------------------
   1:        304186       21086344  [C
   2:        804264       19302336  java.lang.String
   3:        100009        8000720  java.net.URI
   4:        100001        6400064  org.apache.hadoop.fs.FileStatus
   5:        100002        2400048  org.apache.hadoop.fs.permission.FsPermission
   6:        100008        1600128  org.apache.hadoop.fs.Path
{noformat}

> archive throws OutOfMemoryError
> -------------------------------
>
>                 Key: MAPREDUCE-1425
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1425
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: harchive
>            Reporter: Tsz Wo (Nicholas), SZE
>            Assignee: Mahadev konar
>             Fix For: 0.22.0
>
>         Attachments: har.sh, m1425_20100129TextFileGenerator.patch, MAPREDUCE-1425.patch
>
>
> {noformat}
> -bash-3.1$ hadoop  archive -archiveName t4.har -p . t4 .
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
>         at java.util.regex.Pattern.compile(Pattern.java:1432)
>         at java.util.regex.Pattern.<init>(Pattern.java:1133)
>         at java.util.regex.Pattern.compile(Pattern.java:847)
>         at java.lang.String.replace(String.java:2208)
>         at org.apache.hadoop.fs.Path.normalizePath(Path.java:146)
>         at org.apache.hadoop.fs.Path.initialize(Path.java:137)
>         at org.apache.hadoop.fs.Path.<init>(Path.java:126)
>         at org.apache.hadoop.fs.Path.makeQualified(Path.java:296)
>         at org.apache.hadoop.hdfs.DistributedFileSystem.makeQualified(DistributedFileSystem.java:244)
>         at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:256)
>         at org.apache.hadoop.tools.HadoopArchives.archive(HadoopArchives.java:393)
>         at org.apache.hadoop.tools.HadoopArchives.run(HadoopArchives.java:736)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>         at org.apache.hadoop.tools.HadoopArchives.main(HadoopArchives.java:751)
> {noformat}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message