hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "dhruba borthakur (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-1989) Add support for simulated Data Nodes - helpful for testing and performance benchmarking of the Name Node without having a large cluster
Date Thu, 15 Nov 2007 01:04:43 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-1989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12542648

dhruba borthakur commented on HADOOP-1989:

I still get a merge failure on DataChecksum.java. I managed to merge it by hand. I am running
the unit test and saw a test failure on TestCrcCorruption:

Testcase: testCrcCorruption took 12.692 sec
    Caused an ERROR
Java heap space
java.lang.OutOfMemoryError: Java heap space
    at org.apache.hadoop.fs.FSInputChecker.set(FSInputChecker.java:396)
    at org.apache.hadoop.fs.FSInputChecker.<init>(FSInputChecker.java:71)
    at org.apache.hadoop.dfs.DFSClient$BlockReader.<init>(DFSClient.java:697)
    at org.apache.hadoop.dfs.DFSClient$BlockReader.newBlockReader(DFSClient.java:755)
    at org.apache.hadoop.dfs.DFSClient$DFSInputStream.fetchBlockByteRange(DFSClient.java:1144)
    at org.apache.hadoop.dfs.DFSClient$DFSInputStream.read(DFSClient.java:1211)
    at org.apache.hadoop.fs.FSInputStream.readFully(FSInputStream.java:66)
    at org.apache.hadoop.fs.FSDataInputStream.readFully(FSDataInputStream.java:56)
    at org.apache.hadoop.dfs.DFSTestUtil.checkFiles(DFSTestUtil.java:150)
    at org.apache.hadoop.dfs.TestCrcCorruption.thistest(TestCrcCorruption.java:181)
    at org.apache.hadoop.dfs.TestCrcCorruption.testCrcCorruption(TestCrcCorruption.java:223)

> Add support for simulated Data Nodes  - helpful for testing and performance benchmarking
of the Name Node without having a large cluster
> ----------------------------------------------------------------------------------------------------------------------------------------
>                 Key: HADOOP-1989
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1989
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: dfs
>    Affects Versions: 0.16.0
>            Reporter: Sanjay Radia
>            Assignee: Sanjay Radia
>            Priority: Minor
>             Fix For: 0.16.0
>         Attachments: SimulatedStoragePatchSubmit.txt, SimulatedStoragePatchSubmit5.txt,
SimulatedStoragePatchSubmit6.txt, SimulatedStoragePatchSubmit7.txt, SimulatedStoragePatchSubmit8.txt
> Proposal is to add an implementation for a Simulated Data Node.
> This will 
>   - allow one to test certain parts of the system (especially the Name Node, protocols)
much more easily and efficiently.
>   - allow one to run performance benchmarks on the Name node without having a large cluster.
>   - Inject faults for testing (e.g. one can add random faults based probability parameters).
> The idea is that the Simulated Data Node will
>  - discard any data written to blocks (but remember the blocks and their sizes)
>  - generate fixed data on the fly when blocks are read (e.g. block is fixed set of bytes
or repeated sequence of strings).
> The Simulated Data Node can also be used for fault injection.
> The data node can be parameterized with probabilities that allow one to control:
>   - Delays on reads and writes, creates, etc
>   - IO Exceptions
>  - Loss of blocks 
>  - Failures

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message