Mailing-List: contact hadoop-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hadoop-dev@lucene.apache.org
Message-ID: <4687944.1181799027244.JavaMail.jira@brutus>
Date: Wed, 13 Jun 2007 22:30:27 -0700 (PDT)
From: "Hairong Kuang (JIRA)" <jira@apache.org>
To: hadoop-dev@lucene.apache.org
Subject: [jira] Commented: (HADOOP-1470) Rework FSInputChecker and
 FSOutputSummer to support checksum code sharing between ChecksumFileSystem
 and block level crc dfs
In-Reply-To: <7723731.1181157326809.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


    [ https://issues.apache.org/jira/browse/HADOOP-1470?page=3Dcom.atlassia=
n.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12504542 ]=20

Hairong Kuang commented on HADOOP-1470:
---------------------------------------

Can not stop thinking because I feel the same as Doug that seek could be ge=
nerialized. :-)

It looks like the current proposed FSInputChecker could be broken into two =
classes:

1. ChecksumFSInputStream is an abstract of datas and checksums or an interl=
eaved stream as in DFS.  ChecksumFSInputStream extends FSInputStream and ha=
s checksum-related abstract methods: readChunk, readChecksum, getChunkBound=
ary, nextChunkSize, seekToNewSource, and reportChecksumError etc.

2. The generic checker FSInputChecker is a concrete class. It contains a Ch=
ecksumFSInputStream, extends FSInputStream, and provides generic implementa=
tion of methods: read, seek, skip etc.

Every checksum file system should implement ChecksumFSInputStream, while ev=
ery non checksum file system implements FSInputStream. So  FSDataInputStrea=
ms contains a FSInputChecker for checksum file systems or a FSInputStream f=
or non checksumed file system.=20

I am still not clear how this approach supports change of Checksum between =
blocks. Any thoughts?


> Rework FSInputChecker and FSOutputSummer to support checksum code sharing=
 between ChecksumFileSystem and block level crc dfs
> -------------------------------------------------------------------------=
---------------------------------------------------
>
>                 Key: HADOOP-1470
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1470
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 0.12.3
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.14.0
>
>         Attachments: genericChecksum.patch, InputChecker-01.java
>
>
> Comment from Doug in HADOOP-1134:
> I'd prefer it if the CRC code could be shared with CheckSumFileSystem. In=
 particular, it seems to me that FSInputChecker and FSOutputSummer could be=
 extended to support pluggable sources and sinks for checksums, respectivel=
y, and DFSDataInputStream and DFSDataOutputStream could use these. Advantag=
es of this are: (a) single implementation of checksum logic to debug and ma=
intain; (b) keeps checksumming as close to possible to data generation and =
use. This patch computes checksums after data has been buffered, and valida=
tes them before it is buffered. We sometimes use large buffers and would li=
ke to guard against in-memory errors. The current checksum code catches a l=
ot of such errors. So we should compute checksums after minimal buffering (=
just bytesPerChecksum, ideally) and validate them at the last possible mome=
nt (e.g., through the use of a small final buffer with a larger buffer behi=
nd it). I do not think this will significantly affect performance, and data=
 integrity is a high priority.=20

--=20
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.