Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Date: Fri, 17 Mar 2017 14:44:41 +0000 (UTC)
From: =?utf-8?Q?L=C3=A1szl=C3=B3_Bence_Nagy_=28JIRA=29?= <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.13057016.1489761868000.49736.1489761881725@Atlassian.JIRA>
In-Reply-To: <JIRA.13057016.1489761868000@Atlassian.JIRA>
References: <JIRA.13057016.1489761868000@Atlassian.JIRA> <JIRA.13057016.1489761868881@jira-lw-us.apache.org>
Subject: [jira] [Created] (HDFS-11542) Fix RawErasureCoderBenchmark decoding
 operation
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
archived-at: Fri, 17 Mar 2017 14:44:49 -0000

L=C3=A1szl=C3=B3 Bence Nagy created HDFS-11542:
----------------------------------------

             Summary: Fix RawErasureCoderBenchmark decoding operation
                 Key: HDFS-11542
                 URL: https://issues.apache.org/jira/browse/HDFS-11542
             Project: Hadoop HDFS
          Issue Type: Sub-task
          Components: erasure-coding
    Affects Versions: 3.0.0-alpha2
            Reporter: L=C3=A1szl=C3=B3 Bence Nagy
            Priority: Minor


There are some issues with the decode operation in the *RawErasureCoderBenc=
hmark.java* file. The decoding method is called like this: *decoder.decode(=
decodeInputs, ERASED_INDEXES, outputs);*.=20

Using RS 6+3 configuration it could be called with these parameters correct=
ly like this: *decode([ d0, NULL, d2, d3, NULL, d5, p0, NULL, p2 ], [ 1, 4,=
 7 ], [ -, -, - ])*. The 1,4,7 indexes are in the *ERASED_INDEXES* array so=
 in the *decodeInputs* array the values at those indexes are set to NULL, a=
ll other data and parity packets are present in the array. The *outputs* ar=
ray's length is 3, where the d1, d4 and p1 packets should be reconstructed.=
 This would be the right solution.

Right now this example would be called like this: *decode([ d0, d1, d2, d3,=
 d4, d5, -, -, - ], [ 1, 4, 7 ], [ -, -, - ])*. So it has two main problems=
 with the *decodeInputs* array. Firstly, the packets are not set to NULL wh=
ere they should be based on the *ERASED_INDEXES* array. Secondly, it does n=
ot have any parity packets for decoding.

The first problem is easy to solve, the values at the proper indexes need t=
o be set to NULL. The latter one is a little harder because right now multi=
ple rounds of encode operations are done one after another and similarly mu=
ltiple decode operations are called one by one. Encode and decode pairs sho=
uld be called one after another so that the encoded parity packets can be u=
sed in the *decodeInputs* array as a parameter for decode. (Of course, thei=
r performance should be still measured separately.)

Moreover, there is one more problem in this file. Right now it works with R=
S 6+3 and the *ERASED_INDEXES* array is fixed to *[ 6, 7, 8 ]*. So the thre=
e parity packets are needed to be reconstructed. This means that no real de=
code performance is measured because no data packet is needed to be reconst=
ructed (even if the decode works properly). Actually, only new parity packe=
ts are needed to be encoded. The exact implementation depends on the underl=
ying erasure coding plugin, but the point is that data packets should also =
be erased to measure real decode performance.

In addition to this, more RS configurations (not just 6+3) could be measure=
d as well to be able to compare them.


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org