Mailing-List: contact dev-help@gora.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@gora.apache.org
Date: Mon, 19 Jan 2015 21:46:36 +0000 (UTC)
From: =?utf-8?Q?Renato_Javier_Marroqu=C3=ADn_Mogrovejo_=28JIRA=29?=
 <jira@apache.org>
To: dev@gora.apache.org
Message-ID: <JIRA.12756937.1416564574000.118886.1421703996272@Atlassian.JIRA>
In-Reply-To: <JIRA.12756937.1416564574000@Atlassian.JIRA>
References: <JIRA.12756937.1416564574000@Atlassian.JIRA>
 <JIRA.12756937.1416564574829@arcas>
Subject: [jira] [Commented] (GORA-401) Serialization and deserialization of
 Persistent does not hold the entity dirty state from Map to Reduce
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


    [ https://issues.apache.org/jira/browse/GORA-401?page=3Dcom.atlassian.j=
ira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D142830=
51#comment-14283051 ]=20

Renato Javier Marroqu=C3=ADn Mogrovejo commented on GORA-401:
--------------------------------------------------------

Thanks [~alfonso.nishikawa]! Understanding things make me happier :)  So ar=
e you currently working on this? Any ideas on how you will do this?

> Serialization and deserialization of Persistent does not hold the entity =
dirty state from Map to Reduce
> -------------------------------------------------------------------------=
------------------------------
>
>                 Key: GORA-401
>                 URL: https://issues.apache.org/jira/browse/GORA-401
>             Project: Apache Gora
>          Issue Type: Bug
>          Components: gora-core
>    Affects Versions: 0.4, 0.5
>         Environment: Tested on gora-0.4, but seems logically to hold on g=
ora-0.5. HBase backend.
>            Reporter: Alfonso Nishikawa
>            Assignee: Alfonso Nishikawa
>            Priority: Critical
>              Labels: serialization
>         Attachments: GORA-401-tests.patch
>
>   Original Estimate: 35h
>          Time Spent: 6.5h
>  Remaining Estimate: 28.5h
>
> After removing __g__dirty field in GORA-326, dirty field is not serialize=
d. In GORA-321 {{[PersistentSerializer|https://github.com/apache/gora/blob/=
master/gora-core/src/main/java/org/apache/gora/mapreduce/PersistentSerializ=
er.java]}} went from using {{[PersistentDatumWriter|https://github.com/apac=
he/gora/blob/apache-gora-0.3/gora-core/src/main/java/org/apache/gora/avro/P=
ersistentDatumWriter.java](/Reader)}} to Avro's {{SpecificDatumWriter}}, de=
legating the serialization of the dirty field to Avro (but really not desir=
able to have that field as a main field in the entities).
> The proposal is to reintroduce the {{PersistentDatumWriter/Reader}} which=
 will serialize the internal fields of the entities.
> This bug affects, for example, Nutch, which loads only some fields in it'=
s phases, serializes entities (from Map to Reduce), and when deserializes f=
inds all fields as "dirty", independently of what fields were modified in t=
he Map, and overwrite all data in datastore (deleting much things: download=
ed content, parsed content, etc).
> This effect can be seen in {{TestPersistentSerialization#testSerderEmploy=
eeTwoFields}}, when debuging in {{TestIOUtils#testSerializeDeserialize}}. P=
roper breakpoints an inspections shows that, entities are "equal" when it's=
 fields are equal. This is fine as "equal" definition, but another test mus=
t be added to check that serialization an deserialization keeps the dirty s=
tate.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)