avro-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ravi Gummadi (JIRA)" <j...@apache.org>
Subject [jira] Updated: (AVRO-61) Support for blocking arrays/maps in python
Date Thu, 13 Aug 2009 06:36:14 GMT

     [ https://issues.apache.org/jira/browse/AVRO-61?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Ravi Gummadi updated AVRO-61:

    Attachment: AVRO-61.patch

Attaching patch that adds support of reading blocking arrays/maps to python.

Please review and provide your comments.

> Support for blocking arrays/maps in python
> ------------------------------------------
>                 Key: AVRO-61
>                 URL: https://issues.apache.org/jira/browse/AVRO-61
>             Project: Avro
>          Issue Type: Bug
>          Components: python
>            Reporter: Thiruvalluvan M. G.
>            Assignee: Ravi Gummadi
>         Attachments: AVRO-61.patch
> We have introduced a blocking version of writer in Java that enables readers to efficiently
skip large arrays and maps. The avro format encoded arrays and maps by encoding the number
of elements by the the elements themselves. Zero element count indicates that the array/map
has ended. The change we introduced is that if the element count is negative, it is followed
by the byte-count of the encoded elements that follow. The reader, on seeing a negative element
count should flip the sign to get the actual number of elements. In addition, if it is interested
in supporting fast skip, it should use the byte count to skip the elements en-bloc instead
of decoding them individually. If it does not want to support fast skip, it has to just read
the byte count and ignore its value.
> The changes are already made in Java's ValueReader to support this. Similar changes need
to done in Python as well.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message