avro-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matt Stevenson <matt.r.steven...@gmail.com>
Subject Re: How to extract byte array from memoryOutputStream
Date Thu, 17 May 2012 16:29:18 GMT
>
> But what is the suggested way of transferring encoded bytes over the
> wire. Avro does not seem to expose the encoded bytes anywhere.
>

I am using a custom method for transporting the encoding bytes, which is
why I had to get at the bytes.
Avro has an HTTP transport mechanisms defined, but I haven't used it and I
don't know if it is fully implemented in the C++ project.
For typical Avro usage you should not be accessing the raw bytes, which is
why nobody bothered to expose them.

So even if I am encoding a small dataset, I would end up
> reading a min "Chunk_size" of memory block.
>

The "count" variable should be the exact number of bytes in the stream.
Both for loops are checking "i<count", and "i" is only incremented in the
inner loop.  Once you reach the count it will break out of both loops.

On Wed, May 16, 2012 at 4:45 AM, Gaurav Nanda <gaurav324@gmail.com> wrote:

> Hi Matt, The approach you are using to read the written bytes would
> end up reading all the bytes allocated but not the bytes actually
> written. So even if I am encoding a small dataset, I would end up
> reading a min "Chunk_size" of memory block.
>
> On Mon, May 14, 2012 at 9:03 PM, Gaurav Nanda <gaurav324@gmail.com> wrote:
> > Thanks Matt !
> >
> > But what is the suggested way of transferring encoded bytes over the
> > wire. Avro does not seem to expose the encoded bytes anywhere.
> >
> > - Gaurav Nanda
> >
> > On Sat, May 12, 2012 at 6:29 AM, Matt Stevenson
> > <matt.r.stevenson@gmail.com> wrote:
> >> I made a copy of the MemoryOutputStream with a header, so you can get
> at the
> >> bytes.
> >>
> >> http://branchingworlds.com/avro/OpenMemoryOutputStream.h
> >> http://branchingworlds.com/avro/OpenMemoryOutputStream.cpp
> >>
> >> Add those to your project.
> >> include "OpenMemoryOutputStream.h" and call openMemoryOutputStream()
> rather
> >> than memoryOutputStream().
> >>
> >> The memory is stored in chunks.  Here's an example of iterating through
> it:
> >>
> >>> auto_ptr<OpenMemoryOutputStream> os = openMemoryOutputStream();
> >>> EncoderPtr e = binaryEncoder();
> >>> e->init(*os);
> >>>
> >>> avro::encode(*e, *t);
> >>> e->flush();
> >>>
> >>> int count = os->byteCount();
> >>> char* data = new char[count];
> >>> int i=0;
> >>> for (std::vector<uint8_t*>::const_iterator it = os->data_.begin();
it
> !=
> >>> os->data_.end() && i<count; ++it) {
> >>>     uint8_t* chunk = *it;
> >>>     int size = os->chunkSize_;
> >>>     for(int j=0; j<size && i<count; j++, i++){
> >>>         data[i] = chunk[j];
> >>>     }
> >>> }
> >>
> >>
> >> On Thu, May 10, 2012 at 12:19 PM, Gaurav Nanda <gaurav324@gmail.com>
> wrote:
> >>>
> >>> I want to extract byte array from memoryOutputStream to transfer the
> >>> encoded data. How do I achieve that?
> >>
> >>
> >>
> >>
> >> --
> >> Matt Stevenson.
>



-- 
Matt Stevenson.

Mime
View raw message