Return-Path: X-Original-To: apmail-avro-user-archive@www.apache.org Delivered-To: apmail-avro-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 36952CD73 for ; Thu, 17 May 2012 16:36:25 +0000 (UTC) Received: (qmail 97985 invoked by uid 500); 17 May 2012 16:36:25 -0000 Delivered-To: apmail-avro-user-archive@avro.apache.org Received: (qmail 97937 invoked by uid 500); 17 May 2012 16:36:24 -0000 Mailing-List: contact user-help@avro.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@avro.apache.org Delivered-To: mailing list user@avro.apache.org Received: (qmail 97927 invoked by uid 99); 17 May 2012 16:36:24 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 May 2012 16:36:24 +0000 X-ASF-Spam-Status: No, hits=1.7 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of gaurav324@gmail.com designates 209.85.213.171 as permitted sender) Received: from [209.85.213.171] (HELO mail-yx0-f171.google.com) (209.85.213.171) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 May 2012 16:36:17 +0000 Received: by yenq11 with SMTP id q11so2484570yen.30 for ; Thu, 17 May 2012 09:35:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=gmB8OqB3B84MJ0lhyR2I9aNw2tUbVHAfLkB5MGthdss=; b=jZhUbTWYGgCQFLbwet6Rw5Wnt+l46ZTYlmKhC9/jaLQPfxerMmSsrVfQoT+3I+aeI2 CXgNc14B3XNv/1QGFkAdg96JXN6OR/HaciWijk1pU5Zc+e5URQOKY0/LXeqTU2MAtDRt deXLquIWP9uXdIoHDsmHbzOQbG+01eezdvhTcWU3eZ7H5jd+zAE3HsrCWVrFUVRe666S nWvAJ70Ahv2LqkcWkU/zLSanF91b0JR1J03Q3nz8z8wrx/Q49BeVYWr1aL/0OPa0IOEF NL5ovhVKK5jSbj6UG9S3Lu/BtEZ2FFLKT3HIrWfsK/XD3G9oX+2v/QCfJjbb4e9g1tte GM6w== MIME-Version: 1.0 Received: by 10.60.172.231 with SMTP id bf7mr7136093oec.45.1337272556752; Thu, 17 May 2012 09:35:56 -0700 (PDT) Received: by 10.60.11.138 with HTTP; Thu, 17 May 2012 09:35:56 -0700 (PDT) In-Reply-To: References: Date: Thu, 17 May 2012 22:05:56 +0530 Message-ID: Subject: Re: How to extract byte array from memoryOutputStream From: Gaurav Nanda To: user@avro.apache.org Content-Type: multipart/alternative; boundary=bcaec54c4df6b68d2b04c03e070c --bcaec54c4df6b68d2b04c03e070c Content-Type: text/plain; charset=ISO-8859-1 Got it Matt. Thanks ! On Thu, May 17, 2012 at 9:59 PM, Matt Stevenson wrote: > But what is the suggested way of transferring encoded bytes over the >> wire. Avro does not seem to expose the encoded bytes anywhere. >> > > I am using a custom method for transporting the encoding bytes, which is > why I had to get at the bytes. > Avro has an HTTP transport mechanisms defined, but I haven't used it and I > don't know if it is fully implemented in the C++ project. > For typical Avro usage you should not be accessing the raw bytes, which is > why nobody bothered to expose them. > > So even if I am encoding a small dataset, I would end up >> reading a min "Chunk_size" of memory block. >> > > The "count" variable should be the exact number of bytes in the stream. > Both for loops are checking "i inner loop. Once you reach the count it will break out of both loops. > > > On Wed, May 16, 2012 at 4:45 AM, Gaurav Nanda wrote: > >> Hi Matt, The approach you are using to read the written bytes would >> end up reading all the bytes allocated but not the bytes actually >> written. So even if I am encoding a small dataset, I would end up >> reading a min "Chunk_size" of memory block. >> >> On Mon, May 14, 2012 at 9:03 PM, Gaurav Nanda >> wrote: >> > Thanks Matt ! >> > >> > But what is the suggested way of transferring encoded bytes over the >> > wire. Avro does not seem to expose the encoded bytes anywhere. >> > >> > - Gaurav Nanda >> > >> > On Sat, May 12, 2012 at 6:29 AM, Matt Stevenson >> > wrote: >> >> I made a copy of the MemoryOutputStream with a header, so you can get >> at the >> >> bytes. >> >> >> >> http://branchingworlds.com/avro/OpenMemoryOutputStream.h >> >> http://branchingworlds.com/avro/OpenMemoryOutputStream.cpp >> >> >> >> Add those to your project. >> >> include "OpenMemoryOutputStream.h" and call openMemoryOutputStream() >> rather >> >> than memoryOutputStream(). >> >> >> >> The memory is stored in chunks. Here's an example of iterating >> through it: >> >> >> >>> auto_ptr os = openMemoryOutputStream(); >> >>> EncoderPtr e = binaryEncoder(); >> >>> e->init(*os); >> >>> >> >>> avro::encode(*e, *t); >> >>> e->flush(); >> >>> >> >>> int count = os->byteCount(); >> >>> char* data = new char[count]; >> >>> int i=0; >> >>> for (std::vector::const_iterator it = os->data_.begin(); it >> != >> >>> os->data_.end() && i> >>> uint8_t* chunk = *it; >> >>> int size = os->chunkSize_; >> >>> for(int j=0; j> >>> data[i] = chunk[j]; >> >>> } >> >>> } >> >> >> >> >> >> On Thu, May 10, 2012 at 12:19 PM, Gaurav Nanda >> wrote: >> >>> >> >>> I want to extract byte array from memoryOutputStream to transfer the >> >>> encoded data. How do I achieve that? >> >> >> >> >> >> >> >> >> >> -- >> >> Matt Stevenson. >> > > > > -- > Matt Stevenson. > --bcaec54c4df6b68d2b04c03e070c Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Got it Matt. Thanks !

On Thu, May 17, 201= 2 at 9:59 PM, Matt Stevenson <matt.r.stevenson@gmail.com><= /span> wrote:
But what is the suggested way of transferring encoded bytes over the
wire. Avro does not seem to expose the encoded bytes anywhere.

I am using a custom method for transporting the encoding= bytes, which is why I had to get at the bytes.
Avro has an HTTP transpo= rt mechanisms defined, but I haven't used it and I don't know if it= is fully implemented in the C++ project.
For typical Avro usage you should not be accessing the raw bytes, which is = why nobody bothered to expose them.

So even if I am encoding a small dataset, I would end up
reading a min "Chunk_size" of memory block.
<= /div>

The "count" variable should be the ex= act number of bytes in the stream.=A0 Both for loops are checking "i&l= t;count", and "i" is only incremented in the inner loop.=A0 = Once you reach the count it will break out of both loops.


On Wed, May 16, 2012 at 4:45 AM, Gaurav Nand= a <gaurav324@gmail.com> wrote:
Hi Matt, The approach you are using to read the written bytes would
end up reading all the bytes allocated but not the bytes actually
written. So even if I am encoding a small dataset, I would end up
reading a min "Chunk_size" of memory block.

On Mon, May 14, 2012 at 9:03 PM, Gaurav Nanda <gaurav324@gmail.com> wrote:
> Thanks Matt !
>
> But what is the suggested way of transferring encoded bytes over the > wire. Avro does not seem to expose the encoded bytes anywhere.
>
> - Gaurav Nanda
>
> On Sat, May 12, 2012 at 6:29 AM, Matt Stevenson
> <ma= tt.r.stevenson@gmail.com> wrote:
>> I made a copy of the MemoryOutputStream with a header, so you can = get at the
>> bytes.
>>
>> http://branchingworlds.com/avro/OpenMemoryOutputStream= .h
>> http://branchingworlds.com/avro/OpenMemoryOutputStre= am.cpp
>>
>> Add those to your project.
>> include "OpenMemoryOutputStream.h" and call openMemoryOu= tputStream() rather
>> than memoryOutputStream().
>>
>> The memory is stored in chunks.=A0 Here's an example of iterat= ing through it:
>>
>>> auto_ptr<OpenMemoryOutputStream> os =3D openMemoryOutput= Stream();
>>> EncoderPtr e =3D binaryEncoder();
>>> e->init(*os);
>>>
>>> avro::encode(*e, *t);
>>> e->flush();
>>>
>>> int count =3D os->byteCount();
>>> char* data =3D new char[count];
>>> int i=3D0;
>>> for (std::vector<uint8_t*>::const_iterator it =3D os->= ;data_.begin(); it !=3D
>>> os->data_.end() && i<count; ++it) {
>>> =A0=A0=A0 uint8_t* chunk =3D *it;
>>> =A0=A0=A0 int size =3D os->chunkSize_;
>>> =A0=A0=A0 for(int j=3D0; j<size && i<count; j++,= i++){
>>> =A0=A0=A0=A0=A0=A0=A0 data[i] =3D chunk[j];
>>> =A0=A0=A0 }
>>> }
>>
>>
>> On Thu, May 10, 2012 at 12:19 PM, Gaurav Nanda <gaurav324@gmail.com> wrote= :
>>>
>>> I want to extract byte array from memoryOutputStream to transf= er the
>>> encoded data. How do I achieve that?
>>
>>
>>
>>
>> --
>> Matt Stevenson.



--
Matt Stevenson.

--bcaec54c4df6b68d2b04c03e070c--