From user-return-192-archive-asf-public=cust-asf.ponee.io@orc.apache.org Thu Feb 8 06:58:29 2018 Return-Path: X-Original-To: archive-asf-public@eu.ponee.io Delivered-To: archive-asf-public@eu.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by mx-eu-01.ponee.io (Postfix) with ESMTP id EC06518064F for ; Thu, 8 Feb 2018 06:58:29 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id D31D9160C5C; Thu, 8 Feb 2018 05:58:29 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id F0F40160C5B for ; Thu, 8 Feb 2018 06:58:28 +0100 (CET) Received: (qmail 66295 invoked by uid 500); 8 Feb 2018 05:58:28 -0000 Mailing-List: contact user-help@orc.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@orc.apache.org Delivered-To: mailing list user@orc.apache.org Received: (qmail 66281 invoked by uid 99); 8 Feb 2018 05:58:27 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 08 Feb 2018 05:58:27 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 445CA1A09FD for ; Thu, 8 Feb 2018 05:58:26 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.379 X-Spam-Level: ** X-Spam-Status: No, score=2.379 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, KAM_NUMSUBJECT=0.5, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id 9HXLlYziAan9 for ; Thu, 8 Feb 2018 05:58:25 +0000 (UTC) Received: from mail-ot0-f180.google.com (mail-ot0-f180.google.com [74.125.82.180]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id BCF335F125 for ; Thu, 8 Feb 2018 05:58:24 +0000 (UTC) Received: by mail-ot0-f180.google.com with SMTP id q9so3169031oti.0 for ; Wed, 07 Feb 2018 21:58:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=0xXfshvPZzMJIB3jCzgPeXFDPpIDec73lrNhYlGluIk=; b=Qafgew1GTzHQwqPTPnAvMkIqjKMmukMHzBIWZ4xCyRbR0Z/6U/C4fkpoWMjMjvcQYz 6/UVuXk0tfWdKx+npDDEM69YhDGi7kYdlfL8aMu3nN9p0wLl7veLX51LR8owVJlP4NgD 0p4uVW+y7T5eMitwC4sSjZPZRHBc0sBp+fLQds5z+SgAGDEqFiODEWR940vL73gWjOdz 1D4MVkOD7NT0UY4EDMATIPB3CibQ7IApdfMx5FonUH/TnFx44+9M0+Yrtn884tjqbXHw Y+FbF6YCMfIGP/zyJ1IqhcaSU5YXHpSrUsiriXR/ef0YZTEINbiQmpwu56fR77svr37p niDA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=0xXfshvPZzMJIB3jCzgPeXFDPpIDec73lrNhYlGluIk=; b=tLUIQGsKe9JBGQYegkg662U0HNZDeF2Bx9dU5snUNOTSFwHz3u2Esd+mXALgRevyvq Bv25mqzeGl5Dgj/Mbo5vDbUkvBAR5CGk9qEQ581DYTJIuI4ijTNg+r/+s827WQVWTbsc 527VWj6taDmgJ4k9hL4F2Z54NmuLTuqBkoaKY7uspU3JyUHa2fxzkhSk675Pd5rPOqna 5Z5TB+G4IL9KWlrNiwzM92IrEG58tPcRov1Ib9rL1LlUvhvQhXlsHryKkTH+dWA9jmWV QRWekwMEMTTwODJqfrnUEEzKBYEhkV369S1EKxcYRSMOfYp0HvZfP4p02IPGq4TVG3D4 K/tg== X-Gm-Message-State: APf1xPAeePseU9fVKEI5d515R/PLM9mS/b6u4Q+wkaGNfat1CBjfz3gw kIpRmpvD8lvDi2Xw0IQrq5+FMO4uRGkYaaFnm1ZqhLh9 X-Google-Smtp-Source: AH8x226XTCqGo768tUGxM0HvnJmyeJpZxAs3yP89qtUzsoPIOlzSAgkMWHUy637wdgBsU3DvlgTMDcFBxWcaISMMRvc= X-Received: by 10.157.53.5 with SMTP id o5mr6174980otc.181.1518069503579; Wed, 07 Feb 2018 21:58:23 -0800 (PST) MIME-Version: 1.0 Received: by 10.157.29.6 with HTTP; Wed, 7 Feb 2018 21:58:23 -0800 (PST) In-Reply-To: References: From: "Owen O'Malley" Date: Wed, 7 Feb 2018 21:58:23 -0800 Message-ID: Subject: Re: Compressed data format for LZ4 To: user@orc.apache.org Content-Type: multipart/alternative; boundary="001a11c046c4815c9e0564ad19ec" --001a11c046c4815c9e0564ad19ec Content-Type: text/plain; charset="UTF-8" Hi David, In general this is probably better on dev@orc, but this works. ORC-77 (62fe9504b) implemented the LZ4 codec using airlift. The structure is the same as the other codecs and it always uses a 3 byte header (#2). .. Owen On Wed, Feb 7, 2018 at 9:41 PM, David Phillips wrote: > We previously preserved an LZ4 CompressionKind and plan to implement it in > the Presto reader and writer. Unlikely Snappy, the LZ4 format does not > record the uncompressed length. Thus, when reading, we need to allocate an > output buffer that is the full compressionBlockSize. This can waste a lot > of memory when there are many streams and many open readers. > > We propose to prefix the LZ4 block with the uncompressed size. I see a few > ways of doing it: > > 1) Variable length integer, the same as Snappy. > 2) Fixed 3-byte integer, little-endian. > 3) Fixed 4-byte integer, little-endian. > > Option #1 is more complicated, uses more CPU to decode, and probably > doesn't save much space; buffers starting at 16kB will use 3 bytes. > > Option #2 restricts the maximum size to be 16MB-1 byte. This is > ridiculously large for a per-stream buffer and not a problem as current > writers cap the buffer size at a reasonable 256kB, so it shouldn't be a > problem in practice, but it's worth calling out here. > > Option #3 is flexible but in practice will waste a byte. > > My vote is for option #2. > > --001a11c046c4815c9e0564ad19ec Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi David,
=C2=A0 =C2=A0In general this is probably bet= ter on dev@orc, but this works. ORC-77 (62fe9504b) implemented the LZ4 code= c using airlift. The structure is the same as the other codecs and it alway= s uses a 3 byte header (#2).

.. Owen

On Wed, Feb 7, 2018= at 9:41 PM, David Phillips <david@acz.org> wrote:
We previously= preserved an LZ4 CompressionKind and plan to implement it in the Presto re= ader and writer. Unlikely Snappy, the LZ4 format does not record the uncomp= ressed length. Thus, when reading, we need to allocate an output buffer tha= t is the full=C2=A0compressionBlockSize. This can waste a lot of memory whe= n there are many streams and many open readers.

We propose to pre= fix the LZ4 block with the uncompressed size. I see a few ways of doing it:=

1) Variable length integer, the same as Snappy.
2) Fixed 3-by= te integer, little-endian.
3) Fixed 4-byte integer, little-endian.
=
Option #1 is more complicated, uses more CPU to decode, and probably= doesn't save much space; buffers starting at 16kB will use 3 bytes.
=
Option #2 restricts the maximum size to be 16MB-1 byte. This is ridi= culously large for a per-stream buffer and not a problem as current writers= cap the buffer size at a reasonable 256kB, so it shouldn't be a proble= m in practice, but it's worth calling out here.

Option #3 is = flexible but in practice will waste a byte.

My vote is for option = #2.


--001a11c046c4815c9e0564ad19ec--