Return-Path: X-Original-To: apmail-hbase-dev-archive@www.apache.org Delivered-To: apmail-hbase-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B577718C78 for ; Tue, 2 Feb 2016 06:28:17 +0000 (UTC) Received: (qmail 48082 invoked by uid 500); 2 Feb 2016 06:27:42 -0000 Delivered-To: apmail-hbase-dev-archive@hbase.apache.org Received: (qmail 47995 invoked by uid 500); 2 Feb 2016 06:27:42 -0000 Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hbase.apache.org Delivered-To: mailing list dev@hbase.apache.org Received: (qmail 47983 invoked by uid 99); 2 Feb 2016 06:27:42 -0000 Received: from Unknown (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Feb 2016 06:27:42 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 03B9CC0873 for ; Tue, 2 Feb 2016 06:27:42 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.481 X-Spam-Level: ** X-Spam-Status: No, score=2.481 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HEADER_FROM_DIFFERENT_DOMAINS=0.001, HTML_MESSAGE=2.5, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-us-west.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id sRLSMEMg-FCY for ; Tue, 2 Feb 2016 06:27:34 +0000 (UTC) Received: from mail-ig0-f169.google.com (mail-ig0-f169.google.com [209.85.213.169]) by mx1-us-west.apache.org (ASF Mail Server at mx1-us-west.apache.org) with ESMTPS id B388520271 for ; Tue, 2 Feb 2016 06:27:33 +0000 (UTC) Received: by mail-ig0-f169.google.com with SMTP id t15so53499082igr.0 for ; Mon, 01 Feb 2016 22:27:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:date:message-id:subject:from:to:content-type; bh=KaiZ8PhSWb1lfpMPtzk83uevZGgz3FpR7N5c6IhY2ro=; b=rxxkuvlwkDLaC8co1rkIchyQmA4bWP2sfFsUNelSTyy61sQHWPmRgcHsywwzujr3jH IQzPpgXyo89KKXyum40UCt925IfpKKRM30XvLk5bOsQwHfALjUMVTiTuUnS2jGOVaCQ4 hm8fpncfGTmg3zSFHiwziYGXYTBerfTjuWtRWKAeRkqU9cXzYgDm+GDdN9C7zcPdCc6h y7PIymovPCyO4dgpAvquqq5uMsfWHmsSox0H5GxLBH4M69uYJCPO+8LIAOX+ax8gRma5 wC6UOKd7FyhP+tpp6EIrVHQmJHmLxdXrjr7CROSLl5oPxtCM6AbBoyWqtRNPEkNLF8iZ KrLA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:date:message-id:subject:from :to:content-type; bh=KaiZ8PhSWb1lfpMPtzk83uevZGgz3FpR7N5c6IhY2ro=; b=Gb2ZCAJBJFoiNkw47AgJhZYnFgw9Wyfcyi9dFUpZd3CIF8xK9On1YdfqkyMv8PK6fw B1+4j92LvRGRiLUZDdiwMycxrHgZuEiuipHePlYyybqhNysjQfx22wBEsCEeXn32OwgK KQs3V4Ptl4mq3O8ots7Pdrfw+iBak6863mBscf08pCQcLjQT6dbI//1PJOe0kUY/+Voy Wbx/Mtofv5tSiCVd5XVlnoN7+lX/JTQZ1cxLqvTL4E46Lhhs62CIKVs/sYGdTiQBOClP dFBTg9jGZIb/JM6xsC5PXPwrZfpGIWSXw4RIEPEq7v21byMomg7Gc3Xma2mjGR+o6pr4 65+w== X-Gm-Message-State: AG10YORrPGaVO1CBorzPvp0Gkmj7/g2bfSaViA1uI4bFTdAT+XJR1RZ8AcYgteGTQS/XF3jWI1u/3zUCh2JsZw== MIME-Version: 1.0 X-Received: by 10.50.43.136 with SMTP id w8mr15333756igl.96.1454394453170; Mon, 01 Feb 2016 22:27:33 -0800 (PST) Sender: saint.ack@gmail.com Received: by 10.64.141.69 with HTTP; Mon, 1 Feb 2016 22:27:33 -0800 (PST) Date: Mon, 1 Feb 2016 22:27:33 -0800 X-Google-Sender-Auth: loHz1xQfWY3QolMi8b16HDDV1Jo Message-ID: Subject: DISCUSS: Protobufs? From: Stack To: HBase Dev List Content-Type: multipart/alternative; boundary=089e011602a8bea468052ac398b0 --089e011602a8bea468052ac398b0 Content-Type: text/plain; charset=UTF-8 We are running into a few issues with protobufs. + PB always copies all data before making a Message. This generates garbage unnecessarily. + CodedInputStream does not support ByteBuffers in 2.5. In 2.6 it does but again, copies the data out of the BB always; this is especially painful when the BB is a DBB with its data offheap and intent is to keep data offheap. There are other issues. CIS allocates 4k buffers regardless (See HBASE-15177). And then there was the HBaseZeroCopyByteString fun and games we had a while back. 3.0 PB adds UnsafeByteStrings so can do zero copy. Thats good. But PB3 is incompatible with PB2 (or at least, it looks like PB2 clients can't talk to PB3 [1]). There is javanano protobufs. All is open access, but it too looks different to PB2 (i've not tried it). Protostuff seems really quiet these times [2]. Fork (and shade)? Thoughts? St.Ack 1. https://github.com/google/protobuf/releases 2. https://groups.google.com/forum/#!forum/protostuff --089e011602a8bea468052ac398b0--