From dev-return-31115-archive-asf-public=cust-asf.ponee.io@ignite.apache.org Fri Feb 16 11:06:25 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 5D5AE180647 for ; Fri, 16 Feb 2018 11:06:24 +0100 (CET) Received: (qmail 94860 invoked by uid 500); 16 Feb 2018 10:06:23 -0000 Mailing-List: contact dev-help@ignite.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@ignite.apache.org Delivered-To: mailing list dev@ignite.apache.org Received: (qmail 94840 invoked by uid 99); 16 Feb 2018 10:06:22 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Feb 2018 10:06:22 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 2227EC13F8 for ; Fri, 16 Feb 2018 10:06:22 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.192 X-Spam-Level: *** X-Spam-Status: No, score=3.192 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, URI_HEX=1.313] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id sliqlr5EP9_n for ; Fri, 16 Feb 2018 10:06:20 +0000 (UTC) Received: from mail-wm0-f47.google.com (mail-wm0-f47.google.com [74.125.82.47]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id D03095F39E for ; Fri, 16 Feb 2018 10:06:19 +0000 (UTC) Received: by mail-wm0-f47.google.com with SMTP id t3so2167217wmc.2 for ; Fri, 16 Feb 2018 02:06:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=b7NDVfb4ZCvmE8Lj9R1vDzgkbNypYVRENIxPr0Irc8c=; b=Oc1+LqO2CWGj2qgbuCgwtwy9Gvv1QolXmcXFzq+39xfcot3MOgPAkwBRW2I2R/d1Mt vUcJLiVNv/YBrRj5K5oDoEkHtVo3U8hUpcNbihi33qASgx5vMEWmBkOYsd8UyoS+Fg78 cW7h/YL4EZvjDHSOWnt4PqPJa8oZQrtwZTI5X/SyMsdnHnwrrTJS2hBrMcarzY5MNa81 MCmVHa/O4GCVejq4GFHZnsHp149mFJdKclEZ3L2oXnMoeqhMpblnQ6emaKt54lBBTUnk TLaFdACTKKpopW/Cq3KEW7LyvaXhoA+IbfglE2zB8yIlL7I9ZC169G5Du8/b9VsBqyBr wUKA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=b7NDVfb4ZCvmE8Lj9R1vDzgkbNypYVRENIxPr0Irc8c=; b=IDMD4r84c45yz+bmu2/OBBUyZ4rXrYLuLAzGzR8ZGFPPLs44xsB5O2AX82ypeutBKC 0GpBZqiYA7yGRDXzRzgVGNuiPIKp7bHeJfPlXw2TU91xqyVuZ+nUAwI4A2ES10kF0PNe TTs99gtihABpzERByEKvwdrQLuP3s3gp1VLbPLzjNcLb1cXCImBNC+QZP145eQ6mn1Xl AP7Ld04ybuczdCj2V18JdthZP8BQ+GHmsrFWqEswCBmDJP/Y4bFOWOhnlF2oc2PTN2Ur MF9XwsYSwMLTV0gYLXFG8Up32lnyVnF1sBYDEOlb+1sRGH8/FmaXSG2owcbdskxEpx8u h/tw== X-Gm-Message-State: APf1xPB1kqNmt9iVJeGzYqCeq8OL+E4+BdTlOZS//nTiedUoPr6OmPsh Sx8I0qQEtcxpfg3F3ZN/Xe0YG4ilh+ACOBV3WntLrQ== X-Google-Smtp-Source: AH8x2270u96g7I9RiG1mLYIqBStRN/l7UmhegeEPw0Mm+o9KBtCK2mIBKWaIHVCpC5nziFvpnzKShmy3zE9MD7mPZFY= X-Received: by 10.80.147.229 with SMTP id o92mr7086813eda.206.1518775578655; Fri, 16 Feb 2018 02:06:18 -0800 (PST) MIME-Version: 1.0 Received: by 10.80.140.148 with HTTP; Fri, 16 Feb 2018 02:06:18 -0800 (PST) In-Reply-To: References: <637C9F25-9FF2-4EA5-8D2F-C6272F95BEBF@apache.org> <1516875829703-0.post@n4.nabble.com> <1517583957653-0.post@n4.nabble.com> From: Nikita Amelchev Date: Fri, 16 Feb 2018 13:06:18 +0300 Message-ID: Subject: Re: Transport compression (not store compression) To: dev@ignite.apache.org Content-Type: multipart/alternative; boundary="f403045c8232dc01350565517e58" --f403045c8232dc01350565517e58 Content-Type: text/plain; charset="UTF-8" Vladimir Ozerov, I also agree that your solution is good. I will check this flag before adding a client to map of clients. If one of the nodes have the flag then the session will be marked "compressed". At the nearest time, I will provide a solution. Dmitriy Setrakyan, I will implement and test compressed flag after I write test with real operations (put, get etc) on yardstick. 2018-02-16 0:05 GMT+03:00 Dmitriy Setrakyan : > Vova, I think your solution is fine, but I think we will always have some > messages compressed and others not. For example, in many cases, especially > when messages are relatively small, compressing them will introduce an > unnecessary overhead, and most likely slow down the cluster. > > Why not have compression flag or compression bit on per-message level? We > check if the bit is turned on, and if it is, then we uncompress the message > on the receiving side before processing it. > > D. > > On Thu, Feb 15, 2018 at 12:24 AM, Vladimir Ozerov > wrote: > > > I think that we should not guess on how the clients are used. They could > be > > used in any way - in the same network, in another network, in Docker, in > > hypervisor, etc.. This holds for both thin and thick clients. It is > > essential that we design configuration API in a way that compression > could > > be enabled only for some participants. > > > > What if we do this as follows: > > 1) Define "IgniteConfiguration.compressionEnabled" flag > > 2) When two nodes communicate and at least one of them has this flag, > then > > all data sent between them is compressed. > > > > Makes sense? > > > > On Thu, Feb 15, 2018 at 8:50 AM, Nikita Amelchev > > wrote: > > > > > Hello, Igniters. > > > > > > I have not seen such use-cases, where heavy client ignite node placed > in > > a > > > much worse network than the server. I'm not sure we should encourage a > > bad > > > cluster architecture. > > > > > > Usually, in my use-cases, the servers and clients locate in the same > > > network. And if the cluster has SSL enabled, it makes sense to enable > > > compression, even if the network is fast. It also makes sense when we > > have > > > a high load on the network, and the CPU is utilized poorly. > > > > > > I'll do tests on yardstick for real operations like get, put etc. and > SQL > > > requests. > > > > > > I propose to add configurable compression for thin client/ODBC/JDBC as > a > > > separate issue because it increases the current PR. > > > > > > Even if it really makes sense to compress the traffic only between > > > client-server ignite nodes, it should also be a separate issue, that > > would > > > not increase the PR. Especially, since this compression architecture > may > > > not be accepted by the community. > > > > > > 2018-02-05 13:02 GMT+03:00 Nikita Amelchev : > > > > > > > Thanks for your comments, > > > > > > > > I will try to separate network compression for clients and servers. > > > > > > > > It makes sense to enable compression on servers if we have SSL turned > > on. > > > > I tested rebalancing time and compression+ssl is faster. SSL > throughput > > > is > > > > limited by 800 Mbits/sec per connection and if enable compression, it > > > > boosted up to 1100 Mbits. > > > > > > > > 2018-02-02 18:52 GMT+03:00 Alexey Kuznetsov : > > > > > > > >> I think Igor is right. > > > >> > > > >> Ususally servers connected via fast local network. > > > >> But clients could be in external and slow network. > > > >> In this scenario compression will be very useful. > > > >> > > > >> Once I had such scenario - client connected to cluster via 300 kb/s > > > >> network > > > >> and tries to transfer ~10Mb of uncumpressed data. > > > >> So it takse ~30 seconds. > > > >> After I implemented compression it becamed 1M and transfered for ~3 > > > >> seconds. > > > >> > > > >> I think we should take care of all mentioned problems with NIO > threads > > > in > > > >> order to not slow down whole cluster. > > > >> > > > >> > > > >> On Fri, Feb 2, 2018 at 10:05 PM, gvvinblade > > > wrote: > > > >> > > > >> > Nikita, > > > >> > > > > >> > Yes, you're right. Maybe I wasn't clear enough. > > > >> > > > > >> > Usually server nodes are placed in the same fast network segment > > (one > > > >> > datacenter); in any case we need an ability to setup compression > per > > > >> > connection using some filter like useCompression(ClusterNode, > > > >> ClusterNode) > > > >> > to compress traffic only between servers and client nodes. > > > >> > > > > >> > But issue is still there, since the same NIO worker serves both > > client > > > >> and > > > >> > server connections, enabled compression may impact whole cluster > > > >> > performance > > > >> > because NIO threads will compress client messages instead of > > > processing > > > >> > servers' compute requests. That was my concern. > > > >> > > > > >> > Compression for clients is really cool feature and usefull in some > > > >> cases. > > > >> > Probably it makes sense to have two NIO servers with and without > > > >> > compression > > > >> > to process server and client requests separately or pin somehow > > worker > > > >> > threads to client or server sessions... > > > >> > > > > >> > Also we have to think about client connections (JDBC, ODBC, .Net > > thin > > > >> > client, etc) and setup compression for them separately. > > > >> > > > > >> > Anyway I would compare put, get, putAll, getAll and SQL SELECT > > > >> operations > > > >> > for strings and POJOs, one server, several clients with and > without > > > >> > compression, setting up the server to utilize all cores by NIO > > > workers, > > > >> > just > > > >> > to get know possible impact. > > > >> > > > > >> > Possible configuration for servers with 16 cores: > > > >> > > > > >> > Selectors cnt = 16 > > > >> > Connections per node = 4 > > > >> > > > > >> > Where client nodes perform operations in 16 threads > > > >> > > > > >> > Regards, > > > >> > Igor > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > -- > > > >> > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/ > > > >> > > > > >> > > > >> > > > >> > > > >> -- > > > >> Alexey Kuznetsov > > > >> > > > > > > > > > > > > > > > > -- > > > > Best wishes, > > > > Amelchev Nikita > > > > > > > > > > > > > > > > -- > > > Best wishes, > > > Amelchev Nikita > > > > > > -- Best wishes, Amelchev Nikita --f403045c8232dc01350565517e58--