From dev-return-31123-archive-asf-public=cust-asf.ponee.io@ignite.apache.org Fri Feb 16 11:59:05 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id CCAF1180718 for ; Fri, 16 Feb 2018 11:59:04 +0100 (CET) Received: (qmail 95597 invoked by uid 500); 16 Feb 2018 10:59:03 -0000 Mailing-List: contact dev-help@ignite.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@ignite.apache.org Delivered-To: mailing list dev@ignite.apache.org Received: (qmail 95586 invoked by uid 99); 16 Feb 2018 10:59:03 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 16 Feb 2018 10:59:03 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 6063218033B for ; Fri, 16 Feb 2018 10:59:03 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -110.311 X-Spam-Level: X-Spam-Status: No, score=-110.311 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, T_RP_MATCHES_RCVD=-0.01, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id x--cHsFXBYna for ; Fri, 16 Feb 2018 10:59:02 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id E36C15F5FB for ; Fri, 16 Feb 2018 10:59:01 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 2DCD0E01DB for ; Fri, 16 Feb 2018 10:59:00 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 14D6B21E5D for ; Fri, 16 Feb 2018 10:59:00 +0000 (UTC) Date: Fri, 16 Feb 2018 10:59:00 +0000 (UTC) From: "Vladimir Ozerov (JIRA)" To: dev@ignite.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Created] (IGNITE-7736) SQL COPY: streaming model for network packets instead of request-response model MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 Vladimir Ozerov created IGNITE-7736: --------------------------------------- Summary: SQL COPY: streaming model for network packets instead of request-response model Key: IGNITE-7736 URL: https://issues.apache.org/jira/browse/IGNITE-7736 Project: Ignite Issue Type: Task Reporter: Vladimir Ozerov Fix For: 2.5 *Problem* Currently data transfer in COPY command is implemented as a series of request-responses. When request is received, it is parsed synchronously and passed to the streamer, then response is sent. This is not very efficient approach: # We hardly could utilize long fat network channels efficiently as we spend a lot of time waiting for a very small response (ack). # Parsing takes and adding data to the streamer takes time (especially if we reached streamer buffer limitations and are blocked waiting for responses from data nodes). During this period network is not utilized and file data is not transferred further. *Solution* Let's fix the problem iteratively as follows: # Introduce asynchrony - when request is received, send the response immediately before data processing # Then consider sending one ack for several requests instead of sending ack for every request # When multiple simultaneous requests are enabled (previous point), consider moving data processing to separate stream, so that we can read data from the socket as fast as possible -- This message was sent by Atlassian JIRA (v7.6.3#76005)