Return-Path: X-Original-To: apmail-kudu-commits-archive@minotaur.apache.org Delivered-To: apmail-kudu-commits-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id D3F4C1896C for ; Fri, 5 Feb 2016 20:53:44 +0000 (UTC) Received: (qmail 61961 invoked by uid 500); 5 Feb 2016 20:53:44 -0000 Delivered-To: apmail-kudu-commits-archive@kudu.apache.org Received: (qmail 61943 invoked by uid 500); 5 Feb 2016 20:53:44 -0000 Mailing-List: contact commits-help@kudu.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@kudu.incubator.apache.org Delivered-To: mailing list commits@kudu.incubator.apache.org Received: (qmail 61934 invoked by uid 99); 5 Feb 2016 20:53:44 -0000 Received: from Unknown (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 05 Feb 2016 20:53:44 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 5CA7D180179 for ; Fri, 5 Feb 2016 20:53:44 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -3.649 X-Spam-Level: X-Spam-Status: No, score=-3.649 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, KAM_LAZY_DOMAIN_SECURITY=1, RCVD_IN_DNSWL_HI=-5, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RP_MATCHES_RCVD=-0.429] autolearn=disabled Received: from mx1-us-west.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id Iqfnnd-lNdyY for ; Fri, 5 Feb 2016 20:53:43 +0000 (UTC) Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx1-us-west.apache.org (ASF Mail Server at mx1-us-west.apache.org) with SMTP id AC53B20D55 for ; Fri, 5 Feb 2016 20:53:42 +0000 (UTC) Received: (qmail 61795 invoked by uid 99); 5 Feb 2016 20:53:42 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 05 Feb 2016 20:53:42 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 57431E0984; Fri, 5 Feb 2016 20:53:42 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: jdcryans@apache.org To: commits@kudu.incubator.apache.org Date: Fri, 05 Feb 2016 20:53:46 -0000 Message-Id: <57e80036c2f64786939228c617091a7e@git.apache.org> In-Reply-To: <8fc6ca0afa514080be79cf8c5351aa1e@git.apache.org> References: <8fc6ca0afa514080be79cf8c5351aa1e@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: [5/5] incubator-kudu git commit: [python] - Allow to scan batch-by-batch [python] - Allow to scan batch-by-batch Currently the python client would only allow to read all the tuples from the scan at once, which could easily cause oom issues. This patch makes a couple of changes necessary to allow to scan batch-by-batch and adds a test to make sure it works. Change-Id: Ifced955be40943dc4ad648d90a24db98b27eed70 Reviewed-on: http://gerrit.cloudera.org:8080/2052 Reviewed-by: David Ribeiro Alves Tested-by: Kudu Jenkins Reviewed-by: Todd Lipcon (cherry picked from commit 786834d1d4b4313535c2763db7022a25ce1e1dd0) Reviewed-on: http://gerrit.cloudera.org:8080/2074 Reviewed-by: Jean-Daniel Cryans Tested-by: Jean-Daniel Cryans Project: http://git-wip-us.apache.org/repos/asf/incubator-kudu/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-kudu/commit/a2bb870c Tree: http://git-wip-us.apache.org/repos/asf/incubator-kudu/tree/a2bb870c Diff: http://git-wip-us.apache.org/repos/asf/incubator-kudu/diff/a2bb870c Branch: refs/heads/branch-0.7.0 Commit: a2bb870c37e59afeed16f427f0e0b6f942f27d4b Parents: 015049d Author: David Alves Authored: Thu Feb 4 15:01:19 2016 -0800 Committer: Jean-Daniel Cryans Committed: Fri Feb 5 20:52:00 2016 +0000 ---------------------------------------------------------------------- python/kudu/client.pyx | 4 ++-- python/kudu/tests/test_scanner.py | 18 ++++++++++++++++++ 2 files changed, 20 insertions(+), 2 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-kudu/blob/a2bb870c/python/kudu/client.pyx ---------------------------------------------------------------------- diff --git a/python/kudu/client.pyx b/python/kudu/client.pyx index b85cd46..153c7a8 100644 --- a/python/kudu/client.pyx +++ b/python/kudu/client.pyx @@ -792,7 +792,7 @@ cdef class Row: def __dealloc__(self): pass - cdef tuple as_tuple(self): + cpdef tuple as_tuple(self): """ Return the row as a Python tuple """ @@ -1093,7 +1093,7 @@ cdef class Scanner: def read_next_batch_tuples(self): return self.next_batch().as_tuples() - cdef RowBatch next_batch(self): + cpdef RowBatch next_batch(self): """ Retrieve the next batch of rows from the scanner. http://git-wip-us.apache.org/repos/asf/incubator-kudu/blob/a2bb870c/python/kudu/tests/test_scanner.py ---------------------------------------------------------------------- diff --git a/python/kudu/tests/test_scanner.py b/python/kudu/tests/test_scanner.py index 9699beb..0ae2036 100644 --- a/python/kudu/tests/test_scanner.py +++ b/python/kudu/tests/test_scanner.py @@ -121,3 +121,21 @@ class TestScanner(KuduTestBase, unittest.TestCase): with self.assertRaises(kudu.KuduInvalidArgument): scanner.add_predicates([sv >= 1]) + + def test_scan_batch_by_batch(self): + scanner = self.table.scanner() + scanner.set_fault_tolerant() + lower_bound = scanner.new_bound() + lower_bound['key'] = 10 + scanner.add_lower_bound(lower_bound) + upper_bound = scanner.new_bound() + upper_bound['key'] = 90 + scanner.add_exclusive_upper_bound(upper_bound) + scanner.open() + + tuples = [] + while scanner.has_more_rows(): + batch = scanner.next_batch() + tuples.extend(batch.as_tuples()) + + self.assertEqual(sorted(tuples), self.tuples[10:90])