zeppelin-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From felixche...@apache.org
Subject zeppelin git commit: [ZEPPELIN-1470] limiting results from jdbc
Date Mon, 17 Jul 2017 07:52:25 GMT
Repository: zeppelin
Updated Branches:
  refs/heads/master 709c5a70a -> 87705a932

[ZEPPELIN-1470] limiting results from jdbc

### What is this PR for?

One thing we tracked down is that if you issue a large query on a very large table, it will
simply try to load all results (and then cap them on Zeppelin's side), which seems suboptimal
(and will freeze the server). Setting this on the JDBC level seems to solve the problem.

### What type of PR is it?
Bug Fix

### Todos
* [x] Tests

### How should this be tested?

- Create or use a table with a very large number of rows. In our tests, I simply created a:

createdb zeppelin_test

psql zeppelin_test

create table too_many_rows(n int)

And added 5m rows to it.

Making a paragraph like this will hang without setting a limit:

select * from too_many_rows

### Questions:
* Does the licenses files need update?
* Is there breaking changes for older versions?
* Does this needs documentation?

Author: Herval Freire <hfreire@twitter.com>

Closes #2428 from herval/hfreire/limit-row-count and squashes the following commits:

4f66469 [Herval Freire] display truncation message
b538c44 [Herval Freire] limiting results from jdbc

Project: http://git-wip-us.apache.org/repos/asf/zeppelin/repo
Commit: http://git-wip-us.apache.org/repos/asf/zeppelin/commit/87705a93
Tree: http://git-wip-us.apache.org/repos/asf/zeppelin/tree/87705a93
Diff: http://git-wip-us.apache.org/repos/asf/zeppelin/diff/87705a93

Branch: refs/heads/master
Commit: 87705a93244d24448cbd147f63263900241b4b2b
Parents: 709c5a7
Author: Herval Freire <hfreire@twitter.com>
Authored: Thu Jun 29 12:39:07 2017 -0700
Committer: Felix Cheung <felixcheung@apache.org>
Committed: Mon Jul 17 00:52:03 2017 -0700

 .../src/main/java/org/apache/zeppelin/jdbc/JDBCInterpreter.java | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/jdbc/src/main/java/org/apache/zeppelin/jdbc/JDBCInterpreter.java b/jdbc/src/main/java/org/apache/zeppelin/jdbc/JDBCInterpreter.java
index 3483b4b..f3f4326 100644
--- a/jdbc/src/main/java/org/apache/zeppelin/jdbc/JDBCInterpreter.java
+++ b/jdbc/src/main/java/org/apache/zeppelin/jdbc/JDBCInterpreter.java
@@ -683,6 +683,11 @@ public class JDBCInterpreter extends KerberosInterpreter {
       for (int i = 0; i < sqlArray.size(); i++) {
         String sqlToExecute = sqlArray.get(i);
         statement = connection.createStatement();
+        // fetch n+1 rows in order to indicate there's more rows available (for large selects)
+        statement.setFetchSize(getMaxResult());
+        statement.setMaxRows(getMaxResult() + 1);
         if (statement == null) {
           return new InterpreterResult(Code.ERROR, "Prefix not found.");

View raw message