hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tao Li (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HIVE-14170) Beeline IncrementalRows should buffer rows and incrementally re-calculate width if TableOutputFormat is used
Date Tue, 19 Jul 2016 17:38:20 GMT

    [ https://issues.apache.org/jira/browse/HIVE-14170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15384559#comment-15384559
] 

Tao Li edited comment on HIVE-14170 at 7/19/16 5:38 PM:
--------------------------------------------------------

[~stakiar] Another thinking is that we may improve the "buffered page" mode to avoid OOM issue.
For example, we can iterate through the whole result set once to calculate the max column
width (and without loading the result set into memory). Then we iterate the result set again
to print out. The pros is that it requires minimal code change. The cons is that the latency
should be higher because we iterate the result set twice. 


was (Author: taoli-hwx):
[~stakiar] Another thinking is that we may improve the "buffered page" mode to avoid OOM issue.
For example, we can iterate through the whole result set once to calculate the max column
width (and without loading the result set into memory). Then we iterate the result set again
to print out. The pros is that it requires minimal code change. The cons is that the latency
should be higher because we iterate the result set twice. 

> Beeline IncrementalRows should buffer rows and incrementally re-calculate width if TableOutputFormat
is used
> ------------------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-14170
>                 URL: https://issues.apache.org/jira/browse/HIVE-14170
>             Project: Hive
>          Issue Type: Sub-task
>          Components: Beeline
>            Reporter: Sahil Takiar
>            Assignee: Sahil Takiar
>         Attachments: HIVE-14170.1.patch, HIVE-14170.2.patch
>
>
> If {{--incremental}} is specified in Beeline, rows are meant to be printed out immediately.
However, if {{TableOutputFormat}} is used with this option the formatting can look really
off.
> The reason is that {{IncrementalRows}} does not do a global calculation of the optimal
width size for {{TableOutputFormat}} (it can't because it only sees one row at a time). The
output of {{BufferedRows}} looks much better because it can do this global calculation.
> If {{--incremental}} is used, and {{TableOutputFormat}} is used, the width should be
re-calculated every "x" rows ("x" can be configurable and by default it can be 1000).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message