ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "shawn.du" <shawn...@neulion.com.cn>
Subject Re: integrate with prestodb
Date Tue, 17 Oct 2017 02:18:19 GMT
<html>
<head>
    <meta http-equiv='Content-Type' content='text/html; charset=UTF-8'>
</head>
<body>
<style>
    font{
        line-height: 1.7;
    }
</style>
<div style = 'font-family:"微软雅黑"; font-size: 14px; color:#000000; line-height:1.7;'>
    <div>
<div><span>Hi Denis,</span></div><div><span><br></span></div><div>Yes,
We do want to limit the RAM to less than 64G. &nbsp;RAM resource is still an expensive
resource.</div><div>If we store our data with ignite SQL queryable format, our
data may use more than 640G. This is too expensive for us.</div><div><span
style="line-height: 1.7;">So we store data using binary format which works a bit like orc
or parquet. Only several important columns are SQL queryable and the others are not. In this
way, we do store using less RAMs, but we have to use map-reduce to query the data, which is
a little bit of complex: Query in client with SQL, then submit jobs to ignite compute, finally
do some post aggregation in client.</span></div><div>This is why I want
to have a try of Presto. We like SQL, we want all computation on server side.&nbsp;</div><div><br></div><div>welcome
your comments.</div><div><br></div>
<div id="ntes-pcmail-signature" style="font-family:'微软雅黑'">
    <style type="text/css">
        a#ntes-pcmail-signature-default:hover {
            text-decoration: underline;
            color: #199cff;
            cursor: pointer;
        }
        a#ntes-pcmail-signature-default:active {
            text-decoration: underline;
            color: #246fce;
            cursor: pointer;
        }
    </style>

                <div style="font-size:14px; padding: 0;  margin:0;">
                    <div style="font-family:&quot;微软雅黑&quot;; font-size:
13px; color:#000000">
    <style>
        font{
            line-height: 1.5;
        }
    </style>
<div id="ntes-pcmail-signature-default" style="font-size:14px; color:#000; text-decoration:
none;">Thanks</div><div id="ntes-pcmail-signature-default" style="font-size:14px;
color:#000; text-decoration: none;">Shawn</div>
</div>
                </div>

</div><br>
</div><div class="J-reply" style="background-color:#f2f2f2;color:black;padding-top:6px;padding-bottom:6px;border-radius:3px;-moz-border-radius:3px;-webkit-border-radius:3px;margin-top:45px;margin-bottom:20px;font-family:'微软雅黑';">
    <div style="font-size:14px;line-height:1.5;word-break:break-all;margin-left:10px;margin-right:10px">On
<span class="mail-date">10/17/2017 07:57</span>,<a class="mail-to" style="text-decoration:none;color:#2a83f2;"
href="mailto:dmagda@apache.org">Denis Magda&lt;dmagda@apache.org&gt;</a>
wrote: </div>
</div>
<blockquote id="ntes-pcmail-quote" style="margin: 0; padding: 0; font-size: 14px; font-family:
'微软雅黑';">
Hello&nbsp;Shawn,
<br>
<br>Do&nbsp;I&nbsp;understand&nbsp;properly&nbsp;that&nbsp;you&nbsp;have&nbsp;scarce&nbsp;RAM&nbsp;resources&nbsp;and&nbsp;think&nbsp;to&nbsp;exploit&nbsp;Presto&nbsp;as&nbsp;an&nbsp;alternative&nbsp;SQL&nbsp;engine&nbsp;in&nbsp;Ignite&nbsp;that&nbsp;queries&nbsp;both&nbsp;RAM&nbsp;and&nbsp;disk&nbsp;data&nbsp;sets?&nbsp;If&nbsp;that’s&nbsp;the&nbsp;case&nbsp;than&nbsp;just&nbsp;enable&nbsp;Ignite&nbsp;native&nbsp;persistence&nbsp;[1]&nbsp;and&nbsp;you’ll&nbsp;get&nbsp;all&nbsp;the&nbsp;data&nbsp;stored&nbsp;on&nbsp;disk&nbsp;and&nbsp;as&nbsp;much&nbsp;as&nbsp;you&nbsp;can&nbsp;afford&nbsp;in&nbsp;RAM.&nbsp;The&nbsp;SQL&nbsp;works&nbsp;over&nbsp;both&nbsp;tiers&nbsp;transparently&nbsp;for&nbsp;you.
<br>
<br>[1]&nbsp;https://ignite.apache.org/features/persistence.html&nbsp;&lt;https://ignite.apache.org/features/persistence.html&gt;
<br>
<br>—
<br>Denis
<br>
<br>&gt;&nbsp;On&nbsp;Oct&nbsp;16,&nbsp;2017,&nbsp;at&nbsp;2:19&nbsp;AM,&nbsp;Alexey&nbsp;Kukushkin&nbsp;&lt;kukushkinalexey@gmail.com&gt;&nbsp;wrote:
<br>&gt;&nbsp;
<br>&gt;&nbsp;Cross-sending&nbsp;to&nbsp;the&nbsp;DEV&nbsp;community.
<br>&gt;&nbsp;
<br>&gt;&nbsp;On&nbsp;Mon,&nbsp;Oct&nbsp;16,&nbsp;2017&nbsp;at&nbsp;12:14&nbsp;PM,&nbsp;shawn.du&nbsp;&lt;shawn.du@neulion.com.cn&nbsp;&lt;mailto:shawn.du@neulion.com.cn&gt;&gt;&nbsp;wrote:
<br>&gt;&nbsp;Hi&nbsp;community,
<br>&gt;&nbsp;
<br>&gt;&nbsp;I&nbsp;am&nbsp;trying&nbsp;to&nbsp;implement&nbsp;a&nbsp;connector&nbsp;for&nbsp;presto&nbsp;to&nbsp;connect&nbsp;ignite.&nbsp;
<br>&gt;&nbsp;I&nbsp;think&nbsp;it&nbsp;will&nbsp;be&nbsp;a&nbsp;very&nbsp;interest&nbsp;thing&nbsp;to&nbsp;connect&nbsp;ignite&nbsp;and&nbsp;presto.
<br>&gt;&nbsp;
<br>&gt;&nbsp;In&nbsp;fact,&nbsp;currently&nbsp;we&nbsp;use&nbsp;ignite&nbsp;and&nbsp;it&nbsp;works&nbsp;very&nbsp;well.&nbsp;&nbsp;but&nbsp;in&nbsp;order&nbsp;to&nbsp;save&nbsp;memory,&nbsp;we&nbsp;build&nbsp;compressed&nbsp;binary&nbsp;data.
<br>&gt;&nbsp;thus&nbsp;we&nbsp;cannot&nbsp;query&nbsp;them&nbsp;using&nbsp;SQL.&nbsp;We&nbsp;use&nbsp;ignite&nbsp;map-reduce&nbsp;to&nbsp;query&nbsp;the&nbsp;data.
<br>&gt;&nbsp;
<br>&gt;&nbsp;Using&nbsp;presto,&nbsp;we&nbsp;may&nbsp;use&nbsp;SQL&nbsp;again.&nbsp;If&nbsp;it&nbsp;is&nbsp;fast&nbsp;enough,&nbsp;ignite&nbsp;will&nbsp;be&nbsp;our&nbsp;in&nbsp;memory&nbsp;storage&nbsp;and&nbsp;not&nbsp;responsible&nbsp;for&nbsp;computing&nbsp;or&nbsp;only&nbsp;for&nbsp;simple&nbsp;query.
<br>&gt;&nbsp;The&nbsp;only&nbsp;thing&nbsp;I&nbsp;concern&nbsp;about&nbsp;is&nbsp;presto&nbsp;is&nbsp;fast&nbsp;enough&nbsp;or&nbsp;not&nbsp;like&nbsp;Ignite.&nbsp;For&nbsp;now&nbsp;all&nbsp;ignite&nbsp;query&nbsp;cost&nbsp;less&nbsp;than&nbsp;5&nbsp;seconds&nbsp;and&nbsp;most&nbsp;are&nbsp;hundreds&nbsp;of&nbsp;milliseconds.
<br>&gt;&nbsp;Also&nbsp;presto&nbsp;provides&nbsp;a&nbsp;connector&nbsp;for&nbsp;redis.&nbsp;&nbsp;I&nbsp;don't&nbsp;know&nbsp;community&nbsp;has&nbsp;interest&nbsp;to&nbsp;contribute&nbsp;to&nbsp;presto-ignite?
<br>&gt;&nbsp;
<br>&gt;&nbsp;Thanks
<br>&gt;&nbsp;Shawn
<br>&gt;&nbsp;
<br>&gt;&nbsp;
<br>&gt;&nbsp;
<br>&gt;&nbsp;
<br>&gt;&nbsp;--&nbsp;
<br>&gt;&nbsp;Best&nbsp;regards,
<br>&gt;&nbsp;Alexey
<br>
<br></blockquote><!--😀-->
</div>
</body>
</html>
Mime
View raw message