Mailing-List: contact user-help@impala.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@impala.incubator.apache.org
MIME-Version: 1.0
From: =?UTF-8?Q?Dejan_Proki=C4=87?= <dejanp@nordeus.com>
Date: Tue, 27 Dec 2016 10:58:37 +0100
Message-ID: <CAMRLWVDOLW1ecbajk+WeK+G5RgNuHLG0=KCmKDKyWRn2F9RoWg@mail.gmail.com>
Subject: Bug with complex types
To: user@impala.incubator.apache.org
Content-Type: multipart/alternative; boundary=94eb2c0b770e69b5020544a0e5c0
archived-at: Tue, 27 Dec 2016 09:58:51 -0000

--94eb2c0b770e69b5020544a0e5c0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Hi,

I noticed a bug when selecting data from a table with complex type. This is
definition of my table:

CREATE TABLE event (
  user_id BIGINT,
  event_id INT,
  event_type_id INT,
  ts_bigint BIGINT,
  processed_ts_bigint BIGINT,
  event_map MAP<STRING, STRING>,
  server_id INT
)
PARTITIONED BY (date_id INT)
STORED AS PARQUET;

I want to run this query:

WITH
battles AS (
SELECT
  *
FROM event e,
  (SELECT
    MAX(CASE WHEN key=3D'won' THEN value END) AS won
  FROM e.event_map) m
WHERE e.date_id =3D 2420 AND e.event_id =3D 10012
)
SELECT won, count(*) FROM battles
GROUP BY won;

Map element with key =3D 'won' has only 'true' and 'false' values. If battl=
es
subquery has more than 5 million rows the query returns strange results,
and when it has less rows query returns correct data.

Are you aware of this issue? I didn't send you my data since it is pretty
large for mail (20 GB).

Thanks

*Dejan Proki=C4=87* | Data Engineer | Nordeus

--94eb2c0b770e69b5020544a0e5c0
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi,<div><br></div><div>I noticed a bug when selecting data=
 from a table with complex type. This is definition of my table:</div><div>=
<br></div><div><pre style=3D"color:rgb(0,0,0);font-family:menlo;font-size:9=
pt"><span style=3D"color:rgb(0,0,128);font-weight:bold">CREATE TABLE </span=
>event (<br>  user_id BIGINT,<br>  event_id <span style=3D"color:rgb(0,0,12=
8);font-weight:bold">INT</span>,<br>  event_type_id <span style=3D"color:rg=
b(0,0,128);font-weight:bold">INT</span>,<br>  ts_bigint BIGINT,<br>  proces=
sed_ts_bigint BIGINT,<br>  event_map MAP&lt;STRING, STRING&gt;,<br>  server=
_id <span style=3D"color:rgb(0,0,128);font-weight:bold">INT<br></span>)<br>=
PARTITIONED <span style=3D"color:rgb(0,0,128);font-weight:bold">BY </span>(=
date_id <span style=3D"color:rgb(0,0,128);font-weight:bold">INT</span>)<br>=
STORED <span style=3D"color:rgb(0,0,128);font-weight:bold">AS </span>PARQUE=
T;<br></pre></div><div>I want to run this query:</div><div><pre style=3D"co=
lor:rgb(0,0,0);font-family:menlo;font-size:9pt"><span style=3D"color:rgb(0,=
0,128);font-weight:bold">WITH<br></span>battles <span style=3D"color:rgb(0,=
0,128);font-weight:bold">AS </span>(<br><span style=3D"color:rgb(0,0,128);f=
ont-weight:bold">SELECT<br></span><span style=3D"color:rgb(0,0,128);font-we=
ight:bold">  </span>*<br><span style=3D"color:rgb(0,0,128);font-weight:bold=
">FROM </span>event e,<br>  (<span style=3D"color:rgb(0,0,128);font-weight:=
bold">SELECT<br></span><span style=3D"color:rgb(0,0,128);font-weight:bold">=
    MAX</span>(<span style=3D"color:rgb(0,0,128);font-weight:bold">CASE WHE=
N key</span>=3D<span style=3D"color:rgb(0,128,0);font-weight:bold">&#39;won=
&#39; </span><span style=3D"color:rgb(0,0,128);font-weight:bold">THEN value=
 END</span>) <span style=3D"color:rgb(0,0,128);font-weight:bold">AS </span>=
won<br>  <span style=3D"color:rgb(0,0,128);font-weight:bold">FROM </span>e.=
event_map) m<br><span style=3D"color:rgb(0,0,128);font-weight:bold">WHERE <=
/span>e.date_id =3D <span style=3D"color:rgb(0,0,255)">2420 </span><span st=
yle=3D"color:rgb(0,0,128);font-weight:bold">AND </span>e.event_id =3D <span=
 style=3D"color:rgb(0,0,255)">10012<br></span>)<br><span style=3D"color:rgb=
(0,0,128);font-weight:bold">SELECT </span>won, <span style=3D"color:rgb(0,0=
,128);font-weight:bold">count</span>(*) <span style=3D"color:rgb(0,0,128);f=
ont-weight:bold">FROM </span>battles<br><span style=3D"color:rgb(0,0,128);f=
ont-weight:bold">GROUP BY </span>won;</pre></div><div>Map element with key =
=3D &#39;won&#39; has only &#39;true&#39; and &#39;false&#39; values. If ba=
ttles subquery has more than 5 million rows the query returns strange resul=
ts, and when it has less rows query returns correct data.</div><div><br></d=
iv><div>Are you aware of this issue? I didn&#39;t send you my data since it=
 is pretty large for mail (20 GB).</div><div><br></div><div>Thanks</div><di=
v><br clear=3D"all"><div><div class=3D"gmail_signature"><div dir=3D"ltr"><f=
ont face=3D"Arial"><b>Dejan Proki=C4=87</b> | Data Engineer | Nordeus</font=
><br></div></div></div>
</div></div>

--94eb2c0b770e69b5020544a0e5c0--