Mailing-List: contact user-help@carbondata.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@carbondata.apache.org
MIME-Version: 1.0
In-Reply-To: <CAPcQB38Q70Mk09o=DOu9Gb5q8YRPVYFZuVAPh_-hNkjEJgn80Q@mail.gmail.com>
References: <CAPcQB3_X_nTRtCCovBVtY-YixvsrvHjViWo_at2ehg7mV=ujEg@mail.gmail.com>
 <CAPcQB3-mibj2y0si_BHi4xz4SYt0VQqFieNpNFcQQ0WQQhmF=g@mail.gmail.com> <CAPcQB38Q70Mk09o=DOu9Gb5q8YRPVYFZuVAPh_-hNkjEJgn80Q@mail.gmail.com>
From: Ravindra Pesala <ravi.pesala@gmail.com>
Date: Thu, 3 Aug 2017 19:34:17 +0530
Message-ID: <CAGRwxWn9UexEr1q1Gy=OqkK44VijyQyn33Ax=L9irRNFCNp-7Q@mail.gmail.com>
Subject: Re: spark2.2.0 support
To: user@carbondata.apache.org
Content-Type: multipart/alternative; boundary="f403045f50483b6db00555d9db86"
archived-at: Thu, 03 Aug 2017 14:04:23 -0000

--f403045f50483b6db00555d9db86
Content-Type: text/plain; charset="UTF-8"

Hi,

In the ongoing version we support only spark 2.1.1 version, As spark 2.2.0
is relatively new major released version we require more time for upgrading
and testing Carbondata completely. So as per the effort we plan the release
of upgraded version.

And about your query , yes the code should be version independent, that
means same should work for 2.1.0, 2.1.1 and 2.2.0 as well.

Regards,
Ravindra.

On 3 August 2017 at 12:57, john cheng <zqhxuyuan@gmail.com> wrote:

> After dig source code about error happened(https://github.com/
> apache/carbondata/blob/master/integration/spark-common/src/
> main/scala/org/apache/spark/sql/optimizer/CarbonDecoderOptimizerHelper.
> scala#L87) between spark1.6/spark2.1.x and spark2.2.0 . I notice
> spark1.6's InsertIntoTable has child LogicalPlan attribute, but spark2.2.0
> change to query LogicalPlan attribute. but CarbonDecoderOptimizerHelper
> is at spark-common module, which means spark1.6/spark2.1 and spark2.2 both
> use this class. So I change to i.query, then build with spark2.2.0, It's OK
> then. Should this code here being version compatible or version independent?
>
> 2017-08-03 14:52 GMT+08:00 john cheng <zqhxuyuan@gmail.com>:
>
>> If I build carbondata with spark2.1.x, it works. But our spark version is
>> spark2.2.0. If use spark2.1.x builded jar, and execute on spark2.2.0,
>> there're still errors when create CarbonSession, the errors is :
>> ClassNotFoundException: o.a.s.sql.hive.HiveSessionState
>>
>> 2017-08-03 14:45 GMT+08:00 john cheng <zqhxuyuan@gmail.com>:
>>
>>> Hi carbon guys, At now carbondata seems not support spark2.2.0. I add
>>> spark2.2 as a new profile, and build like this: mvn -DskipTests -Pspark-2.2
>>> -Dspark.version=2.2.0 -Dhadoop.version=2.6.0 clean package
>>>
>>> But there're errors on spark common module:
>>>
>>> [ERROR] /Users/zhengqh/Github/carbondata-parent-1.1.1/integration/sp
>>> ark-common/src/main/scala/org/apache/spark/sql/optimizer/
>>> CarbonDecoderOptimizerHelper.scala:87: error: value child is not a
>>> member of org.apache.spark.sql.catalyst.plans.logical.InsertIntoTable
>>> [INFO]       case i: InsertIntoTable => process(i.child, nodeList)
>>> [INFO]                                            ^
>>> [WARNING] 11 warnings found
>>> [ERROR] one error found
>>>
>>> do you guys plan to support spark2.2.0. or at now I should downgrade to
>>> spark2.1.x?
>>>
>>
>>
>


-- 
Thanks & Regards,
Ravi

--f403045f50483b6db00555d9db86
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi,<div><br></div><div>In the ongoing version we support o=
nly spark 2.1.1 version, As spark 2.2.0 is relatively new major released ve=
rsion we require more time for upgrading and testing Carbondata completely.=
 So as per the effort we plan the release of upgraded version.</div><div><b=
r></div><div>And about your query , yes the code should be version independ=
ent, that means same should work for 2.1.0, 2.1.1 and 2.2.0 as well.<br><di=
v><br></div><div>Regards,</div><div>Ravindra.</div></div></div><div class=
=3D"gmail_extra"><br><div class=3D"gmail_quote">On 3 August 2017 at 12:57, =
john cheng <span dir=3D"ltr">&lt;<a href=3D"mailto:zqhxuyuan@gmail.com" tar=
get=3D"_blank">zqhxuyuan@gmail.com</a>&gt;</span> wrote:<br><blockquote cla=
ss=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;pa=
dding-left:1ex"><div dir=3D"ltr">After dig source code about error happened=
(<a href=3D"https://github.com/apache/carbondata/blob/master/integration/sp=
ark-common/src/main/scala/org/apache/spark/sql/optimizer/CarbonDecoderOptim=
izerHelper.scala#L87" target=3D"_blank">https://github.com/<wbr>apache/carb=
ondata/blob/master/<wbr>integration/spark-common/src/<wbr>main/scala/org/ap=
ache/spark/<wbr>sql/optimizer/<wbr>CarbonDecoderOptimizerHelper.<wbr>scala#=
L87</a>) between spark1.6/spark2.1.x and spark2.2.0 . I notice spark1.6&#39=
;s InsertIntoTable has child LogicalPlan attribute, but spark2.2.0 change t=
o query LogicalPlan attribute. but=C2=A0<wbr>CarbonDecoderOptimizerHelper i=
s at spark-common module, which means spark1.6/spark2.1 and spark2.2 both u=
se this class. So I change to i.query, then build with spark2.2.0, It&#39;s=
 OK then. Should this code here being version compatible or version indepen=
dent?</div><div class=3D"HOEnZb"><div class=3D"h5"><div class=3D"gmail_extr=
a"><br><div class=3D"gmail_quote">2017-08-03 14:52 GMT+08:00 john cheng <sp=
an dir=3D"ltr">&lt;<a href=3D"mailto:zqhxuyuan@gmail.com" target=3D"_blank"=
>zqhxuyuan@gmail.com</a>&gt;</span>:<br><blockquote class=3D"gmail_quote" s=
tyle=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div=
 dir=3D"ltr"><div>If I build carbondata with spark2.1.x, it works. But our =
spark version is spark2.2.0. If use spark2.1.x builded jar, and execute on =
spark2.2.0, there&#39;re still errors when create CarbonSession, the errors=
 is : ClassNotFoundException: o.a.s.sql.hive.HiveSessionStat<wbr>e</div></d=
iv><div class=3D"m_7873018178787302647HOEnZb"><div class=3D"m_7873018178787=
302647h5"><div class=3D"gmail_extra"><br><div class=3D"gmail_quote">2017-08=
-03 14:45 GMT+08:00 john cheng <span dir=3D"ltr">&lt;<a href=3D"mailto:zqhx=
uyuan@gmail.com" target=3D"_blank">zqhxuyuan@gmail.com</a>&gt;</span>:<br><=
blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px=
 #ccc solid;padding-left:1ex"><div dir=3D"ltr">Hi carbon guys, At now carbo=
ndata seems not support spark2.2.0. I add spark2.2 as a new profile, and bu=
ild like this:=C2=A0mvn -DskipTests -Pspark-2.2 -Dspark.version=3D2.2.0 -Dh=
adoop.version=3D2.6.0 clean package<div><br></div><div>But there&#39;re err=
ors on spark common module:</div><div><br></div><div><div>[ERROR] /Users/zh=
engqh/Github/carbonda<wbr>ta-parent-1.1.1/integration/sp<wbr>ark-common/src=
/main/scala/org/<wbr>apache/spark/sql/optimizer/<wbr>CarbonDecoderOptimizer=
Helper.<wbr>scala:87: error: value child is not a member of org.apache.spar=
k.sql.catalyst.<wbr>plans.logical.InsertIntoTable</div><div>[INFO] =C2=A0 =
=C2=A0 =C2=A0 case i: InsertIntoTable =3D&gt; process(i.child, nodeList)</d=
iv><div>[INFO] =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0^</div><div>[WARNING] 11 warnings found</div><div>[ERRO=
R] one error found</div></div><div><br></div><div>do you guys plan to suppo=
rt spark2.2.0. or at now I should downgrade to spark2.1.x?</div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br><br clear=3D"all"><div><br></div>-- <br>=
<div class=3D"gmail_signature" data-smartmail=3D"gmail_signature">Thanks &a=
mp; Regards,<br>Ravi</div>
</div>

--f403045f50483b6db00555d9db86--