Mailing-List: contact dev-help@spark.apache.org; run by ezmlm
Precedence: bulk
Received-SPF: neutral (athena.apache.org: local policy)
MIME-Version: 1.0
In-Reply-To: 
 <CAAswR-4nM0JteU8zbwZT-V69JJtWEhmbSw1M2Xw2sSxmU59oig@mail.gmail.com>
References: 
 <CAJ4HpHEND==wxS9TcpeBKC9dFbc-odVidFNMYYxnqu+Ng89rdw@mail.gmail.com>
 <CAAswR-5Qu2kj=EFoz4uT+PdDGK2Tsi4N1WarVS=XDami8iWcgA@mail.gmail.com>
 <CAJ4HpHEgppwd0J+pJouG5PMDdXrkXHSq6weYYdoHOU04veLs+A@mail.gmail.com>
 <CAAswR-4nM0JteU8zbwZT-V69JJtWEhmbSw1M2Xw2sSxmU59oig@mail.gmail.com>
From: Michael Armbrust <michael@databricks.com>
Date: Wed, 3 Dec 2014 11:25:25 -0800
Message-ID: 
 <CAAswR-4d2LiScJCg1kytbsb4VQdr4VJ8uZjnYYnE0JFy5NBZbQ@mail.gmail.com>
Subject: Re: [Thrift,1.2 RC] what happened to
 parquet.hive.serde.ParquetHiveSerDe
To: Yana Kadiyska <yana.kadiyska@gmail.com>
Cc: "dev@spark.incubator.apache.org" <dev@spark.incubator.apache.org>
Content-Type: multipart/alternative; boundary=001a11c37eb4763558050954cf28

--001a11c37eb4763558050954cf28
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Here's a fix: https://github.com/apache/spark/pull/3586

On Wed, Dec 3, 2014 at 11:05 AM, Michael Armbrust <michael@databricks.com>
wrote:

> Thanks for reporting. As a workaround you should be able to SET
> spark.sql.hive.convertMetastoreParquet=3Dfalse, but I'm going to try to f=
ix
> this before the next RC.
>
> On Wed, Dec 3, 2014 at 7:09 AM, Yana Kadiyska <yana.kadiyska@gmail.com>
> wrote:
>
>> Thanks Michael, you are correct.
>>
>> I also opened https://issues.apache.org/jira/browse/SPARK-4702 -- if
>> someone can comment on why this might be happening that would be great.
>> This would be a blocker to me using 1.2 and it used to work so I'm a bit
>> puzzled. I was hoping that it's again a result of the default profile
>> switch but it didn't seem to be the case
>>
>> (ps. please advise if this is more user-list appropriate. I'm posting to
>> dev as it's an RC)
>>
>> On Tue, Dec 2, 2014 at 8:37 PM, Michael Armbrust <michael@databricks.com=
>
>> wrote:
>>
>>> In Hive 13 (which is the default for Spark 1.2), parquet is included an=
d
>>> thus we no longer include the Hive parquet bundle. You can now use the
>>> included
>>> ParquetSerDe: org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSer=
De
>>>
>>> If you want to compile Spark 1.2 with Hive 12 instead you can pass
>>> -Phive-0.12.0 and  parquet.hive.serde.ParquetHiveSerDe will be included=
 as
>>> before.
>>>
>>> Michael
>>>
>>> On Tue, Dec 2, 2014 at 9:31 AM, Yana Kadiyska <yana.kadiyska@gmail.com>
>>> wrote:
>>>
>>>> Apologies if people get this more than once -- I sent mail to dev@spar=
k
>>>> last night and don't see it in the archives. Trying the incubator list
>>>> now...wanted to make sure it doesn't get lost in case it's a bug...
>>>>
>>>> ---------- Forwarded message ----------
>>>> From: Yana Kadiyska <yana.kadiyska@gmail.com>
>>>> Date: Mon, Dec 1, 2014 at 8:10 PM
>>>> Subject: [Thrift,1.2 RC] what happened to
>>>> parquet.hive.serde.ParquetHiveSerDe
>>>> To: dev@spark.apache.org
>>>>
>>>>
>>>> Hi all, apologies if this is not a question for the dev list -- figure=
d
>>>> User list might not be appropriate since I'm having trouble with the R=
C
>>>> tag.
>>>>
>>>> I just tried deploying the RC and running ThriftServer. I see the
>>>> following
>>>> error:
>>>>
>>>> 14/12/01 21:31:42 ERROR UserGroupInformation: PriviledgedActionExcepti=
on
>>>> as:anonymous (auth:SIMPLE)
>>>> cause:org.apache.hive.service.cli.HiveSQLException:
>>>> java.lang.RuntimeException:
>>>> MetaException(message:java.lang.ClassNotFoundException Class
>>>> parquet.hive.serde.ParquetHiveSerDe not found)
>>>> 14/12/01 21:31:42 WARN ThriftCLIService: Error executing statement:
>>>> org.apache.hive.service.cli.HiveSQLException:
>>>> java.lang.RuntimeException:
>>>> MetaException(message:java.lang.ClassNotFoundException Class
>>>> parquet.hive.serde.ParquetHiveSerDe not found)
>>>> at
>>>>
>>>> org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.=
run(Shim13.scala:192)
>>>> at
>>>>
>>>> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementIn=
ternal(HiveSessionImpl.java:231)
>>>> at
>>>>
>>>> org.apache.hive.service.cli.session.HiveSessionImpl.executeStatement(H=
iveSessionImpl.java:212)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at
>>>>
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.j=
ava:57)
>>>> at
>>>>
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess=
orImpl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>> at
>>>>
>>>> org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessio=
nProxy.java:79)
>>>> at
>>>>
>>>> org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSe=
ssionProxy.java:37)
>>>> at
>>>>
>>>> org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSession=
Proxy.java:64)
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>> at javax.security.auth.Subject.doAs(Subject.java:415)
>>>> =E2=80=8B
>>>>
>>>>
>>>> I looked at a working installation that I have(build master a few week=
s
>>>> ago) and this class used to be included in spark-assembly:
>>>>
>>>> ls *.jar|xargs grep parquet.hive.serde.ParquetHiveSerDe
>>>> Binary file spark-assembly-1.2.0-SNAPSHOT-hadoop2.0.0-mr1-cdh4.2.0.jar
>>>> matches
>>>>
>>>> but with the RC build it's not there?
>>>>
>>>> I tried both the prebuilt CDH drop and later manually built the tag wi=
th
>>>> the following command:
>>>>
>>>>  ./make-distribution.sh --tgz -Phive -Dhadoop.version=3D2.0.0-mr1-cdh4=
.2.0
>>>> -Phive-thriftserver
>>>> $JAVA_HOME/bin/jar -tvf
>>>> spark-assembly-1.2.0-hadoop2.0.0-mr1-cdh4.2.0.jar
>>>> |grep parquet.hive.serde.ParquetHiveSerDe
>>>>
>>>> comes back empty...
>>>>
>>>
>>>
>>
>

--001a11c37eb4763558050954cf28--