pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rohini Palaniswamy <rohini.adi...@gmail.com>
Subject Re: Dependency version on Kryo
Date Wed, 14 Oct 2015 17:15:33 GMT
That should be fine. We wanted to get rid of the kryo dependency in ORC and
use the shaded one that hive uses. But that is in hive-exec jar which is
huge and has too many other jars packed in and did not want to add that as
dependency to Pig.

On Mon, Oct 12, 2015 at 8:54 PM, Xuefu Zhang <xzhang@cloudera.com> wrote:

> Hi all,
>
> It was found in PIG-4693 (https://issues.apache.org/jira/browse/PIG-4693)
> that Pig is currently dependent on Kryo 2.22. However, Spark depends on
> 2.21. The two versions are not completely compatible. We tried several ways
> to solve the problem but unfortunately none worked. This is mainly because
> Spark doesn't give user an opportunity to provide their own kryo library
> (SPARK-10910). Please refer to the full discussions in PIG-4693.
>
> It seems that Pig brought in kryo dependency for ORC. I'm wondering if
> there is any specific reasons for kryo 2.22 and if not, whether we can
> downgrade the dependency to 2.21 instead. Our initial test shows that kryo
> 2.21 works just fine for ORC. This obviously solve our problem as well.
>
> Your input to this is greatly appreciated.
>
> Thanks,
> Xuefu
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message