Mailing-List: contact user-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hive.apache.org
User-Agent: Microsoft-MacOutlook/14.5.2.150604
Date: Tue, 14 Jul 2015 18:26:59 -0700
Subject: Re: Optimizing UDF
From: Gopal Vijayaraghavan <gopalv@apache.org>
Sender: Gopal Vijayaraghavan <gopal@hortonworks.com>
To: "user@hive.apache.org" <user@hive.apache.org>
Message-ID: <D1CB03F4.30493%gopal@hortonworks.com>
Thread-Topic: Optimizing UDF
References: <BLU168-W94C736CF9C7DE5661F8487B49B0@phx.gbl>
 <D1CAD783.303CA%gopal@hortonworks.com>
 <BLU168-W27A57DB2EEBB71583546A7B49B0@phx.gbl>
In-Reply-To: <BLU168-W27A57DB2EEBB71583546A7B49B0@phx.gbl>
Mime-version: 1.0
Content-type: text/plain;
	charset="ISO-8859-1"
Content-transfer-encoding: quoted-printable


=20
> I am already using Tez (sorry, forgot to mention this), and my goal is
>indeed to build the instance once per container.

Put a log line in your UDF init() and check if it is being called multiple
times per container. If you=B9re loading the data everytime, then that might
be something to fix.

The other aspect is that there=B9s GC pauses that can happen due to that and
such extraneous reasons for the slow-down.

But first, look at how many times you are loading the distributed cache
data per container.

Cheers,
Gopal