Mailing-List: contact user-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hive.apache.org
User-Agent: Microsoft-MacOutlook/14.5.1.150515
Date: Wed, 17 Jun 2015 23:47:12 -0700
Subject: Re: Hive - Tez error with big join - Container expired.
From: Gopal Vijayaraghavan <gopalv@apache.org>
Sender: Gopal Vijayaraghavan <gopal@hortonworks.com>
To: "user@hive.apache.org" <user@hive.apache.org>
Message-ID: <D1A7B6B3.2E8EF%gopal@hortonworks.com>
Thread-Topic: Hive - Tez error with big join - Container expired.
References: <00f001d0a945$8a03d950$9e0b8bf0$@web-computing.de>
In-Reply-To: <00f001d0a945$8a03d950$9e0b8bf0$@web-computing.de>
Mime-version: 1.0
Content-type: text/plain;
	charset="ISO-8859-1"
Content-transfer-encoding: quoted-printable

> I have a pretty big Hive Query. I=B9m joining over 3 Hive-Tables which
>have thousands of lines each. I=B9m grouping this join by several columns.

Hive-on-Tez shouldn=B9t have any issue even with billion of lines on a JOIN.

> 0 failed, info=3D[Containercontainer_1434357133795_0008_01_000039 finished
>while trying to launch. Diagnostics: [Container failed. Container expired
>since it was unused]], TaskAttempt 1 failed,

Looks like your node manager is actually not spinning up a container that
was allocated (i.e allocation succeeded, but the task spin up failed).

Which YARN scheduler are you running (fair/capacity?) and do you have any
idea on what the logs on the NodeManager logs say about trying to spin up
this container?

If I=B9m not wrong, you need to also check if the YARN user has a ulimit set
for the total number of processes on the NM nodes.


=20
Cheers,
Gopal