Return-Path: X-Original-To: apmail-hadoop-common-dev-archive@www.apache.org Delivered-To: apmail-hadoop-common-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 6CF88DA86 for ; Mon, 23 Jul 2012 03:25:46 +0000 (UTC) Received: (qmail 65621 invoked by uid 500); 23 Jul 2012 03:25:44 -0000 Delivered-To: apmail-hadoop-common-dev-archive@hadoop.apache.org Received: (qmail 63799 invoked by uid 500); 23 Jul 2012 03:25:35 -0000 Mailing-List: contact common-dev-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: common-dev@hadoop.apache.org Delivered-To: mailing list common-dev@hadoop.apache.org Received: (qmail 63720 invoked by uid 99); 23 Jul 2012 03:25:32 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Jul 2012 03:25:32 +0000 X-ASF-Spam-Status: No, hits=-0.5 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of abhishek.dodda1@gmail.com designates 209.85.160.176 as permitted sender) Received: from [209.85.160.176] (HELO mail-gh0-f176.google.com) (209.85.160.176) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 Jul 2012 03:25:25 +0000 Received: by ghbz10 with SMTP id z10so5949736ghb.35 for ; Sun, 22 Jul 2012 20:25:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:from:date:message-id:subject:to:cc:content-type; bh=myuUUefa7yNMo3InX2kjcOg1dIaRkPVOCvWB7AZXoPw=; b=xnxdDLXHGPmQE6bdDQNRN5tTLeMQp9KPYMGUEs9AiSjlW31XKLbxMhtVpcV+txtFa9 He/WGDw/4TRGYADBYjDbOUIr5fb9y6dV4Tw5xB+V1GztxH6TLfP7WzZubxYxkHmQ27rN DU0wI2T2ABctrqKHXWOlq8gTXz2GTqR2+tLOu8T7l+ztAvJkej/c0H/E3YH4XetPYeVa O5xl+JtnCbBq3W9eh47UsAdRgcATmtRzIl88SrPevYtwKzrwTWpmMAPjbtVeLkZ0YnQ8 2yHx4UUhwReIZfOMCTtY3tYZyiGhkNrpEOeME0Xxo0RXkdFexFqgZbHttz1ORJYxF9hD YoXQ== Received: by 10.42.22.206 with SMTP id p14mr7540613icb.23.1343013904752; Sun, 22 Jul 2012 20:25:04 -0700 (PDT) MIME-Version: 1.0 Received: by 10.42.148.73 with HTTP; Sun, 22 Jul 2012 20:24:44 -0700 (PDT) From: abhiTowson cal Date: Sun, 22 Jul 2012 23:24:44 -0400 Message-ID: Subject: hive query optimization To: user@hive.apache.org, common-dev Cc: dev@hive.apache.org Content-Type: text/plain; charset=ISO-8859-1 X-Virus-Checked: Checked by ClamAV on apache.org Hi all, Some queries in hive are executing for too long.So i have overriden some parameters in hive, for some querys performance increased rapidly when i overriden this properities for some querys no change in performance.Can any one you tell me any other optimizations in hive apart from partitions and buckets, set io.sort.mb=512; set io.sort.factor=100; set mapred.reduce.parallel.copies=40; set hive.map.aggr =true; set hive.exec.parallel=true; set hive.groupby.skewindata=true; set mapred.job.reuse.jvm.num.tasks=-1; default values were io.sort.mb=256; io.sort.factor=10; mapred.reduce.parallel.copies=10; Thanks Abhishek