Return-Path: Delivered-To: apmail-hadoop-hive-user-archive@minotaur.apache.org Received: (qmail 94734 invoked from network); 6 Oct 2010 20:43:49 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 6 Oct 2010 20:43:49 -0000 Received: (qmail 95844 invoked by uid 500); 6 Oct 2010 20:43:48 -0000 Delivered-To: apmail-hadoop-hive-user-archive@hadoop.apache.org Received: (qmail 95779 invoked by uid 500); 6 Oct 2010 20:43:48 -0000 Mailing-List: contact hive-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hive-user@hadoop.apache.org Delivered-To: mailing list hive-user@hadoop.apache.org Received: (qmail 95771 invoked by uid 99); 6 Oct 2010 20:43:48 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Oct 2010 20:43:48 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_HELO_PASS,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of apache@jacobrideout.net designates 74.50.50.180 as permitted sender) Received: from [74.50.50.180] (HELO jacobrideout.net) (74.50.50.180) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 06 Oct 2010 20:43:40 +0000 Received: from mail-bw0-f48.google.com (mail-bw0-f48.google.com [209.85.214.48]) (using TLSv1 with cipher RC4-MD5 (128/128 bits)) (No client certificate requested) by jacobrideout.net (Postfix) with ESMTPSA id 95FF740113 for ; Wed, 6 Oct 2010 14:43:19 -0600 (MDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=jacobrideout.net; s=2009a; t=1286397799; bh=sDf17WF2Wa5qkZ07zU01+dCmhuwXqPYjvi4WsuN5M 7A=; h=MIME-Version:In-Reply-To:References:From:Date:Message-ID: Subject:To:Content-Type:Content-Transfer-Encoding; b=iJCjmhFvQu0nO hdDmZ6g2ipp0gkmoKSXmtiVOYA0WzYA1CFxDtezx+lX1tsqT2/4pKJZPzlWJq/jzdRA A+DTxnULGlfVBFmIB0MxdU7LFkg9dtCADX7NlDzD270nZk1NZ+yr//vbP83ElU6cJK4 pvvMafsKIeiCWZgfGErSZXUo= Received: by bwz14 with SMTP id 14so8689219bwz.35 for ; Wed, 06 Oct 2010 13:43:17 -0700 (PDT) Received: by 10.204.123.137 with SMTP id p9mr10057171bkr.206.1286397797736; Wed, 06 Oct 2010 13:43:17 -0700 (PDT) MIME-Version: 1.0 Received: by 10.204.59.206 with HTTP; Wed, 6 Oct 2010 13:42:57 -0700 (PDT) In-Reply-To: <516763.59949.qm@web110401.mail.gq1.yahoo.com> References: <832828.22906.qm@web110415.mail.gq1.yahoo.com> <205715.68702.qm@web110404.mail.gq1.yahoo.com> <690191.95062.qm@web110414.mail.gq1.yahoo.com> <516763.59949.qm@web110401.mail.gq1.yahoo.com> From: Jacob R Rideout Date: Wed, 6 Oct 2010 14:42:57 -0600 Message-ID: Subject: Re: How to output SeqFile To: hive-user@hadoop.apache.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org On Wed, Oct 6, 2010 at 2:35 PM, gaurav jain wrote: > I do have that. > > However I am not writing directly to the table partition. Instead, I firs= t write > my data in a tmp directory (eventually moved to the hdfs table partition)= =C2=A0and > then publish that partition using alter table statement in metastore. > > Something like this: > > -- create table x ... stored as SeqFile > -- insert overwrite directory 'd' select * from table y > -- distcp 'd' =C2=A0x/dateint=3D.../hour=3D... > -- alter table x add partition .... > > In the second step above I need to produce SeqFile. > > > Thanks for prompt reply. > Gaurav Jain > > > ----- Original Message ---- > From: Yang > To: jainy_gaurav@yahoo.com > Sent: Wed, October 6, 2010 1:28:42 PM > Subject: Re: How to output SeqFile > > Gaurav: > > not sure if I understand your question correctly.... > when you create the output table, that has an option to set the > output table SerDe > > Regards > Yang > > On Wed, Oct 6, 2010 at 1:18 PM, gaurav jain wrot= e: >> >> >> >> >> How can I produce a sequence file from query >> >> insert overwrite directory .... >> >> >> I have set: >> >> SET io.seqfile.compression.type=3DBLOCK; >> SET hive.exec.compress.output=3Dtrue; >> set mapred.output.compression.type=3DBLOCK; >> set mapred.output.compression.codec=3Dorg.apache.hadoop.io.compress.Gzip= Codec; >> >> >> >> It seems to produce Text .gz format files. >> >> >> >> Regards, >> Gaurav Jain >> >> >> >> > > > > > if you are inserting into the directory rather than the table, hive won't know to look at the metadata description of the table you need something like: insert overwrite table x select * from table y