Return-Path: Delivered-To: apmail-hadoop-core-user-archive@www.apache.org Received: (qmail 14664 invoked from network); 21 Apr 2008 06:33:52 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 21 Apr 2008 06:33:52 -0000 Received: (qmail 49678 invoked by uid 500); 21 Apr 2008 06:33:46 -0000 Delivered-To: apmail-hadoop-core-user-archive@hadoop.apache.org Received: (qmail 49645 invoked by uid 500); 21 Apr 2008 06:33:45 -0000 Mailing-List: contact core-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: core-user@hadoop.apache.org Delivered-To: mailing list core-user@hadoop.apache.org Received: (qmail 49636 invoked by uid 99); 21 Apr 2008 06:33:45 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 20 Apr 2008 23:33:45 -0700 X-ASF-Spam-Status: No, hits=1.2 required=10.0 tests=SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [203.99.254.144] (HELO rsmtp2.corp.hki.yahoo.com) (203.99.254.144) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Apr 2008 06:32:59 +0000 Received: from [10.66.74.35] (sevenshade-lr.eglbp.corp.yahoo.com [10.66.74.35]) by rsmtp2.corp.hki.yahoo.com (8.13.8/8.13.8/y.rout) with ESMTP id m3L6X1Lq026350 for ; Sun, 20 Apr 2008 23:33:02 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns; h=message-id:date:from:user-agent:mime-version:to:subject: references:in-reply-to:content-type:content-transfer-encoding; b=1RqVSXgLWYDzdfwzsJWi0nJesbZPA7IspjiRBW0YPdbWkMSzhGO2EYDaIFqL6cQh Message-ID: <480C351D.10408@yahoo-inc.com> Date: Mon, 21 Apr 2008 12:03:01 +0530 From: Amar Kamat User-Agent: Thunderbird 2.0.0.12 (X11/20080213) MIME-Version: 1.0 To: core-user@hadoop.apache.org Subject: Re: Splitting in various files References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org Aayush Garg wrote: > Could anyone please tell? > > On Sat, Apr 19, 2008 at 1:33 PM, Aayush Garg wrote: > > >> Hi, >> >> I have written the following code for writing my key,value pairs in the >> file, and this file is then read by another MR. >> >> Path pth = new Path("./dir1/dir2/filename"); >> FileSystem fs = pth.getFileSystem(jobconf); >> SequenceFile.Writer sqwrite = new >> SequenceFile.Writer(fs,conf,pth,Text.class,Custom.class); >> sqwrite.append(Key,value); >> sqwrite.close(); >> >> I problem is I get my data written in one file(filename).. How can it be >> split across in the number of files. If I give only the path of directory in >> What do you mean by splitting a file across multiple files? If you want a separate file for each map/reduce task then you can use conf.get("mapred.task.id") to get the task id that is unique for that task. Now you can name the file like Path pth = new Path("./dir1/dir2/" + filename + "-" + conf.get("mapred.task.id")); Amar >> this progam then it does not get compiled. >> >> I give only the path of directory /dir1/dir2 to another Map Reduce and it >> reads the file. >> >> Thanks, >> >> >> > > >