Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 1A101E5A5 for ; Thu, 6 Dec 2012 23:08:41 +0000 (UTC) Received: (qmail 30846 invoked by uid 500); 6 Dec 2012 23:08:36 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 30751 invoked by uid 500); 6 Dec 2012 23:08:36 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 30744 invoked by uid 99); 6 Dec 2012 23:08:36 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Dec 2012 23:08:36 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of harsh@cloudera.com designates 209.85.223.176 as permitted sender) Received: from [209.85.223.176] (HELO mail-ie0-f176.google.com) (209.85.223.176) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Dec 2012 23:08:29 +0000 Received: by mail-ie0-f176.google.com with SMTP id 13so12281535iea.35 for ; Thu, 06 Dec 2012 15:08:09 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type:x-gm-message-state; bh=yHRF5zDG60rU2Kno7RvreSfqiZ3k8+6nV5cmLJxW27A=; b=aPsKzgPSRpVSX0f6GV4fTQKA7VLiORTi8K8FvS3jLvwRbihUo9KF9rrS7QbfKmAA3Z WdOloD6RWeeDhpdWrE9e9wtpjhyzawYwP+ZAJwoOdn5AmJiOjsj6YKDh0SNQoSHDeZhr EdBnNrOO8DTyem2FDkPbf16htwfYUWsC5eoSTzFfni3DjsrUrwDybDxfLrp/4DkYZxUJ QA4nJxwBQLHQBEzgdMangQAQr6GONsqxsIRQH37E/Zi3tLZTiiNH84anim0avNMkEHzQ QjLDx5s27YPP/5rJuR3kBHfl9o7dhk2qZxiXyhFcZeG+e51hFiZX7875+LY3u2iQY510 xGwQ== Received: by 10.43.46.2 with SMTP id um2mr2836074icb.18.1354835288882; Thu, 06 Dec 2012 15:08:08 -0800 (PST) MIME-Version: 1.0 Received: by 10.64.6.129 with HTTP; Thu, 6 Dec 2012 15:07:48 -0800 (PST) In-Reply-To: References: From: Harsh J Date: Fri, 7 Dec 2012 04:37:48 +0530 Message-ID: Subject: Re: DFS and the RecordReader To: "" Content-Type: text/plain; charset=ISO-8859-1 X-Gm-Message-State: ALoCoQlpTmvs9eXGVfPQICuNOIfmf5JbWfNXRqvFnvvKmgp/2vsrbxG2cQn3ODOgWNaRsrj4lFfV X-Virus-Checked: Checked by ClamAV on apache.org Ah ok, understood what you seem to be looking for. Lets follow the simple LineReader implementation in that case. TextInputFormat uses LineRecordReader: [1] - Line 52 LineRecordReader has the calls you look for and wraps over a LineReader implementation, to take care of reading lines over block boundaries: [2] - Line 88 LineReader has all the functional code to make it work for anyone reading lines off of text files: [3] [1] - http://svn.apache.org/viewvc/hadoop/common/tags/release-2.0.2-alpha/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/TextInputFormat.java?view=markup [2] - http://svn.apache.org/viewvc/hadoop/common/tags/release-2.0.2-alpha/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/LineRecordReader.java?view=markup [3] - http://svn.apache.org/viewvc/hadoop/common/tags/release-2.0.2-alpha/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LineReader.java?view=markup On Fri, Dec 7, 2012 at 4:17 AM, Jay Vyas wrote: > Hmm... so when a record reader calls fs.open(...) , I guess Im looking for > an example of how the input stream is created... ? -- Harsh J