Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of ananth.t.sarathy@gmail.com
 designates 209.85.218.214 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=b10Uj+RiulxXAoexs+FLDISHaYPRKvEdzdR6Nj2ewDrNXRxN2QPtv8e/pNt08L/wy6
         4yWtFPsFmOfi5VtcIy3ER/vAJy3XSqETGhpDwgU/Mx+dyGuhQnocznv7yRL4OYutANkL
         p9djfRU8keIEMWqBEVO2ThMFL7l6cp7D+O44s=
MIME-Version: 1.0
In-Reply-To: <C6B2C879.F417%scott@richrelevance.com>
References: <ad681e7f0908200802k36acf528mcb9b7b6acc0cd316@mail.gmail.com>
	 <C6B2C879.F417%scott@richrelevance.com>
Date: Thu, 20 Aug 2009 12:48:48 -0400
Message-ID: <ad681e7f0908200948o35f2d2e3g6dc5a25875837605@mail.gmail.com>
Subject: Re: Faster alternative to FSDataInputStream
From: "Ananth T. Sarathy" <ananth.t.sarathy@gmail.com>
To: common-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=001485f7d8de90bddc047195865d

--001485f7d8de90bddc047195865d
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

ok.. i seems that's the case.  that seems kind of  selfdefeating though.

Ananth T Sarathy


On Thu, Aug 20, 2009 at 12:31 PM, Scott Carey <scott@richrelevance.com>wrote:

> If it always takes a very long time to start transferring data, get a few
> stack dumps (jstack or kill -e) during this period to see what it is doing
> during this time.
>
> Most likely, the client is doing nothing but waiting on the remote side.
>
>
> On 8/20/09 8:02 AM, "Ananth T. Sarathy" <ananth.t.sarathy@gmail.com>
> wrote:
>
> > it's not really 1 mbps so much it takes 2 minutes to start doing the
> > reads.....
> >
> > Ananth T Sarathy
> >
> >
> > On Wed, Aug 19, 2009 at 4:30 PM, Scott Carey <scott@richrelevance.com
> >wrote:
> >
> >>
> >> On 8/19/09 10:58 AM, "Raghu Angadi" <rangadi@yahoo-inc.com> wrote:
> >>
> >>> Edward Capriolo wrote:
> >>>>> On Wed, Aug 19, 2009 at 11:11 AM, Edward Capriolo
> >>>>> <edlinuxguru@gmail.com>wrote:
> >>>>>
> >>>>>>>> It would be as fast as underlying filesystem goes.
> >>>>>> I would not agree with that statement. There is overhead.
> >>>
> >>> You might be misinterpreting my comment. There is of course some over
> >>> head (at the least the procedure calls).. depending on you underlying
> >>> filesystem, there could be extra buffer copies and CRC overhead. But
> >>> none of that explains transfer as slow as 1 MBps (if my interpretation
> >>> of of results is correct).
> >>>
> >>> Raghu.
> >>
> >>
> >> Yes, there is nothing about distributing work for parallel execution
> that
> >> is
> >> going to make a single 20MB file transfer faster.   That is very slow,
> and
> >> should be on the order of a second or so, not multiple minutes.
> >>  Something else is wrong.
> >>
> >>
> >>
> >
>
>

--001485f7d8de90bddc047195865d--