Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id ABA7610D03 for ; Thu, 27 Feb 2014 05:04:02 +0000 (UTC) Received: (qmail 70408 invoked by uid 500); 27 Feb 2014 05:03:54 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 69832 invoked by uid 500); 27 Feb 2014 05:03:45 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 69825 invoked by uid 99); 27 Feb 2014 05:03:43 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Feb 2014 05:03:43 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_REMOTE_IMAGE X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of dwivedishashwat@gmail.com designates 209.85.215.41 as permitted sender) Received: from [209.85.215.41] (HELO mail-la0-f41.google.com) (209.85.215.41) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 27 Feb 2014 05:03:39 +0000 Received: by mail-la0-f41.google.com with SMTP id gl10so1321074lab.14 for ; Wed, 26 Feb 2014 21:03:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=TABmSa9RfbNuKugaSjwERDy7N0FXwUPr3s6thP4SKtM=; b=hf87OvJ/X03BMjDaBlhUsw8RRSXCMgx4Z5UM5LtqZFvzV7rM6GUGDqOR83J9FAI7BV beb+P64pq3Y4/O6FXUdf5Qfy7NZcwPtW3XqmTyt68p6tYwx5PkWiCnlfIgCKrR6ThE4S rE0JjTLKlnrbndA7TwDRcyHWya+JUUFnUsf7xdZmHSXOlRJ78CjB2QjmxGy5373K1G5y IdEuq7I4IBfmy4q9dQKpm8D8rap59LHg3WlHB+FklIx1f32dQU1vZDYu1aywkQDq28f4 I/kmKfwY9LtJEJ2mqDSb0qH+AOPRVqeMwpSkv35mcFxiksHIP2tBthYP99KC8S1Hxg7V rfeg== X-Received: by 10.112.129.168 with SMTP id nx8mr4825008lbb.37.1393477397990; Wed, 26 Feb 2014 21:03:17 -0800 (PST) MIME-Version: 1.0 Received: by 10.114.75.137 with HTTP; Wed, 26 Feb 2014 21:02:57 -0800 (PST) In-Reply-To: <006001cf3373$a93a7570$fbaf6050$@gmail.com> References: <006001cf3373$a93a7570$fbaf6050$@gmail.com> From: shashwat shriparv Date: Thu, 27 Feb 2014 10:32:57 +0530 Message-ID: Subject: Re: Client and NN communication in Hadoop To: user@hadoop.apache.org Cc: navaz.enc@gmail.com Content-Type: multipart/alternative; boundary=047d7b343d684f536604f35c3d76 X-Virus-Checked: Checked by ClamAV on apache.org --047d7b343d684f536604f35c3d76 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable The application that try to write the file, for example the binary "hadoop" found in the bin dir of hadoop is a hadoop client, whoever wants to read or write from to hadoop is a client, you can also write hadoop client using java apis which hadoop provide. like following is an example of hadoop client which tries to read file from hadoop import java.io.*; import java.util.*; import java.net.*; import org.apache.hadoop.fs.*; import org.apache.hadoop.conf.*; import org.apache.hadoop.io.*; import org.apache.hadoop.mapred.*; import org.apache.hadoop.util.*; public class Cat{ public static void main (String [] args) throws Exception{ try{ Path pt=3Dnew Path("hdfs://shashwat:9000/home/shriparv/fileonhdfstoread.txt"); FileSystem filesysObject =3D FileSystem.get(new Configuration()); BufferedReader bufferReaderObject=3Dnew BufferedReader(new InputStreamReader(filesysObject.open(pt))); String line; line=3DbufferReaderObject.readLine(); while (line !=3D null){ System.out.println(line); line=3DbufferReaderObject.readLine(); } }catch(Exception e){ } } } Hope this clear some of your doubts * Warm Regards_**=E2=88=9E_* * Shashwat Shriparv* [image: http://www.linkedin.com/pub/shashwat-shriparv/19/214/2a9][image: https://twitter.com/shriparv] [image: https://www.facebook.com/shriparv] [imag= e: http://google.com/+ShashwatShriparv] [image: http://www.youtube.com/user/sShriparv/videos][image: http://profile.yahoo.com/SWXSTW3DVSDTF2HHSRM47AV6DI/] On Thu, Feb 27, 2014 at 9:53 AM, navaz wrote: > Hi > > > > I have a Hadoop cluster running with 3 slaves and 1 Master. Slaves are > Datanodes and running Tasktarckers. Namenode is running jobtracker and > secondary namenode. I am running sample mapreduce after loading a file in= to > HDFS. > > > > According to Hadoop architecture ,before writing a file into HDFS , > client will contact the Namenode and get the location details of DNs and > then client directly write the file into DN. What is this client ? is it > an application running on Namenode ? Is user and client both are differen= t > ? How can I see the messages between client and datanodes? > > > > Appreciate your response. > > > > > > Thanks & Regards > > > > Abdul Navaz > > Graduate Student > > University of Houston Main Campus,Texas > > Mob: (281) 685-0388 > > > --047d7b343d684f536604f35c3d76 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
The application that try to w= rite the file, for example the binary "hadoop" found in the bin d= ir of hadoop is a hadoop client, whoever wants to read or write from to had= oop is a client, you can also write hadoop client using java apis which had= oop provide.

like = following is an example of hadoop client which tries to read file from hado= op

i= mport java.io.*;
import java.util.*;
import java.net.*;
=
import org.apache.hadoo= p.fs.*;
import org.apache.hadoop.conf.*;
import org.apache.hadoo= p.io.*;
import org.apache.hadoop.mapred.*;
import org.apache.hadoo= p.util.*;
=C2=A0
public class Cat{
=C2=A0 =C2=A0 =C2=A0 =C2=A0 public static void main (String [] args) = throws Exception{
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 try{
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 Path pt=3Dnew Path("hdfs://shashwat:9000/home/shriparv/= fileonhdfstoread.txt");
=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 FileSystem f= ilesysObject =3D FileSystem.get(new Configuration());
=C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 BufferedReader bufferReaderObject=3Dnew BufferedReader(new InputStre= amReader(filesysObject.open(pt)));
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 String line;
=C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 line=3DbufferReader= Object.readLine();
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 while (line !=3D null){
=C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 System.out.println(line);
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 line=3DbufferReaderObject.readLi= ne();
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 }
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 }catch(Except= ion e){
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 }
=C2=A0 =C2=A0 =C2=A0 =C2=A0 }
}
<= br>
Hope this clear some of your doubts


Warm Regards_= =E2=88=9E_
= Shashwat Shrip= arv
<= font size=3D"4">
3D"http://www.linke=3D"ht=3D"https://www.f=3D"http://google.c=3D"http://www.you=<= font size=3D"4">3D"http://profil== <= /span>= <= /span>


On Thu, Feb 27, 2014 at 9:53 AM, navaz <= span dir=3D"ltr"><navaz.enc@gmail.com> wrote:

Hi

=C2=A0

I have a Hadoop cluster running with 3 slaves= and 1 Master. Slaves are Datanodes and running Tasktarckers. Namenode is r= unning jobtracker and secondary namenode. I am running sample mapreduce aft= er loading a file into HDFS.=C2=A0

=C2=A0

According to = Hadoop architecture ,before writing a file into HDFS , =C2=A0client will co= ntact the Namenode and get the location details of DNs and then client dire= ctly write the file into DN.=C2=A0 What is this client ? is it an applicati= on running on Namenode ? Is user and client both are different ? How can I = see the messages between client and datanodes?

=C2=A0

Appreciate yo= ur response.

=C2=A0

=C2=A0

Thanks & Regards

=C2=A0

Abdul Navaz=

Graduate Student

University of Houston Main Campus,Texas=

Mob: (281) 685-0388

=C2=A0


--047d7b343d684f536604f35c3d76--