hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Divij Durve <divij.t...@gmail.com>
Subject Re: hadoop cluster startup error
Date Wed, 15 Jul 2009 15:57:24 GMT
I moved over from hadoop-0.19.0 to 0.20.0 yesterday as i could not figure
out what exactly was going on. I have everything setup to work now 1
namenode + 2 datanodes. I am trying to speed up the retrieval of data from a
database using a perl script. The file consists of numbers and characters. A
chunk of that file is pulled out and then queried on a database. I was
wondering where i might find a tutorial on how to make all this work. Also
with 2 datanodes should dfs.replication be set to 2 or 1? not sure about
that.
Thanks
Divij

On Tue, Jul 14, 2009 at 9:57 PM, Aaron Kimball <aaron@cloudera.com> wrote:

> I think you might have the commands backwards. hadoop fs -get will
> copy from HDFS to the local filesystem. So it assumes that
> /usr/divij/wordcount/input is in HDFS and that you want to write it to
> local file "file01".
>
> If you want to put data into HDFS, do the opposite: hadoop fs -put
> (localfile) (hdfs_dest)
> Also note that whereas Linux/Unix fans type "/usr", Hadoop spells your
> home directory "/user/divij" (note the "e" in "user").
>
> As for not starting the JobTracker/TaskTracker: can you look through
> the JobTracker and TaskTracker logs in your $HADOOP_HOME/logs
> directory (should be named hadoop-divij-jobtracker-gobi.log and
> hadoop-divij-tasktracker-gobi.log) and reply to this thread with any
> ERROR or FATAL messages present in either?
>
> Thanks,
> - Aaron
>
>
> On Tue, Jul 14, 2009 at 12:47 PM, Divij Durve<divij.tech@gmail.com> wrote:
> > I have started the cluster and inspite of that i dont get any of the
> > processes when i type ps. the only thing its showing is 2 java processes
> on
> > the name node.
> >
> > im trying to do the word count example but this is what is happening:
> >
> > [divij@gobi bin]$ ./hadoop fs -get /usr/divij/wordcount/input ./file01
> > [divij@gobi bin]$ ./hadoop fs -get /usr/divij/wordcount/input ./file02
> > [divij@gobi bin]$ ./hadoop fs -ls
> > ls: Cannot access .: No such file or directory.
> > [divij@gobi bin]$ ./hadoop fs -ls /usr/divij/wordcount/input
> > [divij@gobi bin]$ ./hadoop dfs -cat >/usr/divij/wordcount/input/file01
> > -bash: /usr/divij/wordcount/input/file01: No such file or directory
> > [divij@gobi bin]$ ./hadoop dfs -cat /usr/divij/wordcount/input/file01
> > cat: File does not exist: /usr/divij/wordcount/input/file01
> > [divij@gobi bin]$ ./hadoop fs -get /usr/divij/wordcount/input/ ./file01
> > [divij@gobi bin]$ ./hadoop fs -cat /usr/divij/wordcount/input/file01
> > cat: File does not exist: /usr/divij/wordcount/input/file01
> > [divij@gobi bin]$ ./hadoop dfs -cat /usr/divij/wordcount/input/file01
> > cat: File does not exist: /usr/divij/wordcount/input/file01
> > [divij@gobi bin]$ ./hadoop fs -test -e /usr/divij/wordcount/input/file01
> > [divij@gobi bin]$ ./hadoop fs -tail /usr/divij/wordcount/input/file01
> > tail: File does not exist: /usr/divij/wordcount/input/file01
> > [divij@gobi bin]$
> >
> > i really dont know whats wrong. can anyone help?
> >
> >
> > On Mon, Jul 13, 2009 at 6:15 PM, Aaron Kimball <aaron@cloudera.com>
> wrote:
> >
> >> I'm not convinced anything is wrong with the TaskTracker. Can you run
> jobs?
> >> Does the pi example work? If not, what error does it give?
> >>
> >> If you're trying to configure your SecondaryNameNode on a different host
> >> than your NameNode, you'll need to do some configuration tweaking. I
> wrote
> >> a
> >> blog post with instructions on how to do this, here:
> >>
> >>
> http://www.cloudera.com/blog/2009/02/10/multi-host-secondarynamenode-configuration/
> >>
> >> Good luck!
> >> - Aaron
> >>
> >> On Mon, Jul 13, 2009 at 3:00 PM, Divij Durve <divij.tech@gmail.com>
> wrote:
> >>
> >> > here is the log file from the secondary namenode:
> >> >
> >> > 2009-07-13 17:36:13,274 INFO
> >> > org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: STARTUP_MSG:
> >> > /************************************************************
> >> > STARTUP_MSG: Starting SecondaryNameNode
> >> > STARTUP_MSG:   host = kalahari.mssm.edu/127.0.0.1
> >> > STARTUP_MSG:   args = []
> >> > STARTUP_MSG:   version = 0.19.0
> >> > STARTUP_MSG:   build =
> >> > https://svn.apache.org/repos/asf/hadoop/core/branches/branch-0.19 -r
> >> > 713890;
> >> > compiled by 'ndaley' on Fri Nov 14 03:12:29 UTC 2008
> >> > ************************************************************/
> >> > 2009-07-13 17:36:13,350 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> >> > Initializing JVM Metrics with processName=SecondaryNameNode,
> >> sessionId=null
> >> > 2009-07-13 17:36:13,445 INFO
> >> org.apache.hadoop.hdfs.server.common.Storage:
> >> > Recovering storage directory
> >> > /home/divij/hive/build/hadoopcore/hadoop-0.19.0/hadoo
> >> > p-tmp/dfs/namesecondary from failed checkpoint.
> >> > 2009-07-13 17:36:13,516 INFO org.mortbay.http.HttpServer: Version
> >> > Jetty/5.1.4
> >> > 2009-07-13 17:36:13,522 INFO org.mortbay.util.Credential: Checking
> >> Resource
> >> > aliases
> >> > 2009-07-13 17:36:13,729 INFO org.mortbay.util.Container: Started
> >> > org.mortbay.jetty.servlet.WebApplicationHandler@6602e323
> >> > 2009-07-13 17:36:13,772 INFO org.mortbay.util.Container: Started
> >> > WebApplicationContext[/static,/static]
> >> > 2009-07-13 17:36:13,855 INFO org.mortbay.util.Container: Started
> >> > org.mortbay.jetty.servlet.WebApplicationHandler@77546dbc
> >> > 2009-07-13 17:36:13,866 INFO org.mortbay.util.Container: Started
> >> > WebApplicationContext[/logs,/logs]
> >> > 2009-07-13 17:36:13,945 INFO
> org.mortbay.jetty.servlet.XMLConfiguration:
> >> No
> >> > WEB-INF/web.xml in
> >> >
> file:/home/divij/hive/build/hadoopcore/hadoop-0.19.0/webapps/secondary.
> >> > Serving files and default/dynamic servlets only
> >> > 2009-07-13 17:36:13,946 INFO org.mortbay.util.Container: Started
> >> > org.mortbay.jetty.servlet.WebApplicationHandler@52c00025
> >> > 2009-07-13 17:36:13,960 INFO org.mortbay.util.Container: Started
> >> > WebApplicationContext[/,/]
> >> > 2009-07-13 17:36:13,962 INFO org.mortbay.http.SocketListener: Started
> >> > SocketListener on 0.0.0.0:50090
> >> > 2009-07-13 17:36:13,962 INFO org.mortbay.util.Container: Started
> >> > org.mortbay.jetty.Server@2f74219d
> >> > 2009-07-13 17:36:13,962 INFO
> >> > org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Secondary
> >> > Web-server up at: 0.0.0.0:50090
> >> > 2009-07-13 17:36:13,962 WARN
> >> > org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Checkpoint
> >> > Period   :3600 secs (60 min)
> >> > 2009-07-13 17:36:13,963 WARN
> >> > org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Log Size
> >> > Trigger    :67108864 bytes (65536 KB)
> >> > 2009-07-13 17:41:14,018 ERROR
> >> > org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Exception in
> >> > doCheckpoint:
> >> > 2009-07-13 17:41:14,019 ERROR
> >> > org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode:
> >> > java.net.ConnectException: Connection refused
> >> >        at java.net.PlainSocketImpl.socketConnect(Native Method)
> >> >        at
> >> >
> >> >
> >>
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:310)
> >> >        at
> >> >
> >> >
> >>
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:174)
> >> >        at
> >> >
> >>
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:163)
> >> >        at java.net.Socket.connect(Socket.java:537)
> >> >        at java.net.Socket.connect(Socket.java:487)
> >> >        at sun.net.NetworkClient.doConnect(NetworkClient.java:174)
> >> >        at sun.net.www.http.HttpClient.openServer(HttpClient.java:409)
> >> >        at sun.net.www.http.HttpClient.openServer(HttpClient.java:530)
> >> >        at sun.net.www.http.HttpClient.<init>(HttpClient.java:240)
> >> >        at sun.net.www.http.HttpClient.New(HttpClient.java:321)
> >> >        at sun.net.www.http.HttpClient.New(HttpClient.java:338)
> >> > :
> >> >
> >> >
> >> > also there seems to be a problem with the task tracker. :
> >> > task tracker log file from a data node:
> >> >
> >> > 2009-07-13 17:28:00,174 INFO org.apache.hadoop.ipc.Server: IPC Server
> >> > Responder: starting
> >> > 2009-07-13 17:28:00,174 INFO org.apache.hadoop.ipc.Server: IPC Server
> >> > listener on 49833: starting
> >> > 2009-07-13 17:28:00,176 INFO org.apache.hadoop.ipc.Server: IPC Server
> >> > handler 1 on 49833: starting
> >> > 2009-07-13 17:28:00,176 INFO org.apache.hadoop.ipc.Server: IPC Server
> >> > handler 0 on 49833: starting
> >> > 2009-07-13 17:28:00,177 INFO org.apache.hadoop.ipc.Server: IPC Server
> >> > handler 2 on 49833: starting
> >> > 2009-07-13 17:28:00,177 INFO org.apache.hadoop.ipc.Server: IPC Server
> >> > handler 3 on 49833: starting
> >> > 2009-07-13 17:28:00,177 INFO org.apache.hadoop.mapred.TaskTracker:
> >> > TaskTracker up at: localhost/127.0.0.1:49833
> >> > 2009-07-13 17:28:00,178 INFO org.apache.hadoop.mapred.TaskTracker:
> >> Starting
> >> > tracker tracker_localhost:localhost/127.0.0.1:49833
> >> > 2009-07-13 17:28:00,312 INFO org.apache.hadoop.mapred.TaskTracker:
> >> Starting
> >> > thread: Map-events fetcher for all reduce tasks on
> >> > tracker_localhost:localhost/127.0.0.1:49833
> >> > 2009-07-13 17:28:00,314 INFO org.apache.hadoop.mapred.IndexCache:
> >> > IndexCache
> >> > created with max memory = 10485760
> >> > 2009-07-13 17:28:00,317 INFO org.apache.hadoop.mapred.TaskTracker:
> >> > TaskTracker's tasksMaxMem is not set. TaskMemoryManager is disabled.
> >> > (END)
> >> >
> >> >
> >> > can anyone tell me where i went wrong?
> >> >
> >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message