hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shuja Rehman <shujamug...@gmail.com>
Subject Re: 2011-06-10 13:14:29,767 FATAL org.apache.hadoop.mapred.JobTracker: java.net.BindException: Problem binding to server1/10.0.3.85:8021 : Address already in use
Date Sat, 11 Jun 2011 09:39:43 GMT
The history folder was deleted by some how and I have created this folder
and its internal folders manually and then restarted the job tracker and
then it works. I am not sure why it shows the "Address already in use"
exception on missing any folders.

On Sat, Jun 11, 2011 at 2:42 AM, Alex Kozlov <alexvk@cloudera.com> wrote:

> Can you be more explicit for the benefit of others?  How can a history
> folder result in the "Address already in use" exception?
>
> On Fri, Jun 10, 2011 at 2:33 PM, Shuja Rehman <shujamughal@gmail.com>wrote:
>
>> Thanks Alex,
>> the problem was due to history folder.
>>
>> On Sat, Jun 11, 2011 at 2:18 AM, Alex Kozlov <alexvk@cloudera.com> wrote:
>>
>>> 1. Check the /etc/hosts file that it translates the local machine name
>>> (server1) into local IP (or just contains loopback interface record
>>> "localhost").  What does ping server1 returns with?
>>> 2. Try `telnet server1 8022` as well as `telnet 10.0.3.85 8022`: does it
>>> connect?
>>>
>>> There is another unrelated error about /var/log/hadoop-0.20/history: make
>>> sure you have x permissions along the whole path.
>>>
>>>
>>> Alex K
>>>
>>>
>>> On Fri, Jun 10, 2011 at 1:44 PM, Shuja Rehman <shujamughal@gmail.com>wrote:
>>>
>>>> i have checked using netstat. the port is not used by any process. more
>>>> over i change   from 8021 to 8022 in mapred.site.xml. but the error still
>>>> remains.
>>>> the complete stack trace is as follow,
>>>> ************************************************************/
>>>> 2011-06-10 13:34:53,415 INFO org.apache.hadoop.mapred.JobTracker:
>>>> STARTUP_MSG:
>>>> /************************************************************
>>>> STARTUP_MSG: Starting JobTracker
>>>> STARTUP_MSG:   host = server1/10.0.3.85
>>>> STARTUP_MSG:   args = []
>>>> STARTUP_MSG:   version = 0.20.2-cdh3u0
>>>> STARTUP_MSG:   build =  -r 81256ad0f2e4ab2bd34b04f53d25a6c23686dd14;
>>>> compiled by 'root' on Fri Mar 25 20:07:24 EDT 2011
>>>> ************************************************************/
>>>> 2011-06-10 13:34:54,125 INFO
>>>> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
>>>> Updating the current master key for generating delegation tokens
>>>> 2011-06-10 13:34:54,127 INFO org.apache.hadoop.mapred.JobTracker:
>>>> Scheduler configured with (memSizeForMapSlotOnJT, memSizeForReduceSlotOnJT,
>>>> limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1, -1)
>>>> 2011-06-10 13:34:54,128 INFO org.apache.hadoop.util.HostsFileReader:
>>>> Refreshing hosts (include/exclude) list
>>>> 2011-06-10 13:34:54,159 INFO
>>>> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
>>>> Starting expired delegation token remover thread,
>>>> tokenRemoverScanInterval=60 min(s)
>>>> 2011-06-10 13:34:54,159 INFO
>>>> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
>>>> Updating the current master key for generating delegation tokens
>>>> 2011-06-10 13:34:54,204 INFO org.apache.hadoop.mapred.JobTracker:
>>>> Starting jobtracker with owner as mapred
>>>> 2011-06-10 13:34:54,233 INFO org.apache.hadoop.ipc.Server: Starting
>>>> Socket Reader #1 for port 8022
>>>> 2011-06-10 13:34:54,234 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
>>>> Initializing RPC Metrics with hostName=JobTracker, port=8022
>>>> 2011-06-10 13:34:54,236 INFO
>>>> org.apache.hadoop.ipc.metrics.RpcDetailedMetrics: Initializing RPC Metrics
>>>> with hostName=JobTracker, port=8022
>>>> 2011-06-10 13:34:54,312 INFO org.mortbay.log: Logging to
>>>> org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
>>>> org.mortbay.log.Slf4jLog
>>>> 2011-06-10 13:34:54,366 INFO org.apache.hadoop.http.HttpServer: Added
>>>> global filtersafety
>>>> (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
>>>> 2011-06-10 13:34:54,403 WARN org.apache.hadoop.util.NativeCodeLoader:
>>>> Unable to load native-hadoop library for your platform... using builtin-java
>>>> classes where applicable
>>>> 2011-06-10 13:34:54,418 WARN org.apache.hadoop.mapred.JobTracker: Error
>>>> starting tracker: org.apache.hadoop.util.Shell$ExitCodeException: chmod:
>>>> cannot access `/var/log/hadoop-0.20/history': No such file or directory
>>>>
>>>>         at org.apache.hadoop.util.Shell.runCommand(Shell.java:255)
>>>>         at org.apache.hadoop.util.Shell.run(Shell.java:182)
>>>>         at
>>>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:375)
>>>>         at org.apache.hadoop.util.Shell.execCommand(Shell.java:461)
>>>>         at org.apache.hadoop.util.Shell.execCommand(Shell.java:444)
>>>>         at
>>>> org.apache.hadoop.fs.RawLocalFileSystem.execCommand(RawLocalFileSystem.java:508)
>>>>         at
>>>> org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:499)
>>>>         at
>>>> org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:319)
>>>>         at
>>>> org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189)
>>>>         at org.apache.hadoop.mapred.JobHistory.init(JobHistory.java:335)
>>>>         at
>>>> org.apache.hadoop.mapred.JobTracker$2.run(JobTracker.java:2171)
>>>>         at
>>>> org.apache.hadoop.mapred.JobTracker$2.run(JobTracker.java:2168)
>>>>         at java.security.AccessController.doPrivileged(Native Method)
>>>>         at javax.security.auth.Subject.doAs(Subject.java:396)
>>>>         at
>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
>>>>         at
>>>> org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2168)
>>>>
>>>>         at
>>>> org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2043)
>>>>         at
>>>> org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:294)
>>>>         at
>>>> org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:286)
>>>>         at
>>>> org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4767)
>>>>
>>>> 2011-06-10 13:34:55,423 INFO
>>>> org.apache.hadoop.security.UserGroupInformation: JAAS Configuration already
>>>> set up for Hadoop, not re-installing.
>>>> 2011-06-10 13:34:55,423 INFO
>>>> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
>>>> Updating the current master key for generating delegation tokens
>>>> 2011-06-10 13:34:55,424 INFO org.apache.hadoop.mapred.JobTracker:
>>>> Scheduler configured with (memSizeForMapSlotOnJT, memSizeForReduceSlotOnJT,
>>>> limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1, -1)
>>>> 2011-06-10 13:34:55,424 INFO org.apache.hadoop.util.HostsFileReader:
>>>> Refreshing hosts (include/exclude) list
>>>> 2011-06-10 13:34:55,447 INFO org.apache.hadoop.mapred.JobTracker:
>>>> Starting jobtracker with owner as mapred
>>>> 2011-06-10 13:34:55,447 FATAL org.apache.hadoop.mapred.JobTracker:
>>>> java.net.BindException: Problem binding to server1/10.0.3.85:8022 :
>>>> Address already in use
>>>>
>>>>         at org.apache.hadoop.ipc.Server.bind(Server.java:230)
>>>>         at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:319)
>>>>         at org.apache.hadoop.ipc.Server.<init>(Server.java:1510)
>>>>         at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:539)
>>>>         at org.apache.hadoop.ipc.RPC.getServer(RPC.java:500)
>>>>         at
>>>> org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2136)
>>>>         at
>>>> org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2043)
>>>>         at
>>>> org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:294)
>>>>         at
>>>> org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:286)
>>>>         at
>>>> org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4767)
>>>> Caused by: java.net.BindException: Address already in use
>>>>         at sun.nio.ch.Net.bind(Native Method)
>>>>         at
>>>> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:119)
>>>>         at
>>>> sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
>>>>         at org.apache.hadoop.ipc.Server.bind(Server.java:228)
>>>>         ... 9 more
>>>>
>>>> 2011-06-10 13:34:55,448 INFO
>>>> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
>>>> Starting expired delegation token remover thread,
>>>> tokenRemoverScanInterval=60 min(s)
>>>> 2011-06-10 13:34:55,448 INFO
>>>> org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager:
>>>> Updating the current master key for generating delegation tokens
>>>> 2011-06-10 13:34:55,448 INFO org.apache.hadoop.mapred.JobTracker:
>>>> SHUTDOWN_MSG:
>>>> /************************************************************
>>>> SHUTDOWN_MSG: Shutting down JobTracker at server1/10.0.3.85
>>>> ************************************************************/
>>>>
>>>>
>>>>
>>>>
>>>> On Sat, Jun 11, 2011 at 1:33 AM, Harsh J <harsh@cloudera.com> wrote:
>>>>
>>>>> On face value, it appears to look like (1) A JT instance is already up
>>>>> and bound to that port (2) Something else is using 8021 and has bound
>>>>> it to itself.
>>>>>
>>>>> On Sat, Jun 11, 2011 at 1:57 AM, Shuja Rehman <shujamughal@gmail.com>
>>>>> wrote:
>>>>> > Hi All,
>>>>> >
>>>>> > i am not sure why i start getting this error. can  anybody let me
>>>>> know?
>>>>> >
>>>>> > 2011-06-10 13:14:29,767 FATAL org.apache.hadoop.mapred.JobTracker:
>>>>> > java.net.BindException: Problem binding to server1/10.0.3.85:8021
:
>>>>> Address
>>>>> > already in use
>>>>> >         at org.apache.hadoop.ipc.Server.bind(Server.java:230)
>>>>> >         at
>>>>> org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:319)
>>>>> >         at org.apache.hadoop.ipc.Server.<init>(Server.java:1510)
>>>>> >         at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:539)
>>>>> >         at org.apache.hadoop.ipc.RPC.getServer(RPC.java:500)
>>>>> >         at
>>>>> org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2136)
>>>>> >         at
>>>>> org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:2043)
>>>>> >         at
>>>>> > org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:294)
>>>>> >         at
>>>>> > org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:286)
>>>>> >         at
>>>>> org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4767)
>>>>> > Caused by: java.net.BindException: Address already in use
>>>>> >         at sun.nio.ch.Net.bind(Native Method)
>>>>> >         at
>>>>> >
>>>>> sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:119)
>>>>> >         at
>>>>> sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59)
>>>>> >         at org.apache.hadoop.ipc.Server.bind(Server.java:228)
>>>>> >         ... 9 more
>>>>> >
>>>>> >
>>>>> > Thanks
>>>>> >
>>>>> > --
>>>>> > Regards
>>>>> > Shuja-ur-Rehman Baig
>>>>> >
>>>>> >
>>>>> >
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Harsh J
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Regards
>>>> Shuja-ur-Rehman Baig
>>>> <http://pk.linkedin.com/in/shujamughal>
>>>>
>>>>
>>>
>>
>>
>> --
>> Regards
>> Shuja-ur-Rehman Baig
>> <http://pk.linkedin.com/in/shujamughal>
>>
>>
>


-- 
Regards
Shuja-ur-Rehman Baig
<http://pk.linkedin.com/in/shujamughal>

Mime
View raw message