hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Joseph Evans (Updated) (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MAPREDUCE-3531) Sometimes java.lang.IllegalArgumentException: Invalid key to HMAC computation in NODE_UPDATE also causing RM to stop scheduling
Date Wed, 14 Dec 2011 18:11:31 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-3531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Robert Joseph Evans updated MAPREDUCE-3531:
-------------------------------------------

    Attachment: MR-3531.txt

Adding in a patch to fix the issue (Just changed HashMap to ConcurrentHashMap).  

I also added in a test, but I don't really like the test that much.  It tries to brute force
reproduce the problem.  On my box it takes about 1.8 sec to run so it should not be too big
of an issue right now.

However, when trying to run without the fix I have seen it reproduce the problem, but I have
also seen it essentially get into a living deadlock where all of the cores are pegged at 100%
utilization, and sometimes only a kill -9 stops it form running.  I am fine with removing
the test, because I don't think it adds much value, but I am also fine with leaving it in.
                
> Sometimes java.lang.IllegalArgumentException: Invalid key to HMAC computation in NODE_UPDATE
also causing RM to stop scheduling 
> --------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-3531
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3531
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2, resourcemanager, scheduler
>    Affects Versions: 0.23.1
>            Reporter: Karam Singh
>            Assignee: Robert Joseph Evans
>            Priority: Blocker
>         Attachments: MR-3531.txt
>
>
> Filling this Jira a bit late
> Started 350 cluster
> sbummited large sleep job.
> Foud that job was not running as RM has not allocated resouces to it.
> {code}
> 2011-12-01 11:56:25,200 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
nodeUpdate: <NMHost>:48490 clusterResources: memory: 3225600
> 2011-12-01 11:56:25,202 ERROR org.apache.hadoop.yarn.server.resourcemanager.ResourceManager:
Error in handling event
> type NODE_UPDATE to the scheduler
> java.lang.IllegalArgumentException: Invalid key to HMAC computation
>         at org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:141)
>         at org.apache.hadoop.yarn.server.security.ContainerTokenSecretManager.createPassword(ContainerTokenSecretManager.java:61)
>         atorg.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.createContainer(LeafQueue.java:1108)
>         at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.getContainer(LeafQueue.java:1091)
>         at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainer(LeafQueue.java:1137)
>         at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignNodeLocalContainers(LeafQueue.java:1001)
>         at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainersOnNode(LeafQueue.java:973)
>         at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.assignContainers(LeafQueue.java:760)
>         at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainersToChildQueues(ParentQueue.java:583)
>         at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.assignContainers(ParentQueue.java:513)
>         at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:569)
>         at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:611)
>         at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:77)
>         at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:294)
>         at java.lang.Thread.run(Thread.java:619)
> Caused by: java.security.InvalidKeyException: Secret key expected
>         at com.sun.crypto.provider.HmacCore.a(DashoA13*..)
>         at com.sun.crypto.provider.HmacSHA1.engineInit(DashoA13*..)
>         at javax.crypto.Mac.init(DashoA13*..)
>         at org.apache.hadoop.security.token.SecretManager.createPassword(SecretManager.java:139)
>         ... 14 more
> {code}
> As this stack is from 30 Nov checkou line number may be different

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message