kylin-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Li Yang <liy...@apache.org>
Subject Re: common.HadoopJobStatusChecker:58 : error check status
Date Wed, 19 Jul 2017 06:18:51 GMT
Given the exception actually happens in Hadoop code:
> java.lang.NullPointerException at
org.apache.hadoop.mapreduce.Job.getTrackingURL(Job.java:380)

And you had cube built successfully before. You might want to check recent
changes to your Hadoop env. It seems broken somewhere.

On Fri, Jul 14, 2017 at 11:07 AM, crossme <crossme@aliyun.com> wrote:

>
>
>
> >>>>>Perform third steps of logging information
>
>          Counters: 53
>  File System Counters
>   FILE: Number of bytes read=326082918
>   FILE: Number of bytes written=639475115
>   FILE: Number of read operations=0
>   FILE: Number of large read operations=0
>   FILE: Number of write operations=0
>   HDFS: Number of bytes read=375767996
>   HDFS: Number of bytes written=154906
>   HDFS: Number of read operations=48
>   HDFS: Number of large read operations=0
>   HDFS: Number of write operations=8
>  Job Counters
>   Failed reduce tasks=7
>   Killed reduce tasks=4
>   Launched map tasks=9
>   Launched reduce tasks=15
>   Data-local map tasks=7
>   Rack-local map tasks=2
>   Total time spent by all maps in occupied slots (ms)=554536
>   Total time spent by all reduces in occupied slots (ms)=1035019
>   Total time spent by all map tasks (ms)=554536
>   Total time spent by all reduce tasks (ms)=1035019
>   Total vcore-seconds taken by all map tasks=554536
>   Total vcore-seconds taken by all reduce tasks=1035019
>   Total megabyte-seconds taken by all map tasks=567844864
>   Total megabyte-seconds taken by all reduce tasks=1059859456
>  Map-Reduce Framework
>   Map input records=8758042
>   Map output records=70064417
>   Map output bytes=1547698142
>   Map output materialized bytes=310833429
>   Input split bytes=96597
>   Combine input records=70064417
>   Combine output records=42960200
>   Reduce input groups=289020
>   Reduce shuffle bytes=8450082
>   Reduce input records=1652047
>   Reduce output records=0
>   Spilled Records=87572447
>   Shuffled Maps =36
>   Failed Shuffles=0
>   Merged Map outputs=36
>   GC time elapsed (ms)=6372
>   CPU time spent (ms)=677610
>   Physical memory (bytes) snapshot=8539713536
>   Virtual memory (bytes) snapshot=36823269376
>   Total committed heap usage (bytes)=8535932928
>  Shuffle Errors
>   BAD_ID=0
>   CONNECTION=0
>   IO_ERROR=0
>   WRONG_LENGTH=0
>   WRONG_MAP=0
>   WRONG_REDUCE=0
>  File Input Format Counters
>   Bytes Read=0
>  File Output Format Counters
>   Bytes Written=0
>  org.apache.kylin.engine.mr.steps.FactDistinctColumnsMapper$RawDataCounter
>   BYTES=1721507003
>
>
> >>>>>The following is an error in the kylin.log file.
>   There is no error in the Hadoop log file
>
> 2017-07-14 10:40:10,427 INFO  [pool-9-thread-1] threadpool.
> DefaultScheduler:124 : Job Fetcher: 1 should running, 1
> actual running, 0 stopped, 0 ready, 10 already succeed, 8
> error, 6 discarded, 0 others
> 2017-07-14 10:40:12,442 DEBUG [Job 14691c4a-64d2-4b1d-ace5-
> d2d6ad9618d0-297] dao.ExecutableDao:217 : updating
> job output, id: 14691c4a-64d2-4b1d-ace5-d2d6ad9618d0-02
> 2017-07-14 10:40:22,450 DEBUG [Job 14691c4a-64d2-4b1d-ace5-
> d2d6ad9618d0-297] dao.ExecutableDao:217 : updating
> job output, id: 14691c4a-64d2-4b1d-ace5-d2d6ad9618d0-02
> 2017-07-14 10:40:32,457 DEBUG [Job 14691c4a-64d2-4b1d-ace5-
> d2d6ad9618d0-297] dao.ExecutableDao:217 : updating
> job output, id: 14691c4a-64d2-4b1d-ace5-d2d6ad9618d0-02
> 2017-07-14 10:40:42,467 DEBUG [Job 14691c4a-64d2-4b1d-ace5-
> d2d6ad9618d0-297] dao.ExecutableDao:217 : updating
> job output, id: 14691c4a-64d2-4b1d-ace5-d2d6ad9618d0-02
> 2017-07-14 10:40:52,478 DEBUG [Job 14691c4a-64d2-4b1d-ace5-
> d2d6ad9618d0-297] dao.ExecutableDao:217 : updating
> job output, id: 14691c4a-64d2-4b1d-ace5-d2d6ad9618d0-02
> 2017-07-14 10:41:02,493 DEBUG [Job 14691c4a-64d2-4b1d-ace5-
> d2d6ad9618d0-297] dao.ExecutableDao:217 : updating
> job output, id: 14691c4a-64d2-4b1d-ace5-d2d6ad9618d0-02
> 2017-07-14 10:41:10,430 INFO  [pool-9-thread-1] threadpool.
> DefaultScheduler:124 : Job Fetcher: 1 should running, 1
> actual running, 0 stopped, 0 ready, 10 already succeed, 8
> error, 6 discarded, 0 others
> 2017-07-14 10:41:12,518 DEBUG [Job 14691c4a-64d2-4b1d-ace5-
> d2d6ad9618d0-297] dao.ExecutableDao:217 : updating
> job output, id: 14691c4a-64d2-4b1d-ace5-d2d6ad9618d0-02
> 2017-07-14 10:41:22,527 DEBUG [Job 14691c4a-64d2-4b1d-ace5-
> d2d6ad9618d0-297] dao.ExecutableDao:217 : updating
> job output, id: 14691c4a-64d2-4b1d-ace5-d2d6ad9618d0-02
> 2017-07-14 10:41:32,535 DEBUG [Job 14691c4a-64d2-4b1d-ace5-
> d2d6ad9618d0-297] dao.ExecutableDao:217 : updating
> job output, id: 14691c4a-64d2-4b1d-ace5-d2d6ad9618d0-02
> 2017-07-14 10:41:42,544 DEBUG [Job 14691c4a-64d2-4b1d-ace5-
> d2d6ad9618d0-297] dao.ExecutableDao:217 : updating
> job output, id: 14691c4a-64d2-4b1d-ace5-d2d6ad9618d0-02
> 2017-07-14 10:41:53,548 INFO  [Job 14691c4a-64d2-4b1d-ace5-
> d2d6ad9618d0-297] ipc.Client:867 : Retrying connect to server: dn1/
> 10.50.229.209:51098. Already tried 0 time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=3, sleepTime=
> 1000 MILLISECONDS)
> 2017-07-14 10:41:54,549 INFO  [Job 14691c4a-64d2-4b1d-ace5-
> d2d6ad9618d0-297] ipc.Client:867 : Retrying connect to server: dn1/
> 10.50.229.209:51098. Already tried 1 time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=3, sleepTime=
> 1000 MILLISECONDS)
> 2017-07-14 10:41:55,549 INFO  [Job 14691c4a-64d2-4b1d-ace5-
> d2d6ad9618d0-297] ipc.Client:867 : Retrying connect to server: dn1/
> 10.50.229.209:51098. Already tried 2 time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=3, sleepTime=
> 1000 MILLISECONDS)
> 2017-07-14 10:41:55,663 INFO  [Job 14691c4a-64d2-4b1d-ace5-
> d2d6ad9618d0-297] mapred.ClientServiceDelegate:277 : Application state is
> completed. FinalApplicationStatus=FAILED. Redirecting to job history
> server
> 2017-07-14 10:41:55,686 ERROR [Job 14691c4a-64d2-4b1d-ace5-
> d2d6ad9618d0-297] common.HadoopJobStatusChecker:58 : error check status
> java.io.IOException: Job status not available
>  at org.apache.hadoop.mapreduce.Job.updateStatus(Job.java:334)
>  at org.apache.hadoop.mapreduce.Job.getStatus(Job.java:341)
>  at org.apache.kylin.engine.mr.common.HadoopJobStatusChecker.checkStatus(
> HadoopJobStatusChecker.java:38)
>  at org.apache.kylin.engine.mr.common.MapReduceExecutable.
> doWork(MapReduceExecutable.java:153)
>  at org.apache.kylin.job.execution.AbstractExecutable.
> execute(AbstractExecutable.java:124)
>  at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(
> DefaultChainedExecutable.java:64)
>  at org.apache.kylin.job.execution.AbstractExecutable.
> execute(AbstractExecutable.java:124)
>  at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(
> DefaultScheduler.java:142)
>  at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>  at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>  at java.lang.Thread.run(Thread.java:748)
> 2017-07-14 10:41:55,687 ERROR [Job 14691c4a-64d2-4b1d-ace5-
> d2d6ad9618d0-297] common.MapReduceExecutable:197 : error execute
> MapReduceExecutable{id=14691c4a-64d2-4b1d-ace5-
> d2d6ad9618d0-02, name=Extract Fact Table Distinct Columns, state=RUNNING}
> java.lang.NullPointerException
>  at org.apache.hadoop.mapreduce.Job.getTrackingURL(Job.java:380)
>  at org.apache.kylin.engine.mr.common.HadoopCmdOutput.
> getInfo(HadoopCmdOutput.java:61)
>  at org.apache.kylin.engine.mr.common.MapReduceExecutable.
> doWork(MapReduceExecutable.java:162)
>  at org.apache.kylin.job.execution.AbstractExecutable.
> execute(AbstractExecutable.java:124)
>  at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(
> DefaultChainedExecutable.java:64)
>  at org.apache.kylin.job.execution.AbstractExecutable.
> execute(AbstractExecutable.java:124)
>  at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(
> DefaultScheduler.java:142)
>  at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>  at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>  at java.lang.Thread.run(Thread.java:748)
> 2017-07-14 10:41:55,687 DEBUG [Job 14691c4a-64d2-4b1d-ace5-
> d2d6ad9618d0-297] dao.ExecutableDao:217 : updating
> job output, id: 14691c4a-64d2-4b1d-ace5-d2d6ad9618d0-02
> 2017-07-14 10:41:55,697 DEBUG [Job 14691c4a-64d2-4b1d-ace5-
> d2d6ad9618d0-297] dao.ExecutableDao:217 : updating
> job output, id: 14691c4a-64d2-4b1d-ace5-d2d6ad9618d0-02
> 2017-07-14 10:41:55,703 INFO  [Job 14691c4a-64d2-4b1d-ace5-
> d2d6ad9618d0-297] execution.ExecutableManager:389 : job
> id:14691c4a-64d2-4b1d-ace5-d2d6ad9618d0-02 from RUNNING to ERROR
> 2017-07-14 10:41:55,713 DEBUG [Job 14691c4a-64d2-4b1d-ace5-
> d2d6ad9618d0-297] dao.ExecutableDao:217 : updating
> job output, id: 14691c4a-64d2-4b1d-ace5-d2d6ad9618d0
> 2017-07-14 10:41:55,728 DEBUG [Job 14691c4a-64d2-4b1d-ace5-
> d2d6ad9618d0-297] dao.ExecutableDao:217 : updating
> job output, id: 14691c4a-64d2-4b1d-ace5-d2d6ad9618d0
> 2017-07-14 10:41:55,731 DEBUG [Job 14691c4a-64d2-4b1d-ace5-
> d2d6ad9618d0-297] dao.ExecutableDao:217 : updating
> job output, id: 14691c4a-64d2-4b1d-ace5-d2d6ad9618d0
> 2017-07-14 10:41:55,734 INFO  [Job 14691c4a-64d2-4b1d-ace5-
> d2d6ad9618d0-297] execution.ExecutableManager:389 : job
> id:14691c4a-64d2-4b1d-ace5-d2d6ad9618d0 from RUNNING to ERROR
> 2017-07-14 10:41:55,734 WARN  [Job 14691c4a-64d2-4b1d-ace5-
> d2d6ad9618d0-297] execution.AbstractExecutable:258 : no
> need to send email, user list is empty
> 2017-07-14 10:41:55,745 INFO  [pool-9-thread-1] threadpool.
> DefaultScheduler:124 : Job Fetcher: 0 should running, 0
> actual running, 0 stopped, 0 ready, 10 already succeed, 9
> error, 6 discarded, 0 others
> 2017-07-14 10:42:10,431 INFO  [pool-9-thread-1] threadpool.
> DefaultScheduler:124 : Job Fetcher: 0 should running, 0
> actual running, 0 stopped, 0 ready, 10 already succeed, 9
> error, 6 discarded, 0 others
> 2017-07-14 10:43:10,432 INFO  [pool-9-thread-1] threadpool.
> DefaultScheduler:124 : Job Fetcher: 0 should running, 0
> actual running, 0 stopped, 0 ready, 10 already succeed, 9
> error, 6 discarded, 0 others
> 2017-07-14 10:44:10,431 INFO  [pool-9-thread-1] threadpool.
> DefaultScheduler:124 : Job Fetcher: 0 should running, 0
> actual running, 0 stopped, 0 ready, 10 already succeed, 9
> error, 6 discarded, 0 others
>
>
> ------------------------------------------------------------------
> 发件人:crossme <crossme@aliyun.com>
> 发送时间:2017年7月13日(星期四) 19:51
> 收件人:user <user@kylin.apache.org>
> 主 题:common.HadoopJobStatusChecker:58 : error check status
>
> Hi All
>
>     The Cube build error on Step 3 Extract Fact Table Distinct Columns.
> Here is the error message. Any help please.
>
>     Explain: This created 4 Cube test, only one of which
> all processes run, Cube successfully constructed, can
> query, the rest of the Cube in the third step error, the
> error log is below, do not achieve the status of the Job.
>
> production environment:
>         CDH-5.9   Kylin-2.0
>
> 2017-07-13 14:16:39,835 INFO  [Job 3895f42c-8ee4-4eee-a0fc-
> 9b511f9c0be4-437] mapred.ClientServiceDelegate:277 : Application state is
> completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job
> history server
> 2017-07-13 14:16:39,849 ERROR [Job 3895f42c-8ee4-4eee-a0fc-
> 9b511f9c0be4-437] common.HadoopJobStatusChecker:58 : error check status
> java.io.IOException: Job status not available
>  at org.apache.hadoop.mapreduce.Job.updateStatus(Job.java:334)
>  at org.apache.hadoop.mapreduce.Job.getStatus(Job.java:341)
>  at org.apache.kylin.engine.mr.common.HadoopJobStatusChecker.checkStatus(
> HadoopJobStatusChecker.java:38)
>  at org.apache.kylin.engine.mr.common.MapReduceExecutable.
> doWork(MapReduceExecutable.java:153)
>  at org.apache.kylin.job.execution.AbstractExecutable.
> execute(AbstractExecutable.java:124)
>  at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(
> DefaultChainedExecutable.java:64)
>  at org.apache.kylin.job.execution.AbstractExecutable.
> execute(AbstractExecutable.java:124)
>  at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(
> DefaultScheduler.java:142)
>  at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>  at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>  at java.lang.Thread.run(Thread.java:748)
> 2017-07-13 14:16:39,850 ERROR [Job 3895f42c-8ee4-4eee-a0fc-
> 9b511f9c0be4-437] common.MapReduceExecutable:197 : error execute
> MapReduceExecutable{id=3895f42c-8ee4-4eee-a0fc-
> 9b511f9c0be4-07, name=Convert Cuboid Data to HFile, state=RUNNING}
> java.lang.NullPointerException
>  at org.apache.hadoop.mapreduce.Job.getTrackingURL(Job.java:380)
>  at org.apache.kylin.engine.mr.common.HadoopCmdOutput.
> getInfo(HadoopCmdOutput.java:61)
>  at org.apache.kylin.engine.mr.common.MapReduceExecutable.
> doWork(MapReduceExecutable.java:162)
>  at org.apache.kylin.job.execution.AbstractExecutable.
> execute(AbstractExecutable.java:124)
>  at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(
> DefaultChainedExecutable.java:64)
>  at org.apache.kylin.job.execution.AbstractExecutable.
> execute(AbstractExecutable.java:124)
>  at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(
> DefaultScheduler.java:142)
>  at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
>  at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
>  at java.lang.Thread.run(Thread.java:748)
>
>
> yarn-site.xml
>
> <property>
>          <name>yarn.resourcemanager.webapp.address</name>
>          <value>dn1:8088</value>
> </property>
>
>
>

Mime
View raw message