linux rm 提示io异常,Hadoop异常 java.io.IOException: Job status not available

Hadoop集群上跑mapreduce，在job任务执行完成退出时报 java.io.IOException: Job status not available异常。Job client请求job状态时，Application已经完成转而去Job history server请求job状态，就在这里抛出异常。

[linuxidc@master conf]$ hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar wordcount /user/lizeyi/people.txt  /user/lizeyi/wordcount7

15/06/08 18:36:16 INFO client.RMProxy: Connecting to ResourceManager at master.hadoop/10.3.4.35:8032

15/06/08 18:36:17 INFO input.FileInputFormat: Total input paths to process : 1

15/06/08 18:36:17 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library

15/06/08 18:36:17 INFO lzo.LzoCodec: Successfully loaded & initialized native-lzo library [hadoop-lzo rev 39cf0c71a251a79c50555810ca660450d9682140]

15/06/08 18:36:17 INFO mapreduce.JobSubmitter: number of splits:1

15/06/08 18:36:18 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1433756996622_0004

15/06/08 18:36:18 INFO impl.YarnClientImpl: Submitted application application_1433756996622_0004

15/06/08 18:36:18 INFO mapreduce.Job: The url to track the job: http://master.hadoop:8088/proxy/application_1433756996622_0004/

15/06/08 18:36:18 INFO mapreduce.Job: Running job: job_1433756996622_0004

15/06/08 18:36:40 INFO mapred.ClientServiceDelegate: Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server

java.io.IOException: Job status not available

at org.apache.hadoop.mapreduce.Job.updateStatus(Job.java:322)

at org.apache.hadoop.mapreduce.Job.isComplete(Job.java:609)

at org.apache.hadoop.mapreduce.Job.monitorAndPrintJob(Job.java:1354)

at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1316)

at org.apache.hadoop.examples.WordCount.main(WordCount.java:87)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)

at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:145)

at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at org.apache.hadoop.util.RunJar.main(RunJar.java:212)

须配置Job History Server相关参数，让Job Client可以读取job最后的执行状态，测试Hadoop版本2.5.0

添加参数vim mapred-site.xml

mapreduce.jobhistory.address

master.hadoop:10020

yarn.app.mapreduce.am.staging-dir

/tmp/hadoop-yarn/staging

mapreduce.jobhistory.intermediate-done-dir

${yarn.app.mapreduce.am.staging-dir}/history/done_intermediate

mapreduce.jobhistory.done-dir

${yarn.app.mapreduce.am.staging-dir}/history/done

修改完成配置后，重新执行任务后正常退出

[linuxidc@master conf]$ hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar wordcount /user/lizeyi/people.txt  /user/lizeyi/wordcount9

15/06/08 18:54:04 INFO client.RMProxy: Connecting to ResourceManager at master.hadoop/10.3.4.35:8032

15/06/08 18:54:06 INFO input.FileInputFormat: Total input paths to process : 1

15/06/08 18:54:06 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library

15/06/08 18:54:06 INFO lzo.LzoCodec: Successfully loaded & initialized native-lzo library [hadoop-lzo rev 39cf0c71a251a79c50555810ca660450d9682140]

15/06/08 18:54:06 INFO mapreduce.JobSubmitter: number of splits:1

15/06/08 18:54:06 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1433760669916_0001

15/06/08 18:54:07 INFO impl.YarnClientImpl: Submitted application application_1433760669916_0001

15/06/08 18:54:07 INFO mapreduce.Job: The url to track the job: http://master.hadoop:8088/proxy/application_1433760669916_0001/

15/06/08 18:54:07 INFO mapreduce.Job: Running job: job_1433760669916_0001

15/06/08 18:54:34 INFO mapred.ClientServiceDelegate: Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server

15/06/08 18:54:35 INFO mapreduce.Job: Job job_1433760669916_0001 running in uber mode : false

15/06/08 18:54:35 INFO mapreduce.Job:  map 100% reduce 100%

15/06/08 18:54:35 INFO mapreduce.Job: Job job_1433760669916_0001 completed successfully

15/06/08 18:54:35 INFO mapreduce.Job: Counters: 49

File System Counters

FILE: Number of bytes read=68

FILE: Number of bytes written=205241

FILE: Number of read operations=0

FILE: Number of large read operations=0

FILE: Number of write operations=0

HDFS: Number of bytes read=151

HDFS: Number of bytes written=46

HDFS: Number of read operations=6

HDFS: Number of large read operations=0

HDFS: Number of write operations=2

Job Counters

Launched map tasks=1

Launched reduce tasks=1

Data-local map tasks=1

Total time spent by all maps in occupied slots (ms)=6036

Total time spent by all reduces in occupied slots (ms)=6132

Total time spent by all map tasks (ms)=6036

Total time spent by all reduce tasks (ms)=6132

Total vcore-seconds taken by all map tasks=6036

Total vcore-seconds taken by all reduce tasks=6132

Total megabyte-seconds taken by all map tasks=6180864

Total megabyte-seconds taken by all reduce tasks=6279168

Map-Reduce Framework

Map input records=4

Map output records=4

Map output bytes=54

Map output materialized bytes=68

Input split bytes=113

Combine input records=4

Combine output records=4

Reduce input groups=4

Reduce shuffle bytes=68

Reduce input records=4

Reduce output records=4

Spilled Records=8

Shuffled Maps =1

Failed Shuffles=0

Merged Map outputs=1

GC time elapsed (ms)=82

CPU time spent (ms)=2150

Physical memory (bytes) snapshot=447897600

Virtual memory (bytes) snapshot=1986359296

Total committed heap usage (bytes)=355467264

Shuffle Errors

BAD_ID=0

CONNECTION=0

IO_ERROR=0

WRONG_LENGTH=0

WRONG_MAP=0

WRONG_REDUCE=0

File Input Format Counters

Bytes Read=38

File Output Format Counters

Bytes Written=46