我正在使用来自源代码的最新 hadoop 版本 3.0.0。我已经启动并运行了时间轴服务,并配置了 hadoop 以将其用于作业历史记录。但是,当我单击资源管理器 UI 中的历史记录时,出现以下错误:-
HTTP ERROR 404
Problem accessing /jobhistory/job/job_1444395439959_0001. Reason:
NOT_FOUND
有人可以指出我在这里缺少什么吗?以下是我的yarn-site.xml:-
<configuration>
<!-- Site specific YARN configuration properties -->
<property>
<description>The hostname of the Timeline service web application.</description>
<name>yarn.timeline-service.hostname</name>
<value>0.0.0.0</value>
</property>
<property>
<description>Address for the Timeline server to start the RPC server.</description>
<name>yarn.timeline-service.address</name>
<value>${yarn.timeline-service.hostname}:10200</value>
</property>
<property>
<description>The http address of the Timeline service web application.</description>
<name>yarn.timeline-service.webapp.address</name>
<value>${yarn.timeline-service.hostname}:8188</value>
</property>
<property>
<description>The https address of the Timeline service web application.</description>
<name>yarn.timeline-service.webapp.https.address</name>
<value>${yarn.timeline-service.hostname}:8190</value>
</property>
<property>
<description>Handler thread count to serve the client RPC requests.</description>
<name>yarn.timeline-service.handler-thread-count</name>
<value>10</value>
</property>
<property>
<description>Indicate to ResourceManager as well as clients whether
history-service is enabled or not. If enabled, ResourceManager starts
recording historical data that Timelien service can consume. Similarly,
clients can redirect to the history service when applications
finish if this is enabled.</description>
<name>yarn.timeline-service.generic-application-history.enabled</name>
<value>true</value>
</property>
<property>
<description>Store class name for history store, defaulting to file system
store</description>
<name>yarn.timeline-service.generic-application-history.store-class</name>
<value>org.apache.hadoop.yarn.server.applicationhistoryservice.FileSystemApplicationHistoryStore</value>
</property>
<property>
<description>URI pointing to the location of the FileSystem path where the history will be persisted.</description>
<name>yarn.timeline-service.generic-application-history.fs-history-store.uri</name>
<value>/tmp/yarn/system/history</value>
</property>
<property>
<description>T-file compression types used to compress history data.</description>
<name>yarn.timeline-service.generic-application-history.fs-history-store.compression-type</name>
<value>none</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
</configuration>
和我的mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>localhost:10200</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>localhost:8188</value>
</property>
<property>
<name>mapreduce.job.emit-timeline-data</name>
<value>true</value>
</property>
</configuration>
JPS 输出:
6022 NameNode
27976 NodeManager
27859 ResourceManager
6139 DataNode
6310 SecondaryNameNode
28482 ApplicationHistoryServer
29230 Jps
如果您想通过 YARN RM Web UI 查看日志,则需要启用日志聚合。为此,您需要在yarn-site.xml中设置以下参数:
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<property>
<name>yarn.nodemanager.remote-app-log-dir</name>
<value>/app-logs</value>
</property>
<property>
<name>yarn.nodemanager.remote-app-log-dir-suffix</name>
<value>logs</value>
</property>
如果不启用日志聚合,则网管会将日志存储在本地。通过上述设置,日志将聚合在 HDFS 中的“/app-logs/{username}/logs/”处。在此文件夹下,您可以找到迄今为止运行的所有应用程序的日志。同样,日志保留由配置参数“yarn.log-aggregation.retain-seconds”(保留聚合日志的时间)决定。
当 MapReduce 应用程序运行时,您可以从 YARN 的 Web UI 访问日志。应用程序完成后,日志将通过 Job History Server 提供。
另外,在yarn-site.xml中设置以下配置参数:
<property>
<name>yarn.log.server.url</name>
<value>http://{job-history-hostname}:8188/jobhistory/logs</value>
</property>
本文内容由网友自发贡献,版权归原作者所有,本站不承担相应法律责任。如您发现有涉嫌抄袭侵权的内容,请联系:hwhale#tublm.com(使用前将#替换为@)