Hadoop3.0Yarn添加网络、磁盘IO等资源资料汇总及实战配置遇到的问题和解决办法-Hadoop-@太阳城手机版资讯
你好,游客 登录
背景:
阅读新闻

Hadoop3.0Yarn添加网络、磁盘IO等资源资料汇总及实战配置遇到的问题和解决办法

[日期:2018-01-25] 来源:公众账号  作者: [字体: ]

太阳城娱乐3.0扩展Yarn资源,官网给了一些英文资料,这里about云翻译为中文资料。这些资料,还是比较多的,但是在初次阅读的时候,会感到摸不着头脑。但是书读百遍其义自见。下面对这些资料进行汇总。

Hadoop3.0: YARN Resource自定义资源配置说明

http://www.aboutyun.com/forum.php?mod=viewthread&tid=23626

1.yarn默认情况下使用哪些资源?

2.Yarn如何实现扩展自定义资源?

4.哪些配置可以在yarn-site.xml文件或则 resource-types.xml文件配置?

5.yarn有哪三种Containers?

6.三种Container如何配置请求的CPU,memory等值?

太阳城娱乐3.0 Yarn支持网络1:network设计文档说明【中文】

http://www.aboutyun.com/forum.php?mod=viewthread&tid=23653

1.网络作为Yarn的资源,有什么好处?

2.Yarn是否只支持调度和强制执行“传出流量”?

3.Yarn是否支持入口流量?

太阳城娱乐3.0 Yarn支持网络资源2:network设计文档说明【中文】

http://www.aboutyun.com/forum.php?mod=viewthread&tid=23661

1.Resource profiles解决了什么问题?

2.使用profile的好处是什么?

太阳城娱乐3.0扩展Yarn资源模型详解1

http://www.aboutyun.com/forum.php?mod=viewthread&tid=23794

1.countable资源是指哪些?

2.noncountable资源,本文列举了什么资源?

3.标签是否为资源?

4.如何实现扩展YARN资源模型?

 

太阳城娱乐3.0扩展Yarn资源模型详解2:资源Profiles说明

http://www.aboutyun.com/forum.php?mod=viewthread&tid=23813

1.Resource profiles解决了什么问题?

2.使用profile的好处是什么?

Hadoop3 YARN集群中的磁盘I / O调度设计详解1【IO作为资源分配】

http://www.aboutyun.com/forum.php?mod=viewthread&tid=23819

1.磁盘IO实现共享,使用的是什么原理技术?

2.如何描述磁盘I / O资源? 

4.每个NodeManager如何执行每个本地任务的磁盘I / O资源使用?

上面资料分别介绍了如何添加除了内存和vcores资源,并且对所支持的资源进行调度配置。根据以上资料,大家如果读懂,完全可以配置,如想节省时间。后面about云会出这方面的视频。

下面是在实际配置过程中遇到的问题:

提交job之后,程序不运行:

[Bash shell] 纯文本查看 复制代码

?

太阳城娱乐 jar /usr/太阳城娱乐-3.0.0/share/太阳城娱乐/mapreduce/太阳城娱乐-mapreduce-examples-3.0.0.jar wordcount /data/wordcount /output/wordcount
2018-01-12 15:51:13,841 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.1.10:8032
2018-01-12 15:51:15,296 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/太阳城娱乐-yarn/staging/aboutyun/.staging/job_1515742430977_0006
2018-01-12 15:51:15,812 INFO input.FileInputFormat: Total input files to process : 1
2018-01-12 15:51:16,131 INFO mapreduce.JobSubmitter: number of splits:1
2018-01-12 15:51:16,205 INFO Configuration.deprecation: yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-publisher.enabled
2018-01-12 15:51:16,636 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1515742430977_0006
2018-01-12 15:51:16,638 INFO mapreduce.JobSubmitter: Executing with tokens: []
2018-01-12 15:51:17,088 INFO conf.Configuration: found resource resource-types.xml at file:/usr/太阳城娱乐-3.0.0/etc/太阳城娱乐/resource-types.xml
2018-01-12 15:51:17,148 INFO resource.ResourceUtils: Adding resource type - name = network, units = k, type = COUNTABLE
2018-01-12 15:51:17,311 INFO impl.YarnClientImpl: Submitted application application_1515742430977_0006
2018-01-12 15:51:17,459 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1515742430977_0006/
2018-01-12 15:51:17,460 INFO mapreduce.Job: Running job: job_1515742430977_0006

日志也没有错误,然后通过tail -f在停止集群和开启集群中,查看所有日志。原来是nodemanger的配置文件出现问题。

[Bash shell] 纯文本查看 复制代码

?

java.lang.RuntimeException: com.ctc.wstx.exc.WstxParsingException: Unexpected close tag </name>; expected </resource>.
at [row,col {unknown-source}]: [3,68]
        at org.apache.太阳城娱乐.conf.Configuration.loadResource(Configuration.java:3048)
        at org.apache.太阳城娱乐.conf.Configuration.loadResources(Configuration.java:2817)
        at org.apache.太阳城娱乐.conf.Configuration.getProps(Configuration.java:2689)
        at org.apache.太阳城娱乐.conf.Configuration.iterator(Configuration.java:2737)
        at org.apache.太阳城娱乐.yarn.util.resource.ResourceUtils.initializeNodeResourceInformation(ResourceUtils.java:497)
        at org.apache.太阳城娱乐.yarn.util.resource.ResourceUtils.getNodeResourceInformation(ResourceUtils.java:476)
        at org.apache.太阳城娱乐.yarn.server.nodemanager.util.NodeManagerHardwareUtils.getNodeResources(NodeManagerHardwareUtils.java:352)
        at org.apache.太阳城娱乐.yarn.server.nodemanager.NodeStatusUpdaterImpl.serviceInit(NodeStatusUpdaterImpl.java:177)
        at org.apache.太阳城娱乐.service.AbstractService.init(AbstractService.java:164)
        at org.apache.太阳城娱乐.service.CompositeService.serviceInit(CompositeService.java:108)
        at org.apache.太阳城娱乐.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:440)
        at org.apache.太阳城娱乐.service.AbstractService.init(AbstractService.java:164)
        at org.apache.太阳城娱乐.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:833)
        at org.apache.太阳城娱乐.yarn.server.nodemanager.NodeManager.main(NodeManager.java:894)
Caused by: com.ctc.wstx.exc.WstxParsingException: Unexpected close tag </name>; expected </resource>.
at [row,col {unknown-source}]: [3,68]
        at com.ctc.wstx.sr.StreamScanner.constructWfcException(StreamScanner.java:621)
        at com.ctc.wstx.sr.StreamScanner.throwParseError(StreamScanner.java:491)
        at com.ctc.wstx.sr.StreamScanner.throwParseError(StreamScanner.java:475)
        at com.ctc.wstx.sr.BasicStreamReader.reportWrongEndElem(BasicStreamReader.java:3365)
        at com.ctc.wstx.sr.BasicStreamReader.readEndElem(BasicStreamReader.java:3292)
        at com.ctc.wstx.sr.BasicStreamReader.nextFromTree(BasicStreamReader.java:2911)
        at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1123)
        at org.apache.太阳城娱乐.conf.Configuration.loadResource(Configuration.java:2883)
        ... 13 more
2018-01-12 15:57:07,441 INFO org.apache.太阳城娱乐.yarn.server.nodemanager.NodeManager: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NodeManager at master/192.168.1.10
************************************************************/
^C
[aboutyun@master logs]$ tail -f 太阳城娱乐-aboutyun-nodemanager-master.log
        at com.ctc.wstx.sr.BasicStreamReader.reportWrongEndElem(BasicStreamReader.java:3365)
        at com.ctc.wstx.sr.BasicStreamReader.readEndElem(BasicStreamReader.java:3292)
        at com.ctc.wstx.sr.BasicStreamReader.nextFromTree(BasicStreamReader.java:2911)
        at com.ctc.wstx.sr.BasicStreamReader.next(BasicStreamReader.java:1123)
        at org.apache.太阳城娱乐.conf.Configuration.loadResource(Configuration.java:2883)
        ... 13 more
2018-01-12 15:57:07,441 INFO org.apache.太阳城娱乐.yarn.server.nodemanager.NodeManager: SHUTDOWN_MSG:

去掉里面的错误标记问题解决。

yarn-site.xml文件

[Bash shell] 纯文本查看 复制代码

?

<configuration>
 
<!-- Site specific YARN configuration properties -->
<property>
        <name>yarn.resourcemanager.hostname</name>
        <value>master</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.nodemanager.env-whitelist</name>
   <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_HOME,PATH,LANG,TZ</value>
    </property>
    <property>
        <name>yarn.nodemanager.vmem-check-enable</name>
        <value>false</value>
    </property>
    <property>
        <name>yarn.scheduler.enablenetworkscheduling</name>
        <value>enable</value>
    </property>
<property>
        <name>yarn.scheduler.minimumallocationoutboundnetworkbandwidthmbit</name>
        <value>1mbit/sec</value>
    </property>
<property>
        <name>yarn.scheduler.maximumallocationoutboundnetworkbandwidthmbit</name>
        <value>1000mbit/sec</value>
    </property>
<property>
        <name>yarn.nodemanager.resource.network.interface</name>
        <value>eno16777736</value>
    </property>
<property>
        <name>yarn.nodemanager.resource.network.outboundbandwidthmbit</name>
        <value>1000mbit/sec</value>
    </property>
<property>
        <name>yarn.nodemanager.resource.network.outboundbandwidthyarnmbit</name>
        <value>1000mbit/sec</value>
    </property>
<property>
        <name>yarn.nodemanager.strictresourceusag</name>
        <value>elastic</value>
    </property>
 
 
</configuration>

下面说是network是未知的资源network:

[Bash shell] 纯文本查看 复制代码

?

2018-01-19 17:16:36,843 ERROR org.apache.太阳城娱乐.yarn.server.nodemanager.NodeManager: Error starting NodeManager
org.apache.太阳城娱乐.yarn.exceptions.ResourceNotFoundException: Unknown resource 'network'. Known resources are [name: memory-mb, units: Mi, type: COUNTABLE, value: 0, minimum allocation: 0, maximum allocation: 9223372036854775807, name: vcores, units: , type: COUNTABLE, value: 0, minimum allocation: 0, maximum allocation: 9223372036854775807]
        at org.apache.太阳城娱乐.yarn.api.records.Resource.getResourceInformation(Resource.java:251)
        at org.apache.太阳城娱乐.yarn.api.records.Resource.setResourceInformation(Resource.java:311)
        at org.apache.太阳城娱乐.yarn.server.nodemanager.util.NodeManagerHardwareUtils.getNodeResources(NodeManagerHardwareUtils.java:355)
        at org.apache.太阳城娱乐.yarn.server.nodemanager.NodeStatusUpdaterImpl.serviceInit(NodeStatusUpdaterImpl.java:177)
        at org.apache.太阳城娱乐.service.AbstractService.init(AbstractService.java:164)
        at org.apache.太阳城娱乐.service.CompositeService.serviceInit(CompositeService.java:108)
        at org.apache.太阳城娱乐.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:440)
        at org.apache.太阳城娱乐.service.AbstractService.init(AbstractService.java:164)
        at org.apache.太阳城娱乐.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:833)
        at org.apache.太阳城娱乐.yarn.server.nodemanager.NodeManager.main(NodeManager.java:894)
2018-01-19 17:16:36,934 INFO org.apache.太阳城娱乐.yarn.server.nodemanager.NodeManager: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NodeManager at master/192.168.1.10

上面是因为没有弄懂如何添加网络资源,需要区分开与memory和vcore的区别。需仔细阅读

Hadoop3.0: YARN Resource自定义资源配置说明

http://www.aboutyun.com/forum.php?mod=viewthread&tid=23626 并且区分

太阳城娱乐3.0 Yarn支持网络1:network设计文档说明【中文】

这个文档的关系。

[Bash shell] 纯文本查看 复制代码

?

2018-01-20 10:04:54,240 INFO org.apache.太阳城娱乐.service.AbstractService: Service NodeManager failed in state INITED; cause: org.apache.太阳城娱乐.yarn.exceptions.YarnRuntimeException: Attempt to re-define mandatory resource 'memory'.
org.apache.太阳城娱乐.yarn.exceptions.YarnRuntimeException: Attempt to re-define mandatory resource 'memory'.
        at org.apache.太阳城娱乐.yarn.util.resource.ResourceUtils.checkMandatoryResources(ResourceUtils.java:94)
        at org.apache.太阳城娱乐.yarn.util.resource.ResourceUtils.initializeResourcesMap(ResourceUtils.java:276)
        at org.apache.太阳城娱乐.yarn.util.resource.ResourceUtils.initializeResourceTypesIfNeeded(ResourceUtils.java:382)
        at org.apache.太阳城娱乐.yarn.util.resource.ResourceUtils.initializeResourceTypesIfNeeded(ResourceUtils.java:366)
        at org.apache.太阳城娱乐.yarn.util.resource.ResourceUtils.getNumberOfKnownResourceTypes(ResourceUtils.java:354)
        at org.apache.太阳城娱乐.yarn.api.records.impl.LightWeightResource.<init>(LightWeightResource.java:70)
        at org.apache.太阳城娱乐.yarn.api.records.Resource.newInstance(Resource.java:79)
        at org.apache.太阳城娱乐.yarn.server.nodemanager.util.NodeManagerHardwareUtils.getNodeResources(NodeManagerHardwareUtils.java:350)
        at org.apache.太阳城娱乐.yarn.server.nodemanager.NodeStatusUpdaterImpl.serviceInit(NodeStatusUpdaterImpl.java:177)
        at org.apache.太阳城娱乐.service.AbstractService.init(AbstractService.java:164)
        at org.apache.太阳城娱乐.service.CompositeService.serviceInit(CompositeService.java:108)
        at org.apache.太阳城娱乐.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:440)
        at org.apache.太阳城娱乐.service.AbstractService.init(AbstractService.java:164)
        at org.apache.太阳城娱乐.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:833)
        at org.apache.太阳城娱乐.yarn.server.nodemanager.NodeManager.main(NodeManager.java:894)

出现上面问题,同样也是因为没有弄懂如何添加资源及调度的关系。





收藏 推荐 打印 | 录入:Cstor | 阅读:
本文评论   查看全部评论 (0)
表情: 表情 姓名: 字数
点评:
       
评论声明
  • 尊重网上道德,遵守中华人民共和国的各项有关法律法规
  • 承担一切因您的行为而直接或间接导致的民事或刑事法律责任
  • 本站管理人员有权保留或删除其管辖留言中的任意内容
  • 本站有权在网站内转载或引用您的评论
  • 参与本评论即表明您已经阅读并接受上述条款