零基础体验hadoop大数据(6)
时间:2016-10-13 09:16 来源:潇湘夜雨 作者:华嵩阳 点击:次
16/10/13 01:52:32 INFO mapreduce.Job: map 0% reduce 0% 16/10/13 01:52:40 INFO mapreduce.Job: map 100% reduce 0% 16/10/13 01:52:47 INFO mapreduce.Job: map 100% reduce 100% 16/10/13 01:52:47 INFO mapreduce.Job: Job job_1476294690985_0002 completed successfully 16/10/13 01:52:47 INFO mapreduce.Job: Counters: 49 File System Counters FILE: Number of bytes read=55 FILE: Number of bytes written=237633 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=155 HDFS: Number of bytes written=33 HDFS: Number of read operations=6 HDFS: Number of large read operations=0 HDFS: Number of write operations=2 Job Counters Launched map tasks=1 Launched reduce tasks=1 Data-local map tasks=1 Total time spent by all maps in occupied slots (ms)=5422 Total time spent by all reduces in occupied slots (ms)=4856 Total time spent by all map tasks (ms)=5422 Total time spent by all reduce tasks (ms)=4856 Total vcore-milliseconds taken by all map tasks=5422 Total vcore-milliseconds taken by all reduce tasks=4856 Total megabyte-milliseconds taken by all map tasks=5552128 Total megabyte-milliseconds taken by all reduce tasks=4972544 Map-Reduce Framework Map input records=2 Map output records=6 Map output bytes=64 Map output materialized bytes=55 Input split bytes=115 Combine input records=6 Combine output records=4 Reduce input groups=4 Reduce shuffle bytes=55 Reduce input records=4 Reduce output records=4 Spilled Records=8 Shuffled Maps =1 Failed Shuffles=0 Merged Map outputs=1 GC time elapsed (ms)=219 CPU time spent (ms)=1360 Physical memory (bytes) snapshot=310759424 Virtual memory (bytes) snapshot=1690013696 Total committed heap usage (bytes)=152702976 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=40 File Output Format Counters Bytes Written=33 查看计算后的结果： [root@hadoop-master ~]# hadoop fs -cat wordout/part-r-00000 chendu 1 hellow 2 to 1 welcome 2 至此hadoop的安装测试过程已完成，本文只是一个简单的hadoop集群，在实际生产环境中，hadoop还有许多相关的组件。 而且hadoop的使用主要在于MapReduce程序的编写，这些程序一般是由java语言实现。 14、Web访问，要先开放端口或者直接关闭防火墙 (1)输入命令，service stop iptables (2)浏览器打开http://172.18.109.235:8088/ 该接口主要用于监测各节点的性能参数和job任务的完成情况  (3)浏览器打开http://172.18.109.235:50070/ 该接口主要用于监测hdfs的运行情况  15、常见错误 错误： ERROR org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Unexpec ted error starting NodeStatusUpdater org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Recieved SHUTDOWN signal from Resourcemanager ,Registration of NodeManager failed, Message from ResourceManager: NodeManager from hadoop1 doesn't s atisfy minimum allocations, Sending SHUTDOWN signal to the NodeManager 查看日志文件发现报了这些错误，仔细看了下有e3.hadoop doesn’t satisfy minimum allocations, 这条， 我的yarn-site.xml 里面设置yarn.nodemanager.resource.memory-mb 才给了800 默认好像是1024，把这个值加到到1024以上就可以启动了 (责任编辑：liangzh) |