`
lookqlp
  • 浏览: 342123 次
  • 性别: Icon_minigender_1
  • 来自: 上海
社区版块
存档分类
最新评论

如何提示mapreduce,查看systemout信息

阅读更多
又折腾了大半天,只解决了一半的问题吧。
已经解决部分:
可以通过web查看运行job的systemout及其他信息
访问地址http://ip:50030,找到相应job往下查即可。
最后有
stdout logs hello world!
信息。即为systemout输出信息。


通过查询hadoop官方文档,集群搭建部分可以看到logging部分,有关hadoop的日志配置信息。


未解决部分:在map或者reduce函数上写io将一些必要信息进行落地成文本,可执行成功,但文件未能参见也未能写入信息,io具体写法如下:
File dir = new File("/root/bin/qiulp");
			if (!dir.exists()) {
				dir.mkdir();
			}
			File file = new File("/root/bin/qiulp/log.txt");
			BufferedWriter writer = new BufferedWriter(new FileWriter(file));
			writer.write(valuses);
			writer.close();

在windows eclipse远程连linux hadoop运行环境下没能写入文件。经过其他人其他测试环境下成功。不知道本环境为何不成功(本环境所有东西都装在root用户下,包括java环境)。求高手帮我看看。



终于解决了。io写的目录与运行的mapreduce有关,在哪个node上运行的,输出将会在哪个目录。所以没有在name节点上

同样了job日志也此。若配置一下hadoop.job.history.location 或者hadoop.log.dir可否将日志调整到name节点上,还没测试过
分享到:
评论

相关推荐

    Pro Apache Hadoop(Apress,2ed,2014)

    All the old content has been revised too, giving the latest on the ins and outs of MapReduce, cluster design, the Hadoop Distributed File System, and more. This book covers everything you need to ...

    hadoop_the_definitive_guide_3nd_edition.pdf

    Scaling Out Data Flow Combiner Functions Running a Distributed MapReduce Job Hadoop Streaming Ruby Python import import import import import import org.apache.hadoop.fs.Path; org.apache.hadoop.io....

    hadoop-2.2.0-src.tar

    YARN - A general purpose resource management system for Hadoop to allow MapReduce and other other data processing frameworks and services High Availability for HDFS HDFS Federation HDFS Snapshots NFSv...

    hadoop 权威指南(中文版)

    Looking to get the most out of your data? This is your book. Use the Hadoop Distributed File System (HDFS) for storing large datasets, then run distributed computations over those datasets ...

    Sams.Teach.Yourself.Big.Data.Analytics.with.Microsoft.HDInsight

    and "Watch Out!" alerts help you avoid problems. By the time you're finished, you'll be comfortable going beyond the book to create any HDInsight app you can imagine! Table of Contents Part I: ...

    Hadoop in Practice 第二版 英文版

    The first edition of the book covered Hadoop 0.22 (Hadoop 1 wasn’t yet out), and Hadoop 2 has turned the world upside-down and opened up the Hadoop platform to processing paradigms beyond MapReduce....

    Practical.Hive.A.Guide.to.Hadoops.Data.Warehouse.System.1484202724

    Dive into the world of SQL on Hadoop and get the most out of your Hive data warehouses. This book is your go-to resource for using Hive: authors Scott Shaw, Ankur Gupta, David Kjerrumgaard, and ...

    Hadoop Succinctly

    It also walks through a Java MapReduce example, illustrates how to write the same query in Python and .NET, and discusses the wider Hadoop ecosystem. Table of Contents Introducing Hadoop Getting ...

    Hadoop: The Definitive Guide [Paperback]

    Use the Hadoop Distributed File System (HDFS) for storing large datasets, then run distributed computations over those datasets with MapReduce Become familiar with Hadoop’s data and I/O building ...

    Hadoop.Essentials.1784396680

    If you are a system or application developer interested in learning how to solve practical problems using the Hadoop framework, then this book is ideal for you. This book is also meant for Hadoop ...

    HBase.High.Performance.Cookbook.epub

    Find out how the communication between the client and server happens in HBase Grasp when to use and avoid MapReduce and how to perform various tasks with it Get to know the concepts of scaling with ...

    Hadoop in 24 Hours, Sams Teach Yourself

    Mastering basic MapReduce Java programming, and using advanced MapReduce API concepts Making the most of Apache Pig and Apache Hive Implementing and administering YARN Taking advantage of the full ...

    Mahout in Action

    Trying out various distance measures 129 7.6 Summary 129 8 Representing data 130 8.1 Visualizing vectors 131 Transforming data into vectors 132 ■ Preparing vectors for use by Mahout 134 8.2 ...

    Considerations for Big Data Architecture and Approach

    common character tics of scale-out, elasticity and high availability. MapReduce, in conjunction with the Hadoop Distributed File System (HDFS) and HBase database, as part of the Apache Hadoop project ...

    Elasticsearch for Hadoop

    Right after this, you will learn to successfully import Hadoop data into Elasticsearch by writing MapReduce job in a real-world example. This is then followed by a comprehensive look at Elasticsearch...

    Hadoop.Application.Architectures.1491900083

    Best practices for moving data in and out of the system Data processing frameworks, including MapReduce, Spark, and Hive Common Hadoop processing patterns, such as removing duplicate records and using...

    Algorithms.for.Data.Science

    By the end of the book, the reader will have gained the ability to adapt algorithms to new problems and carry out innovative analyses. This book has three parts (a) Data Reduction: Begins with the ...

    Frank Kane's Taming Big Data with Apache Spark and Python 【含代码】

    Frank will start you off by teaching you how to set up Spark on a single system or on a cluster, and you'll soon move on to analyzing large data sets using Spark RDD, and developing and running ...

    Springer.The.Developer’s.Guide.to.Debugging.2008.pdf

    9.6.1 Mismatching System Libraries . . . . . . . . 119 9.6.2 Mismatching Object Files . . . . . . . . . 119 9.6.3 Runtime Crashes . . . . . . . . . . . 120 9.6.4 Determining the Compiler Version . . ....

Global site tag (gtag.js) - Google Analytics