论坛首页 Java企业应用论坛

hadoop伪分布式找不到自定义的mpper类

浏览 2979 次
精华帖 (0) :: 良好帖 (0) :: 新手帖 (0) :: 隐藏帖 (0)
作者 正文
   发表时间:2011-11-21  
hi
问题描述:
  独立模式下运行:hadoop 编译后class /input  /output  正常
  伪分布式下操作:
        前提,已经做了位分布式的配置,启动,也没报错。
      1.把文件放到hdfs中 hdfs df -put /data/  /data/  中
      2. 执行 hadoop class  /data  /output
   异常如下:[root@localhost conf]# hadoop data.Max /user/root/test/sample.txt hdfs://localhost/user/root/output
11/11/22 02:26:13 INFO security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000
11/11/22 02:26:14 WARN conf.Configuration: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
11/11/22 02:26:14 WARN mapreduce.JobSubmitter: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
11/11/22 02:26:14 WARN mapreduce.JobSubmitter: No job jar file set.  User classes may not be found. See Job or Job#setJar(String).
11/11/22 02:26:14 INFO input.FileInputFormat: Total input paths to process : 1
11/11/22 02:26:14 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
11/11/22 02:26:14 INFO mapreduce.JobSubmitter: number of splits:37
11/11/22 02:26:14 INFO mapreduce.JobSubmitter: adding the following namenodes' delegation tokens:null
11/11/22 02:26:14 INFO mapreduce.Job: Running job: job_201111190549_0014
11/11/22 02:26:16 INFO mapreduce.Job:  map 0% reduce 0%
11/11/22 02:26:45 INFO mapreduce.Job: Task Id : attempt_201111190549_0014_m_000000_0, Status : FAILED
java.lang.RuntimeException: java.lang.ClassNotFoundException: data.MaxMapper
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1128)
        at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:167)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:612)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:328)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:742)
        at org.apache.hadoop.mapred.Child.main(Child.java:211)
Caused by: java.lang.ClassNotFoundException: data.MaxMapper
        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
        at sun.misc.Launcher$AppClassL
11/11/22 02:26:45 INFO mapreduce.Job: Task Id : attempt_201111190549_0014_m_000001_0, Status : FAILED
java.lang.RuntimeException: java.lang.ClassNotFoundException: data.MaxMapper
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1128)
        at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:167)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:612)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:328)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:742)
        at org.apache.hadoop.mapred.Child.main(Child.java:211)
Caused by: java.lang.ClassNotFoundException: data.MaxMapper
        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
        at sun.misc.Launcher$AppClassL
11/11/22 02:27:02 INFO mapreduce.Job: Task Id : attempt_201111190549_0014_m_000001_1, Status : FAILED
java.lang.RuntimeException: java.lang.ClassNotFoundException: data.MaxMapper
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1128)
        at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:167)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:612)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:328)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:742)
        at org.apache.hadoop.mapred.Child.main(Child.java:211)
Caused by: java.lang.ClassNotFoundException: data.MaxMapper
        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
        at sun.misc.Launcher$AppClassL
11/11/22 02:27:02 INFO mapreduce.Job: Task Id : attempt_201111190549_0014_m_000000_1, Status : FAILED
java.lang.RuntimeException: java.lang.ClassNotFoundException: data.MaxMapper
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1128)
        at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:167)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:612)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:328)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:742)
        at org.apache.hadoop.mapred.Child.main(Child.java:211)
Caused by: java.lang.ClassNotFoundException: data.MaxMapper
        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
        at sun.misc.Launcher$AppClassL
11/11/22 02:27:18 INFO mapreduce.Job: Task Id : attempt_201111190549_0014_m_000000_2, Status : FAILED
java.lang.RuntimeException: java.lang.ClassNotFoundException: data.MaxMapper
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1128)
        at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:167)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:612)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:328)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:742)
        at org.apache.hadoop.mapred.Child.main(Child.java:211)
Caused by: java.lang.ClassNotFoundException: data.MaxMapper
        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
        at sun.misc.Launcher$AppClassL
11/11/22 02:27:18 INFO mapreduce.Job: Task Id : attempt_201111190549_0014_m_000001_2, Status : FAILED
java.lang.RuntimeException: java.lang.ClassNotFoundException: data.MaxMapper
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1128)
        at org.apache.hadoop.mapreduce.task.JobContextImpl.getMapperClass(JobContextImpl.java:167)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:612)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:328)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:217)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:742)
        at org.apache.hadoop.mapred.Child.main(Child.java:211)
Caused by: java.lang.ClassNotFoundException: data.MaxMapper
        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
        at sun.misc.Launcher$AppClassL

求解决办法
   发表时间:2011-11-21  
User classes may not be found. See Job or Job#setJar(String).

设置
setJarByClass
0 请登录后投票
论坛首页 Java企业应用版

跳转论坛:
Global site tag (gtag.js) - Google Analytics