Hadoop学习--MapperReduce--day08

mapper类的代码：

专业领域包括网站制作、成都网站设计、电子商务商城网站建设、微信营销、系统平台开发， 与其他网站设计及系统开发公司不同，成都创新互联公司的整合解决方案结合了帮做网络品牌建设经验和互联网整合营销的理念，并将策略和执行紧密结合，为客户提供全网互联网整合方案。

实现Mapper类的方法

import java.io.IOException;

import org.apache.hadoop.io.IntWritable;

import org.apache.hadoop.io.LongWritable;

import org.apache.hadoop.io.Text;

import org.apache.hadoop.mapreduce.Mapper;

public class MyMapper extends Mapper<LongWritable, Text, Text, IntWritable> {

// constant

private static final int MISSING = 9999;

// map function

@Override

protected void map(LongWritable key, Text value, Mapper<LongWritable, Text, Text, IntWritable>.Context context)

throws IOException, InterruptedException {

// per line

String line = value.toString();

// get year

String year = line.substring(15, 19);

// get airtemp

int airTemperature;

if (line.charAt(87) == '+') {

airTemperature = Integer.parseInt(line.substring(88, 92));

} else {

airTemperature = Integer.parseInt(line.substring(87, 92));

}

// valid air temp data

String quality = line.substring(92, 93);

if (airTemperature != MISSING && quality.matches("[01459]")) {

context.write(new Text(year), new IntWritable(airTemperature));

}

}

}

Reduce类的代码：

实现Reducer类的方法

import java.io.IOException;

import org.apache.hadoop.io.IntWritable;

import org.apache.hadoop.io.Text;

import org.apache.hadoop.mapreduce.Reducer;

public class MyReducer extends Reducer<Text, IntWritable, Text, IntWritable> {

@Override

protected void reduce(Text key, Iterable<IntWritable> values, Context context)

throws IOException, InterruptedException {

// max

int maxValue = Integer.MIN_VALUE;

// for

for (IntWritable value : values) {

maxValue = Math.max(maxValue, value.get());

}

// output

context.write(key, new IntWritable(maxValue));

}

}

主方法的代码：

import mapper类实现.MyMapper;

import reducer类实现.MyReducer;

import org.apache.hadoop.fs.Path;

import org.apache.hadoop.io.IntWritable;

import org.apache.hadoop.io.Text;

import org.apache.hadoop.mapreduce.Job;

import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;

import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

public class MyMapperApp {

/**

* @param args

* @throws Exception

* @throws IllegalArgumentException

*/

public static void main(String[] args) throws IllegalArgumentException, Exception {

// new job

Job job = Job.getInstance();

// find jar by ClassName

job.setJarByClass(MyMapper.class);

// job name

job.setJobName("Max temperature");

FileInputFormat.addInputPath(job, new Path("file:///mnt/hgfs/test-ncdc-data"));

FileOutputFormat.setOutputPath(job, new Path("file:///home/hadoop/mr/"));

job.setMapperClass(MyMapper.class);

job.setReducerClass(MyReducer.class);

job.setOutputKeyClass(Text.class);

job.setOutputValueClass(IntWritable.class);

System.exit(job.waitForCompletion(true) ? 0 : 1);

}

}

网站标题：Hadoop学习--MapperReduce--day08
转载注明：https://www.cdcxhl.com/article4/jhssoe.html

成都网站建设公司_创新互联，为您提供微信公众号、外贸网站建设、网站排名、全网营销推广、品牌网站制作、软件开发

广告

声明：本网站发布的内容（图片、视频和文字）以用户投稿、用户转载内容为主，如果涉及侵权请尽快告知，我们将会在第一时间删除。文章观点不代表本网站立场，如需处理请联系客服。电话：028-86922220；邮箱：631063699@qq.com。内容未经允许不得转载，或转载时需注明来源： 创新互联