MapReduce就是一个利用分而治之的思想做计算的框架，所谓分，就是将数据打散，分成可以计算的小份，治就是将数据合并，相同键的数据合并成一个集合。MapReduce并不能解决所有的问题，因为他的数据类型是键值对，只能解决特定范围的问题。
数据去重算法，其实就是词频统计的一个变种，词频统计是统计文本中的单词出现的次数，如果一个单词出现两次，就算重复，去掉重复的办法就是保留一个副本，键值对表示的话，我们只需要保留键的值就可以了。其实MapReduce最适合干的工作就是数据去重，因为在reduce阶段，所有的数据的键都是唯一的，正好满足数据去重的要求。
在map阶段，我们将value作为key输出，将new Text()作为value就是空输出，这样我们就把所有的数据遍历了一遍，这里面，我们几乎不用什么复杂的算法。
在shuffle阶段，这些值相同的key会被合并到一块，value-list是一个new Text()的集合，如果有重复的记录，那么这个集合的长度会大于1，而且这些key还会进行默认的排序。
在reduce阶段，我们的输入会变为<key , <value-list>>，我们只需要将key一一输出即可，这样就达到了数据去重的目的。思路其实很简单。
算法代码：
package com.xxx.hadoop.mapred;
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
/**
* 数据去重
*
*/
public class DeDuplicationApp {
public static class Map extends Mapper<Object, Text, Text, Text>{
protected void map(Object key, Text value,
Context context) throws IOException ,InterruptedException {
context.write(value, new Text());
};
}
public static class Reduce extends Reducer<Text, Text, Text, Text>{
protected void reduce(Text key, java.lang.Iterable<Text> values,
Context context) throws IOException ,InterruptedException {
context.write(key, new Text());
};
}
public static void main(String[] args) throws Exception {
String input = "/user/root/deduplication/input",
output = "/user/root/deduplication/output";
System.setProperty("HADOOP_USER_NAME", "root");
Configuration conf = new Configuration();
conf.set("fs.defaultFS", "hdfs://192.168.56.202:9000");
Job job = Job.getInstance(conf);
job.setJarByClass(DeDuplicationApp.class);
job.setMapperClass(Map.class);
job.setReducerClass(Reduce.class);
FileInputFormat.addInputPath(job, new Path(input));
FileOutputFormat.setOutputPath(job, new Path(output));
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(Text.class);
System.exit(job.waitForCompletion(true)?0:1);
}
}
原始数据：
运行程序，控制台打印信息如下：
2019-08-31 14:33:56 [INFO ] [main] [org.apache.hadoop.conf.Configuration.deprecation] session.id is deprecated. Instead, use dfs.metrics.session-id
2019-08-31 14:33:56 [INFO ] [main] [org.apache.hadoop.metrics.jvm.JvmMetrics] Initializing JVM Metrics with processName=JobTracker, sessionId=
2019-08-31 14:33:57 [WARN ] [main] [org.apache.hadoop.mapreduce.JobResourceUploader] Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
2019-08-31 14:33:57 [WARN ] [main] [org.apache.hadoop.mapreduce.JobResourceUploader] No job jar file set. User classes may not be found. See Job or Job#setJar(String).
2019-08-31 14:33:57 [INFO ] [main] [org.apache.hadoop.mapreduce.lib.input.FileInputFormat] Total input paths to process : 3
2019-08-31 14:33:57 [INFO ] [main] [org.apache.hadoop.mapreduce.JobSubmitter] number of splits:3
2019-08-31 14:33:57 [INFO ] [main] [org.apache.hadoop.mapreduce.JobSubmitter] Submitting tokens for job: job_local1661418487_0001
2019-08-31 14:33:58 [INFO ] [main] [org.apache.hadoop.mapreduce.Job] The url to track the job: http://localhost:8080/
2019-08-31 14:33:58 [INFO ] [main] [org.apache.hadoop.mapreduce.Job] Running job: job_local1661418487_0001
2019-08-31 14:33:58 [INFO ] [Thread-3] [org.apache.hadoop.mapred.LocalJobRunner] OutputCommitter set in config null
2019-08-31 14:33:58 [INFO ] [Thread-3] [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter] File Output Committer Algorithm version is 1
2019-08-31 14:33:58 [INFO ] [Thread-3] [org.apache.hadoop.mapred.LocalJobRunner] OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
2019-08-31 14:33:58 [INFO ] [Thread-3] [org.apache.hadoop.mapred.LocalJobRunner] Waiting for map tasks
2019-08-31 14:33:58 [INFO ] [LocalJobRunner Map Task Executor #0] [org.apache.hadoop.mapred.LocalJobRunner] Starting task: attempt_local1661418487_0001_m_000000_0
2019-08-31 14:33:58 [INFO ] [LocalJobRunner Map Task Executor #0] [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter] File Output Committer Algorithm version is 1
2019-08-31 14:33:58 [INFO ] [LocalJobRunner Map Task Executor #0] [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] ProcfsBasedProcessTree currently is supported only on Linux.
2019-08-31 14:33:58 [INFO ] [LocalJobRunner Map Task Executor #0] [org.apache.hadoop.mapred.Task] Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@3c3c2e72
2019-08-31 14:33:58 [INFO ] [LocalJobRunner Map Task Executor #0] [org.apache.hadoop.mapred.MapTask] Processing split: hdfs://192.168.56.202:9000/user/root/deduplication/input/a.txt:0+12
2019-08-31 14:33:58 [INFO ] [LocalJobRunner Map Task Executor #0] [org.apache.hadoop.mapred.MapTask] (EQUATOR) 0 kvi 26214396(104857584)
2019-08-31 14:33:58 [INFO ] [LocalJobRunner Map Task Executor #0] [org.apache.hadoop.mapred.MapTask] mapreduce.task.io.sort.mb: 100
2019-08-31 14:33:58 [INFO ] [LocalJobRunner Map Task Executor #0] [org.apache.hadoop.mapred.MapTask] soft limit at 83886080
2019-08-31 14:33:58 [INFO ] [LocalJobRunner Map Task Executor #0] [org.apache.hadoop.mapred.MapTask] bufstart = 0; bufvoid = 104857600
2019-08-31 14:33:58 [INFO ] [LocalJobRunner Map Task Executor #0] [org.apache.hadoop.mapred.MapTask] kvstart = 26214396; length = 6553600
2019-08-31 14:33:58 [INFO ] [LocalJobRunner Map Task Executor #0] [org.apache.hadoop.mapred.MapTask] Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2019-08-31 14:33:58 [INFO ] [LocalJobRunner Map Task Executor #0] [org.apache.hadoop.mapred.LocalJobRunner]
2019-08-31 14:33:58 [INFO ] [LocalJobRunner Map Task Executor #0] [org.apache.hadoop.mapred.MapTask] Starting flush of map output
2019-08-31 14:33:58 [INFO ] [LocalJobRunner Map Task Executor #0] [org.apache.hadoop.mapred.MapTask] Spilling map output
2019-08-31 14:33:58 [INFO ] [LocalJobRunner Map Task Executor #0] [org.apache.hadoop.mapred.MapTask] bufstart = 0; bufend = 15; bufvoid = 104857600
2019-08-31 14:33:58 [INFO ] [LocalJobRunner Map Task Executor #0] [org.apache.hadoop.mapred.MapTask] kvstart = 26214396(104857584); kvend = 26214388(104857552); length = 9/6553600
2019-08-31 14:33:58 [INFO ] [LocalJobRunner Map Task Executor #0] [org.apache.hadoop.mapred.MapTask] Finished spill 0
2019-08-31 14:33:58 [INFO ] [LocalJobRunner Map Task Executor #0] [org.apache.hadoop.mapred.Task] Task:attempt_local1661418487_0001_m_000000_0 is done. And is in the process of committing
2019-08-31 14:33:58 [INFO ] [LocalJobRunner Map Task Executor #0] [org.apache.hadoop.mapred.LocalJobRunner] map
2019-08-31 14:33:58 [INFO ] [LocalJobRunner Map Task Executor #0] [org.apache.hadoop.mapred.Task] Task 'attempt_local1661418487_0001_m_000000_0' done.
2019-08-31 14:33:58 [INFO ] [LocalJobRunner Map Task Executor #0] [org.apache.hadoop.mapred.LocalJobRunner] Finishing task: attempt_local1661418487_0001_m_000000_0
2019-08-31 14:33:58 [INFO ] [LocalJobRunner Map Task Executor #0] [org.apache.hadoop.mapred.LocalJobRunner] Starting task: attempt_local1661418487_0001_m_000001_0
2019-08-31 14:33:58 [INFO ] [LocalJobRunner Map Task Executor #0] [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter] File Output Committer Algorithm version is 1
2019-08-31 14:33:58 [INFO ] [LocalJobRunner Map Task Executor #0] [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] ProcfsBasedProcessTree currently is supported only on Linux.
2019-08-31 14:33:59 [INFO ] [LocalJobRunner Map Task Executor #0] [org.apache.hadoop.mapred.Task] Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@4b250759
2019-08-31 14:33:59 [INFO ] [LocalJobRunner Map Task Executor #0] [org.apache.hadoop.mapred.MapTask] Processing split: hdfs://192.168.56.202:9000/user/root/deduplication/input/b.txt:0+12
2019-08-31 14:33:59 [INFO ] [LocalJobRunner Map Task Executor #0] [org.apache.hadoop.mapred.MapTask] (EQUATOR) 0 kvi 26214396(104857584)
2019-08-31 14:33:59 [INFO ] [LocalJobRunner Map Task Executor #0] [org.apache.hadoop.mapred.MapTask] mapreduce.task.io.sort.mb: 100
2019-08-31 14:33:59 [INFO ] [LocalJobRunner Map Task Executor #0] [org.apache.hadoop.mapred.MapTask] soft limit at 83886080
2019-08-31 14:33:59 [INFO ] [LocalJobRunner Map Task Executor #0] [org.apache.hadoop.mapred.MapTask] bufstart = 0; bufvoid = 104857600
2019-08-31 14:33:59 [INFO ] [LocalJobRunner Map Task Executor #0] [org.apache.hadoop.mapred.MapTask] kvstart = 26214396; length = 6553600
2019-08-31 14:33:59 [INFO ] [LocalJobRunner Map Task Executor #0] [org.apache.hadoop.mapred.MapTask] Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2019-08-31 14:33:59 [INFO ] [LocalJobRunner Map Task Executor #0] [org.apache.hadoop.mapred.LocalJobRunner]
2019-08-31 14:33:59 [INFO ] [LocalJobRunner Map Task Executor #0] [org.apache.hadoop.mapred.MapTask] Starting flush of map output
2019-08-31 14:33:59 [INFO ] [LocalJobRunner Map Task Executor #0] [org.apache.hadoop.mapred.MapTask] Spilling map output
2019-08-31 14:33:59 [INFO ] [LocalJobRunner Map Task Executor #0] [org.apache.hadoop.mapred.MapTask] bufstart = 0; bufend = 15; bufvoid = 104857600
2019-08-31 14:33:59 [INFO ] [LocalJobRunner Map Task Executor #0] [org.apache.hadoop.mapred.MapTask] kvstart = 26214396(104857584); kvend = 26214388(104857552); length = 9/6553600
2019-08-31 14:33:59 [INFO ] [LocalJobRunner Map Task Executor #0] [org.apache.hadoop.mapred.MapTask] Finished spill 0
2019-08-31 14:33:59 [INFO ] [LocalJobRunner Map Task Executor #0] [org.apache.hadoop.mapred.Task] Task:attempt_local1661418487_0001_m_000001_0 is done. And is in the process of committing
2019-08-31 14:33:59 [INFO ] [LocalJobRunner Map Task Executor #0] [org.apache.hadoop.mapred.LocalJobRunner] map
2019-08-31 14:33:59 [INFO ] [LocalJobRunner Map Task Executor #0] [org.apache.hadoop.mapred.Task] Task 'attempt_local1661418487_0001_m_000001_0' done.
2019-08-31 14:33:59 [INFO ] [LocalJobRunner Map Task Executor #0] [org.apache.hadoop.mapred.LocalJobRunner] Finishing task: attempt_local1661418487_0001_m_000001_0
2019-08-31 14:33:59 [INFO ] [LocalJobRunner Map Task Executor #0] [org.apache.hadoop.mapred.LocalJobRunner] Starting task: attempt_local1661418487_0001_m_000002_0
2019-08-31 14:33:59 [INFO ] [LocalJobRunner Map Task Executor #0] [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter] File Output Committer Algorithm version is 1
2019-08-31 14:33:59 [INFO ] [LocalJobRunner Map Task Executor #0] [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] ProcfsBasedProcessTree currently is supported only on Linux.
2019-08-31 14:33:59 [INFO ] [LocalJobRunner Map Task Executor #0] [org.apache.hadoop.mapred.Task] Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@77ad94ef
2019-08-31 14:33:59 [INFO ] [LocalJobRunner Map Task Executor #0] [org.apache.hadoop.mapred.MapTask] Processing split: hdfs://192.168.56.202:9000/user/root/deduplication/input/c.txt:0+12
2019-08-31 14:33:59 [INFO ] [main] [org.apache.hadoop.mapreduce.Job] Job job_local1661418487_0001 running in uber mode : false
2019-08-31 14:33:59 [INFO ] [main] [org.apache.hadoop.mapreduce.Job] map 100% reduce 0%
2019-08-31 14:33:59 [INFO ] [LocalJobRunner Map Task Executor #0] [org.apache.hadoop.mapred.MapTask] (EQUATOR) 0 kvi 26214396(104857584)
2019-08-31 14:33:59 [INFO ] [LocalJobRunner Map Task Executor #0] [org.apache.hadoop.mapred.MapTask] mapreduce.task.io.sort.mb: 100
2019-08-31 14:33:59 [INFO ] [LocalJobRunner Map Task Executor #0] [org.apache.hadoop.mapred.MapTask] soft limit at 83886080
2019-08-31 14:33:59 [INFO ] [LocalJobRunner Map Task Executor #0] [org.apache.hadoop.mapred.MapTask] bufstart = 0; bufvoid = 104857600
2019-08-31 14:33:59 [INFO ] [LocalJobRunner Map Task Executor #0] [org.apache.hadoop.mapred.MapTask] kvstart = 26214396; length = 6553600
2019-08-31 14:33:59 [INFO ] [LocalJobRunner Map Task Executor #0] [org.apache.hadoop.mapred.MapTask] Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2019-08-31 14:33:59 [INFO ] [LocalJobRunner Map Task Executor #0] [org.apache.hadoop.mapred.LocalJobRunner]
2019-08-31 14:33:59 [INFO ] [LocalJobRunner Map Task Executor #0] [org.apache.hadoop.mapred.MapTask] Starting flush of map output
2019-08-31 14:33:59 [INFO ] [LocalJobRunner Map Task Executor #0] [org.apache.hadoop.mapred.MapTask] Spilling map output
2019-08-31 14:33:59 [INFO ] [LocalJobRunner Map Task Executor #0] [org.apache.hadoop.mapred.MapTask] bufstart = 0; bufend = 15; bufvoid = 104857600
2019-08-31 14:33:59 [INFO ] [LocalJobRunner Map Task Executor #0] [org.apache.hadoop.mapred.MapTask] kvstart = 26214396(104857584); kvend = 26214388(104857552); length = 9/6553600
2019-08-31 14:33:59 [INFO ] [LocalJobRunner Map Task Executor #0] [org.apache.hadoop.mapred.MapTask] Finished spill 0
2019-08-31 14:33:59 [INFO ] [LocalJobRunner Map Task Executor #0] [org.apache.hadoop.mapred.Task] Task:attempt_local1661418487_0001_m_000002_0 is done. And is in the process of committing
2019-08-31 14:33:59 [INFO ] [LocalJobRunner Map Task Executor #0] [org.apache.hadoop.mapred.LocalJobRunner] map
2019-08-31 14:33:59 [INFO ] [LocalJobRunner Map Task Executor #0] [org.apache.hadoop.mapred.Task] Task 'attempt_local1661418487_0001_m_000002_0' done.
2019-08-31 14:33:59 [INFO ] [LocalJobRunner Map Task Executor #0] [org.apache.hadoop.mapred.LocalJobRunner] Finishing task: attempt_local1661418487_0001_m_000002_0
2019-08-31 14:33:59 [INFO ] [Thread-3] [org.apache.hadoop.mapred.LocalJobRunner] map task executor complete.
2019-08-31 14:33:59 [INFO ] [Thread-3] [org.apache.hadoop.mapred.LocalJobRunner] Waiting for reduce tasks
2019-08-31 14:33:59 [INFO ] [pool-6-thread-1] [org.apache.hadoop.mapred.LocalJobRunner] Starting task: attempt_local1661418487_0001_r_000000_0
2019-08-31 14:33:59 [INFO ] [pool-6-thread-1] [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter] File Output Committer Algorithm version is 1
2019-08-31 14:33:59 [INFO ] [pool-6-thread-1] [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] ProcfsBasedProcessTree currently is supported only on Linux.
2019-08-31 14:33:59 [INFO ] [pool-6-thread-1] [org.apache.hadoop.mapred.Task] Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@651b16b8
2019-08-31 14:33:59 [INFO ] [pool-6-thread-1] [org.apache.hadoop.mapred.ReduceTask] Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@12884be
2019-08-31 14:33:59 [INFO ] [pool-6-thread-1] [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] MergerManager: memoryLimit=1265788544, maxSingleShuffleLimit=316447136, mergeThreshold=835420480, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2019-08-31 14:33:59 [INFO ] [EventFetcher for fetching Map Completion Events] [org.apache.hadoop.mapreduce.task.reduce.EventFetcher] attempt_local1661418487_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2019-08-31 14:33:59 [INFO ] [localfetcher#1] [org.apache.hadoop.mapreduce.task.reduce.LocalFetcher] localfetcher#1 about to shuffle output of map attempt_local1661418487_0001_m_000002_0 decomp: 23 len: 27 to MEMORY
2019-08-31 14:33:59 [INFO ] [localfetcher#1] [org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput] Read 23 bytes from map-output for attempt_local1661418487_0001_m_000002_0
2019-08-31 14:33:59 [INFO ] [localfetcher#1] [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] closeInMemoryFile -> map-output of size: 23, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->23
2019-08-31 14:33:59 [INFO ] [localfetcher#1] [org.apache.hadoop.mapreduce.task.reduce.LocalFetcher] localfetcher#1 about to shuffle output of map attempt_local1661418487_0001_m_000001_0 decomp: 23 len: 27 to MEMORY
2019-08-31 14:33:59 [INFO ] [localfetcher#1] [org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput] Read 23 bytes from map-output for attempt_local1661418487_0001_m_000001_0
2019-08-31 14:33:59 [INFO ] [localfetcher#1] [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] closeInMemoryFile -> map-output of size: 23, inMemoryMapOutputs.size() -> 2, commitMemory -> 23, usedMemory ->46
2019-08-31 14:33:59 [INFO ] [localfetcher#1] [org.apache.hadoop.mapreduce.task.reduce.LocalFetcher] localfetcher#1 about to shuffle output of map attempt_local1661418487_0001_m_000000_0 decomp: 23 len: 27 to MEMORY
2019-08-31 14:33:59 [INFO ] [localfetcher#1] [org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput] Read 23 bytes from map-output for attempt_local1661418487_0001_m_000000_0
2019-08-31 14:33:59 [INFO ] [localfetcher#1] [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] closeInMemoryFile -> map-output of size: 23, inMemoryMapOutputs.size() -> 3, commitMemory -> 46, usedMemory ->69
2019-08-31 14:33:59 [INFO ] [EventFetcher for fetching Map Completion Events] [org.apache.hadoop.mapreduce.task.reduce.EventFetcher] EventFetcher is interrupted.. Returning
2019-08-31 14:33:59 [INFO ] [pool-6-thread-1] [org.apache.hadoop.mapred.LocalJobRunner] 3 / 3 copied.
2019-08-31 14:33:59 [INFO ] [pool-6-thread-1] [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] finalMerge called with 3 in-memory map-outputs and 0 on-disk map-outputs
2019-08-31 14:33:59 [INFO ] [pool-6-thread-1] [org.apache.hadoop.mapred.Merger] Merging 3 sorted segments
2019-08-31 14:33:59 [INFO ] [pool-6-thread-1] [org.apache.hadoop.mapred.Merger] Down to the last merge-pass, with 3 segments left of total size: 51 bytes
2019-08-31 14:33:59 [INFO ] [pool-6-thread-1] [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] Merged 3 segments, 69 bytes to disk to satisfy reduce memory limit
2019-08-31 14:33:59 [INFO ] [pool-6-thread-1] [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] Merging 1 files, 69 bytes from disk
2019-08-31 14:33:59 [INFO ] [pool-6-thread-1] [org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl] Merging 0 segments, 0 bytes from memory into reduce
2019-08-31 14:33:59 [INFO ] [pool-6-thread-1] [org.apache.hadoop.mapred.Merger] Merging 1 sorted segments
2019-08-31 14:33:59 [INFO ] [pool-6-thread-1] [org.apache.hadoop.mapred.Merger] Down to the last merge-pass, with 1 segments left of total size: 59 bytes
2019-08-31 14:33:59 [INFO ] [pool-6-thread-1] [org.apache.hadoop.mapred.LocalJobRunner] 3 / 3 copied.
2019-08-31 14:33:59 [INFO ] [pool-6-thread-1] [org.apache.hadoop.conf.Configuration.deprecation] mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords
2019-08-31 14:33:59 [INFO ] [pool-6-thread-1] [org.apache.hadoop.mapred.Task] Task:attempt_local1661418487_0001_r_000000_0 is done. And is in the process of committing
2019-08-31 14:33:59 [INFO ] [pool-6-thread-1] [org.apache.hadoop.mapred.LocalJobRunner] 3 / 3 copied.
2019-08-31 14:33:59 [INFO ] [pool-6-thread-1] [org.apache.hadoop.mapred.Task] Task attempt_local1661418487_0001_r_000000_0 is allowed to commit now
2019-08-31 14:33:59 [INFO ] [pool-6-thread-1] [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter] Saved output of task 'attempt_local1661418487_0001_r_000000_0' to hdfs://192.168.56.202:9000/user/root/deduplication/output/_temporary/0/task_local1661418487_0001_r_000000
2019-08-31 14:33:59 [INFO ] [pool-6-thread-1] [org.apache.hadoop.mapred.LocalJobRunner] reduce > reduce
2019-08-31 14:33:59 [INFO ] [pool-6-thread-1] [org.apache.hadoop.mapred.Task] Task 'attempt_local1661418487_0001_r_000000_0' done.
2019-08-31 14:33:59 [INFO ] [pool-6-thread-1] [org.apache.hadoop.mapred.LocalJobRunner] Finishing task: attempt_local1661418487_0001_r_000000_0
2019-08-31 14:33:59 [INFO ] [Thread-3] [org.apache.hadoop.mapred.LocalJobRunner] reduce task executor complete.
2019-08-31 14:34:00 [INFO ] [main] [org.apache.hadoop.mapreduce.Job] map 100% reduce 100%
2019-08-31 14:34:00 [INFO ] [main] [org.apache.hadoop.mapreduce.Job] Job job_local1661418487_0001 completed successfully
2019-08-31 14:34:00 [INFO ] [main] [org.apache.hadoop.mapreduce.Job] Counters: 35
File System Counters
FILE: Number of bytes read=4066
FILE: Number of bytes written=1095628
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=108
HDFS: Number of bytes written=30
HDFS: Number of read operations=33
HDFS: Number of large read operations=0
HDFS: Number of write operations=6
Map-Reduce Framework
Map input records=9
Map output records=9
Map output bytes=45
Map output materialized bytes=81
Input split bytes=381
Combine input records=0
Combine output records=0
Reduce input groups=6
Reduce shuffle bytes=81
Reduce input records=9
Reduce output records=6
Spilled Records=18
Shuffled Maps =3
Failed Shuffles=0
Merged Map outputs=3
GC time elapsed (ms)=8
Total committed heap usage (bytes)=1476919296
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=36
File Output Format Counters
Bytes Written=30
计算之后的数据：
这里面数据去重，其实是利用了MapReduce的特点，没有复杂的设计思路。普通的数组去重，我们会遍历数组，然后保存遍历的值，后续遍历的值如果在已经遍历过的值集合中，那么就丢弃。而MapReduce框架是一个分而治之的计算框架，分为几个阶段，每一个阶段都有自己的任务，正是这每些阶段将我们的数据做了分组和排序，所以数据就自动的去重了。