读取 Hadoop ArrayWritable 中包装的值

2024-01-10

我是 Hadoop 和 Java 的新手。我的映射器输出文本和 Arraywritable。我在读取 ArrayWritable 值时遇到问题。 Unbale 将 .get() 值转换为整数。附上映射器和减速器代码。有人可以帮我纠正我的减速器代码以便读取 ArrayWritable 值吗?

public static class Temp2Mapper extends Mapper<LongWritable, Text, Text, ArrayWritable>{
    private static final int MISSING=9999;

    @Override public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException{
        String line = value.toString();
        String date = line.substring(07,14);
        int maxTemp,minTemp,avgTemp;

        IntArrayWritable carrier = new IntArrayWritable();
        IntWritable innercarrier[] = new IntWritable[3];
        maxTemp=Integer.parseInt(line.substring(39,45));
        minTemp=Integer.parseInt(line.substring(47,53));
        avgTemp=Integer.parseInt(line.substring(63,69));
        if (maxTemp!= MISSING)
        innercarrier[0]=new IntWritable(maxTemp); // maximum Temperature
        if (minTemp!= MISSING)
        innercarrier[1]=new IntWritable(minTemp); //minimum temperature
        if (avgTemp!= MISSING)
        innercarrier[2]=new IntWritable(avgTemp); // average temperature of 24 hours

        carrier.set(innercarrier);
        context.write(new Text(date), carrier); // Output Text and ArrayWritable
        }
}

public static class Temp2Reducer
extends Reducer<Text, ArrayWritable, Text, IntWritable>{
@Override public void reduce(Text key, Iterable<ArrayWritable> values, Context context ) 
            throws IOException, InterruptedException {

          int max = Integer.MIN_VALUE;
          int[] arr= new int[3];

          for (ArrayWritable val : values) {
              arr = (Int) val.get(); // Error: cannot cast Writable to int
              max = Math.max(max, arr[0]);
          }

          context.write( key, new IntWritable(max) );
        }

}

数组可写#get https://hadoop.apache.org/docs/current/api/org/apache/hadoop/io/ArrayWritable.html#get%28%29方法返回一个数组Writable.

你不能投射一个数组Writable to int。你能做的是:

  1. 迭代这个数组
  2. 投射每个项目(其类型为Writable) 的数组到可写 https://hadoop.apache.org/docs/current/api/org/apache/hadoop/io/IntWritable.html
  3. use IntWritable#get https://hadoop.apache.org/docs/current/api/org/apache/hadoop/io/IntWritable.html#get%28%29方法得到int value.
for (ArrayWritable val: values) {
  for (Writable writable: val.get()) {                 // iterate
     IntWritable intWritable = (IntWritable)writable;  // cast
     int value = intWritable.get();                    // get
     // do your thing with int value
  }
}
本文内容由网友自发贡献,版权归原作者所有,本站不承担相应法律责任。如您发现有涉嫌抄袭侵权的内容,请联系:hwhale#tublm.com(使用前将#替换为@)

读取 Hadoop ArrayWritable 中包装的值 的相关文章

随机推荐