您收到“java.lang.NoClassDefFoundError：org/apache/spark/sql/Dataset”错误，因为 pom.xml 文件中缺少“spark-sql”依赖项。
如果您想使用 Spark 2.0.0 读取 Cassandra 表，那么您需要以下最小依赖项。
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>2.0.0</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.11</artifactId>
<version>2.0.0</version>
</dependency>
<dependency>
<groupId>com.datastax.spark</groupId>
<artifactId>spark-cassandra-connector_2.11</artifactId>
<version>2.0.0-M3</version>
</dependency>
Spark 2.0.0提供了SparkSession和Dataset API。下面是读取 Cassandra 表并打印记录的示例程序。
public class SparkCassandraDatasetApplication {
public static void main(String[] args) {
SparkSession spark = SparkSession
.builder()
.appName("SparkCassandraDatasetApplication")
.config("spark.sql.warehouse.dir", "/file:C:/temp")
.config("spark.cassandra.connection.host", "127.0.0.1")
.config("spark.cassandra.connection.port", "9042")
.master("local[2]")
.getOrCreate();
//Read data
Dataset<Row> dataset = spark.read().format("org.apache.spark.sql.cassandra")
.options(new HashMap<String, String>() {
{
put("keyspace", "mykeyspace");
put("table", "mytable");
}
}).load();
//Print data
dataset.show();
spark.stop();
}
}
如果您仍然想使用 RDD，请使用下面的示例程序。
public class SparkCassandraRDDApplication {
public static void main(String[] args) {
SparkConf conf = new SparkConf()
.setAppName("SparkCassandraRDDApplication")
.setMaster("local[2]")
.set("spark.cassandra.connection.host", "127.0.0.1")
.set("spark.cassandra.connection.port", "9042");
JavaSparkContext sc = new JavaSparkContext(conf);
//Read
JavaRDD<UserData> resultsRDD = javaFunctions(sc).cassandraTable("mykeyspace", "mytable",CassandraJavaUtil.mapRowTo(UserData.class));
//Print
resultsRDD.foreach(data -> {
System.out.println(data.id);
System.out.println(data.username);
});
sc.stop();
}
}
上面程序中使用的Javabean（UserData）如下所示。
public class UserData implements Serializable{
String id;
String username;
public String getId() {
return id;
}
public void setId(String id) {
this.id = id;
}
public String getUsername() {
return username;
}
public void setUsername(String username) {
this.username = username;
}
}
