我正在尝试在 Spark-1.6.1 上使用 DMLC 的 XGBoost 实现。我能够使用 XGBoost 训练我的数据,但在预测方面面临困难。我实际上想以在 Apache Spark mllib 库中完成的方式进行预测,这有助于计算训练误差、精度、召回率、特异性等。
我发布了下面的代码,以及我收到的错误。
我在spark-shell中使用了这个xgboost4j-spark-0.5-jar-with-dependencies.jar来启动。
import org.apache.spark.mllib.regression.LabeledPoint
import org.apache.spark.mllib.linalg.Vectors
import org.apache.spark.mllib.util.MLUtils
import org.apache.spark.SparkContext._
import org.apache.spark.mllib.linalg.Vectors
import org.apache.spark.mllib.regression.LabeledPoint
import ml.dmlc.xgboost4j.scala.Booster
import ml.dmlc.xgboost4j.scala.spark.XGBoost
import ml.dmlc.xgboost4j.scala.DMatrix
import ml.dmlc.xgboost4j.scala.{Booster, DMatrix}
import ml.dmlc.xgboost4j.scala.spark.{DataUtils, XGBoost}
import org.apache.spark.{SparkConf, SparkContext}
//Load and parse the data file.
val data = sc.textFile("file:///home/partha/credit_approval_2_attr.csv")
val data1 = sc.textFile("file:///home/partha/credit_app_fea.csv")
val parsedData = data.map { line =>
val parts = line.split(',').map(_.toDouble)
LabeledPoint(parts(0), Vectors.dense(parts.tail))
}.cache()
val parsedData1 = data1.map { line =>
val parts = line.split(',').map(_.toDouble)
Vectors.dense(parts)
}
//Tuning Parameters
val paramMap = List(
"eta" -> 0.1f,
"max_depth" -> 5,
"num_class" -> 2,
"objective" -> "multi:softmax" ,
"colsample_bytree" -> 0.8,
"alpha" -> 1,
"subsample" -> 0.5).toMap
//Training the model
val numRound = 20
val model = XGBoost.train(parsedData, paramMap, numRound, nWorkers = 1)
val pred = model.predict(parsedData1)
pred.collect()
pred 的输出:
res0: Array[Array[Array[Float]]] = Array(Array(Array(0.0), Array(1.0), Array(1.0), Array(1.0), Array(0.0), Array(0.0), Array(1.0), Array(1.0), Array(0.0), Array(1.0), Array(0.0), Array(0.0), Array(0.0), Array(1.0), Array(1.0), Array(1.0), Array(1.0), Array(1.0), Array(0.0), Array(1.0), Array(1.0), Array(0.0), Array(1.0), Array(1.0), Array(1.0), Array(1.0), Array(1.0), Array(1.0), Array(1.0), Array(1.0), Array(1.0), Array(1.0), Array(1.0), Array(1.0), Array(1.0), Array(1.0), Array(1.0), Array(1.0), Array(0.0), Array(1.0), Array(1.0), Array(1.0), Array(0.0), Array(1.0), Array(1.0), Array(1.0), Array(1.0), Array(1.0), Array(0.0), Array(0.0), Array(0.0), Array(0.0), Array(1.0), Array(0.0), Array(0.0), Array(0.0), Array(0.0), Array(0.0), Array(0.0), Array(1.0), Array(1.0), Array(1.0), Array(...
现在当我使用时:
val labelAndPreds = parsedData.map { point =>
val prediction = model.predict(point.features)
(point.label, prediction)
}
Output:
<console>:66: error: overloaded method value predict with alternatives:
(testSet: ml.dmlc.xgboost4j.scala.DMatrix)Array[Array[Float]] <and>
(testSet: org.apache.spark.rdd.RDD[org.apache.spark.mllib.linalg.Vector])org.apache.spark.rdd.RDD[Array[Array[Float]]]
cannot be applied to (org.apache.spark.mllib.linalg.Vector)
val prediction = model.predict(point.features)
^
然后尝试这个,因为预测需要 RDD[Vector]。
val labelAndPreds1 = parsedData.map { point =>
val prediction = model.predict(Vectors.dense(point.features))
(point.label, prediction)
}
结果是:
<console>:66: error: overloaded method value dense with alternatives:
(values: Array[Double])org.apache.spark.mllib.linalg.Vector <and>
(firstValue: Double,otherValues: Double*)org.apache.spark.mllib.linalg.Vector
cannot be applied to (org.apache.spark.mllib.linalg.Vector)
val prediction = model.predict(Vectors.dense(point.features))
^
显然这是我正在尝试解决的 RDD 类型问题,这对于 Spark 上的 GBT 来说很容易(http://spark.apache.org/docs/latest/mllib-ensembles.html#gradient-boosted-trees-gbts http://spark.apache.org/docs/latest/mllib-ensembles.html#gradient-boosted-trees-gbts).
我是否尝试以正确的方式做到这一点?
任何帮助或建议都会很棒。