Spark Histogram

방법 1. hive Query를 이용한 방법 val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc) var data = sqlContext.sql("select prob from  hl_temp_test.test_prediction_20161109_20161110") var hists = data.selectExpr("histogram_numeric(prob, 3)") 방법 2. Spark rdd Histogram을 이용한 방법 val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc) var data = sqlContext.sql("select prob from  hl_temp_test.test_prediction_20161109_20161110") mRdd.histogram(10) ==> 결과  (Array[Double], Array[Long]) = (Array(4.9408226914238185E-5, 0.09980998273167643, 0.19957055723643863, 0.2993311317412008, 0.399091706245963, 0.4988522807507252, 0.5986128552554874, 0.6983734297602495, 0.7981340042650118, 0.897894578769774, 0.9976551532745361),Array(99176, 10824, 6279, 8606, 19607, 25154, 25149, 21350, 15702, 25709))

※ bin size가 동일한 것 같지는 않음. hive는 bin size가 동일하지 않다고 하던데, Spark은 확인 필요