scala - Some.asInstanceOf[T] or Some.get.asInstanceOf[T] for json -
i have rdd json rows:
val jsons = sc.textfile("hdfs://" + directory + "articles_json/*/*").flatmap(_.split("\n")). map(x => json.parsefull(x))
each json has field "dc:title" , want create rdd these titles , indexes.
val titles_rdd = jsons.filter(x => x.isdefined). map(x => x.get.asinstanceof[map[string, any]]. get("dc:title").get.asinstanceof[string]).zipwithindex()
but, don't understand, should use .get in x => x.get.asinstanceof
in map, or x => x.asinstanceof
? , same question .get after get("dc:title")
?
did try sqlcontext? parsing simpler this.
https://spark.apache.org/docs/1.1.0/sql-programming-guide.html#json-datasets
it great, if can give sample json of yours
edited:
i assume question,
you have list,
scala> val = list(some(1),some(2),some(3),none,some(4)) a: list[option[int]] = list(some(1), some(2), some(3), none, some(4))
you want know whether should using below retrieve values,
scala> val b = a.filter{_.isdefined}.map{x => x.get.asinstanceof[int]} b: list[int] = list(1, 2, 3, 4)
or
like this,
scala> val b = a.filter{_.isdefined}.map{x => x.asinstanceof[int]}
if run above code, you'll below exception.
java.lang.classcastexception: scala.some cannot cast java.lang.integer @ scala.runtime.boxesruntime.unboxtoint(boxesruntime.java:105) @ $anonfun$2.apply(:8) @ $anonfun$2.apply(:8) @ scala.collection.immutable.list.map(list.scala:272) ... 33 elided
reason pretty simple, you want value residing inside some, question how convert desired object.
in above example , line 2,
map (x => x...)
x of type ,if want value , have call function or else won't value.
below link of help. http://www.scala-lang.org/api/current/index.html#scala.some
please let me know if question still stands unclarified
Comments
Post a Comment