scala - Some.asInstanceOf[T] or Some.get.asInstanceOf[T] for json -


i have rdd json rows:

val jsons = sc.textfile("hdfs://" + directory + "articles_json/*/*").flatmap(_.split("\n")).                         map(x => json.parsefull(x)) 

each json has field "dc:title" , want create rdd these titles , indexes.

val titles_rdd = jsons.filter(x => x.isdefined).       map(x => x.get.asinstanceof[map[string, any]].       get("dc:title").get.asinstanceof[string]).zipwithindex() 

but, don't understand, should use .get in x => x.get.asinstanceof in map, or x => x.asinstanceof? , same question .get after get("dc:title")?

did try sqlcontext? parsing simpler this.

https://spark.apache.org/docs/1.1.0/sql-programming-guide.html#json-datasets

it great, if can give sample json of yours

edited:

i assume question,

you have list,

scala> val = list(some(1),some(2),some(3),none,some(4)) a: list[option[int]] = list(some(1), some(2), some(3), none, some(4)) 

you want know whether should using below retrieve values,

scala> val b = a.filter{_.isdefined}.map{x => x.get.asinstanceof[int]} b: list[int] = list(1, 2, 3, 4) 

or

like this,

scala> val b = a.filter{_.isdefined}.map{x => x.asinstanceof[int]} 

if run above code, you'll below exception.

java.lang.classcastexception: scala.some cannot cast java.lang.integer @ scala.runtime.boxesruntime.unboxtoint(boxesruntime.java:105) @ $anonfun$2.apply(:8) @ $anonfun$2.apply(:8) @ scala.collection.immutable.list.map(list.scala:272) ... 33 elided

reason pretty simple, you want value residing inside some, question how convert desired object.

in above example , line 2,

map (x => x...) 

x of type ,if want value , have call function or else won't value.

below link of help. http://www.scala-lang.org/api/current/index.html#scala.some

please let me know if question still stands unclarified


Comments

Popular posts from this blog

Java 3D LWJGL collision -

spring - SubProtocolWebSocketHandler - No handlers -

methods - python can't use function in submodule -