Why RDDs is not listed in the spark UI/storage page -


i newbie spark.

i installed spark 1.3.1 on mac, , played using spark-shell, here did:

scala> val lfile = sc.textfile("/users/jackzhang/downloads/prodpart.txt"); scala> val count = lfile.filter(line => line.contains("xyz_cow")) scala> count.count 

output

res27: long = 1 

i tried run

scala> count.cache 

it did not work either.

my understanding should count rdd materialized in memory (or disk), because run count.count, , count action per spark documentation, , should able see http://localhost:4040/storage, wrong?

as makoton mentioned, https://forums.databricks.com/questions/117/why-is-my-rdd-not-showing-up-in-the-storage-tab-of.html answered question.

to see rdd in "storage" tab, here did after reading post:

scala> val cachecount = count.cache scala> cachecount.collect 

after that, can see rdd in tab, however, tried update rdd name, running:

scala> cachecount.setname("test") scala> cachecount.cache scala> cachecount.collect 

the name of rdd did not updated. means part of answer posted in forum incorrect.


Comments

Popular posts from this blog

Android : Making Listview full screen -

javascript - Parse JSON from the body of the POST -

javascript - Chrome Extension: Interacting with iframe embedded within popup -