Need some inputs in feature extraction in Apache Spark -

-

i new apache spark , trying use mlib utility analysis. collated code convert data features , apply linear regression algorithm that. facing issues . please , excuse if silly question

my person data looks like

1,1000.00,36 2,2000.00,35 3,2345.50,37 4,3323.00,45

just simple example code working

import org.apache.spark.sparkcontext import org.apache.spark.sparkcontext._ import org.apache.spark.sparkconf import org.apache.spark.mllib.linalg.{vector, vectors} import org.apache.spark.mllib.regression.labeledpoint  case class person(rating: string, income: double, age: int) val persondata = sc.textfile("d:/spark/mydata/persondata.txt").map(_.split(",")).map(p => person(p(0), p(1).todouble, p(2).toint))  def preparefeatures(people: seq[person]): seq[org.apache.spark.mllib.linalg.vector] = {   val maxincome = people.map(_ income) max   val maxage = people.map(_ age) max    people.map (p =>     vectors.dense(       if (p.rating == "a") 0.7 else if (p.rating == "b") 0.5 else 0.3,       p.income / maxincome,       p.age.todouble / maxage)) }   def preparefeatureswithlabels(features: seq[org.apache.spark.mllib.linalg.vector]): seq[labeledpoint] =   (0d 1 (1d / features.length)) zip(features) map(l => labeledpoint(l._1, l._2))  ---its working till here. ---it breaks in below code  val data = sc.parallelize(preparefeatureswithlabels(preparefeatures(people))  scala> val data = sc.parallelize(preparefeatureswithlabels(preparefeatures(people))) <console>:36: error: not found: value people error occurred in application involving default arguments.        val data = sc.parallelize(preparefeatureswithlabels(preparefeatures(people)))                                                                            ^

please advise

you seem going in right direction there few minor problems. first off trying reference value (people) haven't defined. more seem writing code work sequences, , instead should modify code work rdds (or dataframes). seem using parallelize try , parallelize operation, parallelize helper method take local collection , make available distributed rdd. i'd recommend looking @ programming guides or additional documentation better understanding of spark apis. best of luck adventures spark.


Comments

Post a Comment

Popular posts from this blog

Android : Making Listview full screen -

-
<linearlayout xmlns:android="http://schemas.android.com/apk/res/android" xmlns:tools="http://schemas.android.com/tools" android:layout_width="match_parent" android:layout_height="match_parent" android:orientation="vertical" tools:context=".mainactivity"> <edittext android:layout_margintop="5dp" android:layout_width="match_parent" android:layout_height="50dp" android:background="@drawable/location_edittext" android:id="@+id/locationeditext" android:textsize="15sp" android:textcolor="#999" android:paddingleft="15dp" /> <com.example.ravi_gupta.slider.viewpagercustomduration android:layout_width="wrap_content" android:layout_height="0dp" android:id="@+id/viewpager" android:scrollbarstyle="outsideoverlay" android:layout_weight="0.5...
Read more

javascript - Parse JSON from the body of the POST -

-
i'm using cameratag.com record video on website. service can send json response server can save data database. don't know how proceed here: the documentation states please note json repsonse passed in body of post , not form parameters. here example of how parse response in ruby: data = json.parse(request.raw_post) video_uuid = data["uuid"] but don't know how data on javascript. i'm not experienced in js please bare me :) post sample: { "uuid": "00012710-4b65-0131-ffe6-22000a499ea4", "camera_uuid": "98373a20-79ee-0130-3e42-1231390fcc11", "created_at": "2013-12-20t05:25:02.000z", "percent_complete": 100, "short_code": "kktb4q", "metadata": { }, "state": "published", "recorded_from": "http://singwho.com", "publish_type": null, "formats": [ { "name...
Read more

javascript - How to Hide Date Menu from Datepicker in yii2 -

-
basically in datetimepicker widget date choose first time, how can hide date option widget , filter according time. tried not working want hide remove button [ 'attribute'=>'time', 'label'=>'time', 'format'=>'raw', 'value'=>function($data){ return date('h:i:s',strtotime($data->time)); }, 'filter'=> datetimepicker::widget([ 'model' => $searchmodel, 'attribute' => 'time', 'clientoptions' => [ 'autoclose' => true, 'format' => 'hh:ii', 'showon' =...
Read more