sql - regexp in PySpark -

-

i trying reproduce results of django orm query in pyspark:

social_filter = '(facebook|flipboard|linkedin|pinterest|reddit|twitter)' collection.objects.filter(social__iregex=social_filter)

my main problem should case insensitive.

i have tried this:

social_filter = "social ilike 'facebook' or social ilike 'flipboard' or social ilike 'linkedin' or social ilike 'pinterest' or social ilike 'reddit' or social ilike 'twitter'" df = sessions.filter(social_filter)

which result in following error:

py4jjavaerror: error occurred while calling o31.filter. : java.lang.runtimeexception: [1.22] failure: end of input expected  social ilike 'facebook' or social ilike 'flipboard' or social ilike 'linkedin' or social ilike 'pinterest' or social ilike 'reddit' or social ilike 'twitter'

and following expression:

social_filter = "social  ~* (facebook|flipboard|linkedin|pinterest|reddit|twitter)" df = sessions.filter(social_filter)

crashes this:

py4jjavaerror: error occurred while calling o31.filter. : java.lang.runtimeexception: [1.17] failure: identifier expected  social  ~* (facebook|flipboard|linkedin|pinterest|reddit|twitter)        ^     @ scala.sys.package$.error(package.scala:27)     @ org.apache.spark.sql.catalyst.sqlparser.parseexpression(sqlparser.scala:45)     @ org.apache.spark.sql.dataframe.filter(dataframe.scala:652)     @ sun.reflect.nativemethodaccessorimpl.invoke0(native method)

please, help!

how following:

>>> rdd = sc.parallelize([row(name='bob', social='twitter'),                            row(name='steve', social='facebook')]) >>> df = sqlcontext.createdataframe(rdd) >>> df.where("lower(social) 'twitter'").collect() [row(name=u'bob', social=u'twitter')]

you can of social networks want if need actual regular expression. otherwise, if match exact, can this:

>>> df.where("lower(social) in ('twitter', 'facebook')").collect() [row(name=u'bob', social=u'twitter'), row(name=u'steve', social=u'facebook')]

Comments

Post a Comment

Popular posts from this blog

Android : Making Listview full screen -

-
<linearlayout xmlns:android="http://schemas.android.com/apk/res/android" xmlns:tools="http://schemas.android.com/tools" android:layout_width="match_parent" android:layout_height="match_parent" android:orientation="vertical" tools:context=".mainactivity"> <edittext android:layout_margintop="5dp" android:layout_width="match_parent" android:layout_height="50dp" android:background="@drawable/location_edittext" android:id="@+id/locationeditext" android:textsize="15sp" android:textcolor="#999" android:paddingleft="15dp" /> <com.example.ravi_gupta.slider.viewpagercustomduration android:layout_width="wrap_content" android:layout_height="0dp" android:id="@+id/viewpager" android:scrollbarstyle="outsideoverlay" android:layout_weight="0.5...
Read more

javascript - Parse JSON from the body of the POST -

-
i'm using cameratag.com record video on website. service can send json response server can save data database. don't know how proceed here: the documentation states please note json repsonse passed in body of post , not form parameters. here example of how parse response in ruby: data = json.parse(request.raw_post) video_uuid = data["uuid"] but don't know how data on javascript. i'm not experienced in js please bare me :) post sample: { "uuid": "00012710-4b65-0131-ffe6-22000a499ea4", "camera_uuid": "98373a20-79ee-0130-3e42-1231390fcc11", "created_at": "2013-12-20t05:25:02.000z", "percent_complete": 100, "short_code": "kktb4q", "metadata": { }, "state": "published", "recorded_from": "http://singwho.com", "publish_type": null, "formats": [ { "name...
Read more

javascript - How to Hide Date Menu from Datepicker in yii2 -

-
basically in datetimepicker widget date choose first time, how can hide date option widget , filter according time. tried not working want hide remove button [ 'attribute'=>'time', 'label'=>'time', 'format'=>'raw', 'value'=>function($data){ return date('h:i:s',strtotime($data->time)); }, 'filter'=> datetimepicker::widget([ 'model' => $searchmodel, 'attribute' => 'time', 'clientoptions' => [ 'autoclose' => true, 'format' => 'hh:ii', 'showon' =...
Read more