asp.net - Scraping using Html Agility Package -

-

i trying scrape data news article using htmlagilitypackage link follows http://www.ndtv.com/india-news/vyapam-scam-documents-show-chief-minister-shivraj-chouhan-delayed-probe-780528

i have written following code below extract comments in articles reason variable atags returning null value

code:

var gethtmlweb = new htmlweb();         var document = gethtmlweb.load(txtinputurl.text);         var atags =    document.documentnode.selectnodes("//div[@class='com_user_text']");         int counter = 1;         if (atags != null)         {             foreach (var atag in atags)             {                 lbloutput.text += lbloutput.text + ". " + atag.innerhtml + "\t" + "<br />";                 counter++;             }         }

i have used xpath still same result //div[@class='newcomment_list']/ul/li/div[@class='headerwrap']/div[@class='com_user_text'] please me correct xpath extract comments searched on net no solution.

do 'view source' on page , search com_user_text. user comments don't appear @ all. loaded via javascript after page loaded. when load page content via gethtmlweb.load(), don't user comments.

as this answer says, html agility not tool capable of emulating browser , running javascript. instead, need watin "allows programmatic access web pages through given browser engine , load full document."


Comments

Post a Comment

Popular posts from this blog

Android : Making Listview full screen -

-
<linearlayout xmlns:android="http://schemas.android.com/apk/res/android" xmlns:tools="http://schemas.android.com/tools" android:layout_width="match_parent" android:layout_height="match_parent" android:orientation="vertical" tools:context=".mainactivity"> <edittext android:layout_margintop="5dp" android:layout_width="match_parent" android:layout_height="50dp" android:background="@drawable/location_edittext" android:id="@+id/locationeditext" android:textsize="15sp" android:textcolor="#999" android:paddingleft="15dp" /> <com.example.ravi_gupta.slider.viewpagercustomduration android:layout_width="wrap_content" android:layout_height="0dp" android:id="@+id/viewpager" android:scrollbarstyle="outsideoverlay" android:layout_weight="0.5&quo
Read more

javascript - Parse JSON from the body of the POST -

-
i'm using cameratag.com record video on website. service can send json response server can save data database. don't know how proceed here: the documentation states please note json repsonse passed in body of post , not form parameters. here example of how parse response in ruby: data = json.parse(request.raw_post) video_uuid = data["uuid"] but don't know how data on javascript. i'm not experienced in js please bare me :) post sample: { "uuid": "00012710-4b65-0131-ffe6-22000a499ea4", "camera_uuid": "98373a20-79ee-0130-3e42-1231390fcc11", "created_at": "2013-12-20t05:25:02.000z", "percent_complete": 100, "short_code": "kktb4q", "metadata": { }, "state": "published", "recorded_from": "http://singwho.com", "publish_type": null, "formats": [ { "name&qu
Read more

javascript - Chrome Extension: Interacting with iframe embedded within popup -

-
i writing chrome extension needs interact urls user not have open. therefore using hidden iframes embedded within popup, , attempting click button within iframe. however, receiving same origin policy error. know possible extension interact iframes of different domain via content scripts when iframe on tab user has open, not sure if possible use content scripts interact iframes directly in popup. here code: manifest.json "content_scripts": [ { "js": [ "bin/jquery.min.js", "interaction.js" ], "all_frames": true, "run_at": "document_start", "matches": [ "http://*/*", "https://*/*" ] }], "permissions": [ "activetab", "tabs", "http://*/", "https://*/" ], interaction.js $(document).ready(function() { $('div#iframes').append("<iframe id='shop' src=
Read more