Retrieving data from AJAX generated website

ronstudio

May I ask if anyone ever tried to retrieve data from website which is created by AJAX in website? What kind of tools do you use?

Thanks in advance~

Jackass_TMxCK

Do you know or even hear of "Same Origin Policy"?


If you solve this restriction, then it has no difference other than a normal website with RESTful API feeding data

ronstudio

回覆 2# Jackass_TMxCK

Thanks for the help! I'll do further research regarding this area!

Jackass_TMxCK

回覆  Jackass_TMxCK

Thanks for the help! I'll do further research regarding this area!
ronstudio 發表於 1/4/2015 08:18 AM

    I think of something, server side cURL has no limitations on some origin policy

Check out PHP simple proxy

ronstudio

Thanks, so far I notice most solution people using are loading some kind of browser module in order to simulate the web browsing in order to take the data.  At least in Perl,  there is a module which simulate firefox to achieve this.

This is doable,  but really takes more effort and need furtfurther study

chi251155

回覆 1# ronstudio


    curl or wget

ronstudio

The problem of using simple wget is the content of the page are generated with AJAX javascript. So when I use the simple wget, it gets the html structure but nothing of the content which I need to parse.

chi251155

回覆 7# ronstudio


    use wget and curl to get the message of ajax. you know ajax is a method of communication right? you can easily find the url of the data source, it is easier than crawling the web page, as the contents are always formatted in json or xml.

Jackass_TMxCK

回覆  ronstudio


    use wget and curl to get the message of ajax. you know ajax is a method of co ...
chi251155 發表於 5/4/2015 12:06 AM

    Same Origin Policy....