News Feed
Sections




News Archive
Looking for more information on how to do PHP the right way? Check out PHP: The Right Way

Juozas Kaziukenas' Blog:
Scraping login requiring websites with cURL
February 24, 2009 @ 08:44:43

Several sites have areas that have content protected behind a login making them difficult to pull into a script. Juozas Kaziukenas has created an option to help you past this hurdle - a PHP class (that uses cURL) that can POST the login data to the script and pull back the session ID.

But how you are going to do all this work with cookies and session id? Luckily, PHP has cURL extension which simplifies connecting to remote addresses, using cookies, staying in one session, POSTing data, etc. It's really powerful library, which basically allows you to use all HTTP headers functionality. For secure pages crawling, I've created very simple Secure_Crawler class.

The class uses the built-in cURL functionality to send the POST information (in this case the username and password, but it can be easily changed for whatever the form requires) and provides a get() method to use for fetching other pages once you're connected.

0 comments voice your opinion now!
login require scrape curl secure crawler tutorial username password


blog comments powered by Disqus

Similar Posts

Jordi Boggiano: Authentication management in Composer

HowTo Forge: Installing And Working With Xoops Under Ubuntu 6.10

Mike Purcell's Blog: Symfony - sfGuardPlugin - Use Email Instead of Username

Maarten Balliauw's Blog: Reuse Excel business logic with PHPExcel

Eirik Hoem's Blog: Prado Framework 3.1.2 released


Community Events





Don't see your event here?
Let us know!


list language install community experience framework podcast series release code refactor opinion unittest introduction developer configure threedevsandamaybe interview testing laravel

All content copyright, 2014 PHPDeveloper.org :: info@phpdeveloper.org - Powered by the Solar PHP Framework