AnalyticBridge

Data Intelligence, Business Analytics

Simple Perl code to visit web pages with a web robot

Scrape web pages, measure length, time to download, etc. Avoid common pitfalls by using time out (2 seconds) and max size (64 KB) parameters. URL's to visit are stored in an input text file.

See attachment.

Attachments:

robotCF2.pl, 1 KB

Replies to This Discussion

Permalink Reply by Ross Farrelly on July 5, 2010 at 3:57pm

Hi - I'm hoping someone can help me with getting this bot to run. I'm new to perl so please excuse me if I'm missing something obvious. This is what I've done so far:
1) Installed Strawberry perl
2) Saved robotCF2.pl to c:\strawberry
3) created textfile called URLlist.txt in c:\strawberry containing one url: www.smh.com.au
4) At the commsnd prompt I've changed director to c:\strawberry then submitted the command: perl robotCF2.pl
5) The program runs and creates a file called robotout.txt in c:\strawberry but the file is empty.

Any assistance to fix this would be greatly appreciated.

Cheers

Permalink Reply by Ross Farrelly on August 10, 2010 at 4:46am

Don't worry - I now have this working

RSS

Welcome to
AnalyticBridge

Sign Up
or Sign In

Or sign in with:

Top Content

1	How to develop churn prediction model for telecom company?
2	Map of Data Science University Programs
3	Use PRESS, not R squared to judge predictive power of regression
4	Text mining: alternative to Wordflex?
5	Audience expansion
6	Simple solutions to make videos with R
7	Is Open Source Really Free?
8	Understanding the Kalman Filter Application in Economic Time Series Data
9	Three classes of metrics: centrality, volatility, and bumpiness
10	Identifying the number of clusters: finally a solution