|
|
Web Robot Technology
A web robot (also called web crawler, web spider or knowbot) automatically browses web pages,
follows links and harvests relevant information found online.
- A simple application: downloading historical stock prices.
Web robot: sample output (Yahoo Finance)
Date Open High Low Close Volume
28-Jun-01 52.80 53.26 51.51 52.08 14363200
27-Jun-01 53 53.15 52.15 52.68 10456900
26-Jun-01 52.10 53 51.76 52.85 11302400
25-Jun-01 53.70 53.84 52.70 53 11224600
22-Jun-01 54.05 54.30 52.75 53.10 13431000
21-Jun-01 52.60 55 52.30 55 26473200
20-Jun-01 49.74 52.80 49.74 52.80 16370300
19-Jun-01 50.40 50.90 49.53 49.84 10924200
18-Jun-01 50.75 50.80 49.55 49.80 13073400
15-Jun-01 49.40 50.90 48.18 50.90 27328600
14-Jun-01 51.50 51.53 50.06 50.41 12707400
13-Jun-01 51.95 52.89 51.50 51.55 84009000
12-Jun-01 51.90 52.65 50.60 52.10 10597000
|
What can you do with our web robot, besides downloading stock quotes?
- Copyright infringement tracking:
search the Web continuously - millions of web pages - for breaches of copyright, counterfeit goods, misuse of trademarks. See
a small sample of web pages extracted and scored with the web robot.
- Check broken links and site performance, track reciprocal links that have been discontinued or are
no longer working,
automatically check your ranking on search engines, monitor affiliate programs,
automatically measure load time
and size (in bytes) for all your web pages, find web sites that link to you.
- Automatic Web Search, Database Queries and Data Harvesting:
The web robot can be programmed to automatically query a search engine, then extract
relevant information and create a simple text file on distilled data.
It can also download information from Internet databases: Financial news, historical stock quotes,
trading signals, weather data, lottery results, e-mail directories, etc. The robot will also download binary
files (images).
- Miscellaneous: Automate Category Assignment across entire Web-Site: The web robot
can automatically assign a category to every web page on your site by
mechanized keyword density analysis. It can also
upload information on external Web Sites through
Html forms. For instance, you can perform automatic posting in some message boards.
What our clients are doing with our web robot
-
A major internet company uses the robot to check the
accuracy of product and price information published online.
- A major search engine company has created a large database of keywords and short phrases while browsing
the web with our robot. The database was used to improve search results relevancy and identify
correlated queries.
- A web hosting company in The Netherlands is monitoring trademark violations with our customized robot.
-
Prints.com uses our robot to download their auctions from
Ebay
and upload them on their website.
They also download product information
(price, title, picture) on
Amazon with another customized version of the robot.
- The
National Radar
Facility of University of Colorado downloads
webcam images automatically generated on their server, every day.
- We are using our web robot to create a domain name database and assign rankings
to each domain.
|
|
