Price crawler is used to populate prices for merchant products.

Each merchant registered can choose for his website to be spidered for price updation in his merchant area. The crawler spiders' merchant website in search of product and its price. It searches $ sign (or any currency sign) and when it finds a dollar sign it then searches for the product name before it (it works through tables, javascripts and other tags). If it finds product name prior to the product price it updates the price of that product for that merchant in the price-comparison database.

Once updation is done, crawler moves ahead on the same page looking for other products and there prices and continue on updating prices for that merchant. Once that page is finished, crawler then goes into the links on that page for their crawling. Therefore, page by page, whole website is spidered for that merchant.

Once a merchant's whole website is spidered, it moves ahead to spider another merchant's website.

Note: Crawling is resource hungry task and requires time, cpu and other resources of the server.

Click on Manage Merchants > Current Merchants In the merchant listing click on the Spidering URLs icon (see image below).

This will display a listing of urls for the merchant. You can use the form below the listing to add a new url, or you can edit a previous url.

1. URL
It is the url from where the crawler will start crawling for the prices of this merchant.

2. Spider URLs
It should contain a phrase which should be present in the subsiquent urls to be spidered. Meaning, when you will go to the above url, you will notice there are links to the other categories and products on that page as well as links to about us and privacy policy. You dont want your crawler to waste time in unnecessary pages so you instruct it to spider only those urls which contains that particular phrase.

This will ensure that crawler will only go into pages where price info will be present. Therefore saving time and resources. If left blank it will go into all links like about us and others.

3. Process URLs
It should contain a phrase which is present in the urls of the product price page. So that only those pages are processed for price extraction.

Now click on the Run Price Spider to start spidering.
You can view log of the actions perform once task is finished by the spider.