Pagination where item page is accessed

How to?
#1

Hi Everyone,

Having an issue scraping the site below.
Looking to scrape information for each company(product) but am having an issue with the pagination.
While the scraper seems to load more companies/products it only will scrape 12 then move onto the next category.

Can anyone assist in helping with this one?

Thanks,
Ed

Url: https://www.truelocal.com.au/business-type

Sitemap:
{"_id":"true_local","startUrl":["https://www.truelocal.com.au/business-type"],"selectors":[{"id":"categories","type":"SelectorElementClick","selector":"div.browse-categories-list","parentSelectors":["_root"],"multiple":true,"delay":0,"clickElementSelector":"li.browse-category","clickType":"clickOnce","discardInitialElements":false,"clickElementUniquenessType":"uniqueText"},{"id":"subcategories","type":"SelectorLink","selector":"a.browse-subcategory-link","parentSelectors":["categories"],"multiple":true,"delay":0},{"id":"contactPage","type":"SelectorLink","selector":"li.ng-scope:nth-of-type(n+2) a.item-title","parentSelectors":["subcategories"],"multiple":true,"delay":0},{"id":"Company Name","type":"SelectorText","selector":"h2.headline","parentSelectors":["contactPage"],"multiple":false,"regex":"","delay":0},{"id":"Website","type":"SelectorPopupLink","selector":"bdp-details-contact-website.ng-isolate-scope span.text-frame span","parentSelectors":["contactPage"],"multiple":false,"delay":0},{"id":"Phone2","type":"SelectorText","selector":"span.iconed-text:nth-of-type(2) span.ng-binding","parentSelectors":["contactPage"],"multiple":false,"regex":"","delay":0},{"id":"Phone3","type":"SelectorText","selector":"span.iconed-text:nth-of-type(3) span.ng-binding","parentSelectors":["contactPage"],"multiple":false,"regex":"","delay":0},{"id":"Address","type":"SelectorText","selector":"div.contact-holder","parentSelectors":["contactPage"],"multiple":false,"regex":"","delay":0},{"id":"Load More Results","type":"SelectorElementClick","selector":"bs-load-more.ng-isolate-scope button.btn","parentSelectors":["subcategories"],"multiple":true,"delay":0,"clickElementSelector":"bs-load-more.ng-isolate-scope button.btn","clickType":"clickMore","discardInitialElements":false,"clickElementUniquenessType":"uniqueText"},{"id":"Phone1","type":"SelectorText","selector":"bdp-details-contact-phone.ng-isolate-scope span.iconed-text:nth-of-type(1) span.ng-binding","parentSelectors":["contactPage"],"multiple":false,"regex":"","delay":0},{"id":"ABN","type":"SelectorText","selector":"bdp-details-abn-acn.ng-isolate-scope div.ng-scope:nth-of-type(1) span.text-frame","parentSelectors":["contactPage"],"multiple":false,"regex":"","delay":0}]}

#2

Hey,
Still having this issue.
If anyone can offer any ideas it would be appreciated.

Thanks,
Ed

#3

Hi!

Had to use VPN service otherwise i get blocked lol.

It seems you've made a mistake in your Element Click selector.
Here's an explanation:

image
image.png872×261 9.26 KB

The correct selector for a wrapper is li.ng-scope:nth-of-type(n+2) > div.list-item, try it out.

The problem you will notice is that it won't go beyond page 11 under any circumstances (perhaps if your RAM is bigger than mine, so you could get further little more), as browser window starts to lag if more results getting loaded.

I would recommend you to narrow the results to a particular city, for example. Or use an extension to inject Java Script (Tampermonkey, for example), that will unload results that have already been scraped.

P.S. set Click Selector delay to 3000 ms, or it will stop even earlier. Or set it to 10000 ms, you will get a little further but will eventually stop anyway.