First page scrapes, but fails when pagination added

#1

Greetings!
On this page: https://www.centris.ca/en/triplexes~for-sale~montreal-island?view=Thumbnail I can open and scrape each "Summary" entry (scraping for the price and the gross revenue). BUT if I add an element click selector to move to the next page nothing gets scrapped, not even the first page. I'm lost, please help :smile:

{"_id":"roi","startUrl":["https://www.centris.ca/en/triplexes~for-sale~montreal-island?view=List"],"selectors":[{"id":"next","type":"SelectorElementClick","selector":"div.right-section li.next a","parentSelectors":["_root","next"],"multiple":true,"delay":0,"clickElementSelector":"div.right-section li.next a","clickType":"clickMore","discardInitialElements":false,"clickElementUniquenessType":"uniqueHTMLText"},{"id":"Summary","type":"SelectorLink","selector":"a.btn","parentSelectors":["_root","next"],"multiple":true,"delay":0},{"id":"price","type":"SelectorText","selector":"span#BuyPrice","parentSelectors":["Summary"],"multiple":false,"regex":"","delay":0},{"id":"rent","type":"SelectorText","selector":"tr:contains('Potential gross revenue') span","parentSelectors":["Summary"],"multiple":false,"regex":"","delay":0}]}

#2

Hi there!

Try this one:
{"_id":"centris","startUrl":["https://www.centris.ca/en/triplexes~for-sale~montreal-island?view=Thumbnail"],"selectors":[{"id":"next","type":"SelectorElementClick","selector":"div.thumbnail-content div.row","parentSelectors":["_root"],"multiple":true,"delay":"2000","clickElementSelector":"li[class=next] a:not(.disabled)","clickType":"clickMore","discardInitialElements":false,"clickElementUniquenessType":"uniqueText"},{"id":"price","type":"SelectorText","selector":"span#BuyPrice","parentSelectors":["links"],"multiple":false,"regex":"","delay":0},{"id":"rent","type":"SelectorText","selector":"tr:contains('Potential gross revenue') span","parentSelectors":["links"],"multiple":false,"regex":"","delay":0},{"id":"links","type":"SelectorLink","selector":"a.btn","parentSelectors":["next"],"multiple":false,"delay":0}]}

When adding an Element Click, you pick pagination into Click selector, and Page contents that needs to be scraped, into Selector itself. You've selected pagination for both of the selectors.

I've also set it to Thumbnail view, so it works as expected now.

#3

Nice work @iconoclast. How did you come up with that Click Selector. I don't think I've seen you use this particular CSS selector "li[class=next] a:not(.disabled)" Actually it looks looks more like XPATH. Even when I went to copy the selector it looked more like
#divWrapperPager > ul > li.next > a

#4

It's pure CSS selectors :slight_smile:
[class=next] is an CSS attribute selector. It's easier to pin particular element using it's unique class or ID.
:not(.disabled) is a CSS selector as well. It will select an element that does not contain '.disabled' in it's class.
To be honest, it will stop properly even if I wouldn't put ':not(.disabled)' in the first place. I used it so WebScraper won't have any button to click if it's disabled on a website (last page).

#5

Ah, I see. Why use that particular CSS selector this time? I've not seen you use it before. You're normally all about the nth-last-of or last-child etc.

I guess what I'm trying to learn is what were the signs that lead you to determine using Class or ID was the best options here?

#6

I usually go through next scenario:

  1. If WebScraper picks element correctly, it's ok
  2. If not, I try to narrow it down using either Inspect Tool or selecting it using CSS selectors and CSS attribute selectors.

And lately I just use Inspect tool just to look at class to either pick it or avoid it.