I can't scrape this (with URL) (no reload pagination + different elements)

How to?
#1

Hi, so I'm really stuck on this one, hope someone can help.
I'm trying to get all the data per month per destination from this type of page : https://www.surfguide.com/nova-scotia (sitemap: https://www.surfguide.com/sitemap) but I'm stuck with the pagination.

The page doesn't reload when clicking on a month so I'm using a 'Element Click' selector but the 'Click more' of the pagination doesn't work. And the .div I'm trying to get data from seem to be different from each other.
So I'm getting either no data back or all the data but totally messed up.

Let me know if you find the solution.

Thanks

#2

Hello

I can't see the button "click more" on this page.

Send us your sitemap please.

#3

I was referring to the arrow icons left and right of the Months. I guess they're supposed to work the same but they just don't anything.

#4

I had to select the different months manually to see the data per month
With my version of Chrome the Left & Right arrow icons dont works.
I think you don't have links behind these arrows so Web scraper can't use them

#5

ok so while redoing the sitemap to send it to you I might have found the solution... (I redid it 3 times already... don't know what I missed before :laughing: )

#6

I am not sure to understand if your sitemap is now ok.
If so, could you post it because it's always interesting to read how to turn around difficulties with this marvelous extension.

#7

I think yes it's ok.
Ran a sample, I'm getting all data/month/destination.
here is the site map:
{"_id":"surfguide2","startUrl":["https://www.surfguide.com/sitemap"],"selectors":[{"id":"Page","type":"SelectorLink","selector":"li:nth-of-type(2) > ul > li > a","parentSelectors":["_root"],"multiple":true,"delay":0},{"id":"Name","type":"SelectorText","selector":"h1.ng-binding","parentSelectors":["Page"],"multiple":false,"regex":"","delay":0},{"id":"month","type":"SelectorElementClick","selector":"div.container div.ng-scope:nth-of-type(3) div.content-area","parentSelectors":["Page"],"multiple":true,"delay":0,"clickElementSelector":"ul.slides a.ng-binding","clickType":"clickOnce","discardInitialElements":false,"clickElementUniquenessType":"uniqueText"},{"id":"surf","type":"SelectorText","selector":"div.col-md-6 div.col-sm-6:nth-of-type(1) dl:nth-of-type(1) dd.ng-binding","parentSelectors":["month"],"multiple":false,"regex":"","delay":0},{"id":"Size","type":"SelectorText","selector":"div.col-md-6 div.col-sm-6:nth-of-type(2) dl:nth-of-type(1) dd.ng-binding","parentSelectors":["month"],"multiple":false,"regex":"","delay":0},{"id":"water temp","type":"SelectorText","selector":"div.col-md-6 div.col-sm-6:nth-of-type(1) dl:nth-of-type(2) dd.ng-binding","parentSelectors":["month"],"multiple":false,"regex":"","delay":0},{"id":"swell direction","type":"SelectorText","selector":"div.col-md-6 div.col-sm-6:nth-of-type(2) dl:nth-of-type(2) dd.ng-binding","parentSelectors":["month"],"multiple":false,"regex":"","delay":0},{"id":"consistency","type":"SelectorText","selector":"div.col-md-6 dl:nth-of-type(3) dd.ng-binding","parentSelectors":["month"],"multiple":false,"regex":"","delay":0},{"id":"difficulty","type":"SelectorText","selector":"div.col-md-6 dd.ng-scope","parentSelectors":["month"],"multiple":false,"regex":"","delay":0},{"id":"crowd","type":"SelectorText","selector":"div.specs div.col-sm-6 dl:nth-of-type(4) dd.ng-binding","parentSelectors":["month"],"multiple":false,"regex":"","delay":0},{"id":"rain","type":"SelectorText","selector":"div.col-sm-6:nth-of-type(2) div.specs dl:nth-of-type(1) dd.ng-binding","parentSelectors":["month"],"multiple":false,"regex":"","delay":0},{"id":"rainy days","type":"SelectorText","selector":"div.col-sm-6:nth-of-type(2) div.specs dl:nth-of-type(2) dd.ng-binding","parentSelectors":["month"],"multiple":false,"regex":"","delay":0},{"id":"minimum air temperature","type":"SelectorText","selector":"div.col-sm-6:nth-of-type(2) div.specs dl:nth-of-type(3) dd.ng-binding","parentSelectors":["month"],"multiple":false,"regex":"","delay":0},{"id":"maximum air temperature","type":"SelectorText","selector":"div.col-sm-6 div.specs dl:nth-of-type(4) dd.ng-binding","parentSelectors":["month"],"multiple":false,"regex":"","delay":0},{"id":"wind force","type":"SelectorText","selector":"div.col-sm-6:nth-of-type(3) div.specs dl:nth-of-type(1) dd.ng-binding","parentSelectors":["month"],"multiple":false,"regex":"","delay":0},{"id":"wind direction","type":"SelectorText","selector":"div.col-sm-6:nth-of-type(3) div.specs dl:nth-of-type(2) dd.ng-binding","parentSelectors":["month"],"multiple":false,"regex":"","delay":0},{"id":"wind consistency","type":"SelectorText","selector":"div.col-sm-6:nth-of-type(3) div.specs dl:nth-of-type(3) dd.ng-binding","parentSelectors":["month"],"multiple":false,"regex":"","delay":0},{"id":"month_name","type":"SelectorText","selector":"li.ng-scope.month-active a.ng-binding","parentSelectors":["month"],"multiple":false,"regex":"","delay":0}]}

#8

Thank's a lot, it's beautiful.

Have a good surf season !

#9

ahah thanks!

I'm really amazed at what we can do with this extension