Views all pages, but only getting first page scraped

How to?
#1

The webscraper goes through all the pages by following the NEXT css link pagination. But only returns the first page of data. I put a high delay of 6000 on:
Request interval (ms) 6000
Page load delay (ms) 6000

After reading some of the other folks' questions, I wonder if my syntax is wrong? The URL changes like this:
https://downtownorlandokw.yourkwoffice.com/our-agents?p=1
https://downtownorlandokw.yourkwoffice.com/our-agents?p=2

I know it's forcing a reload as I was getting an error before on my pagination, but after switching to this method, it goes through all the pages, but doesn't scrape past the first page. :thinking:

Sitemap:
{"_id":"KW-DowntownOrlando","startUrl":["Homes for Sale | KW"}]}

#2

Hi,

The issue is probably due to the data selector not being child selectors of the pagination. Please try the sitemap below:

{"_id":"yourkwoffice-com","startUrl":["https://downtownorlandokw.yourkwoffice.com/our-agents?p=1"],"selectors":[{"id":"pagination","paginationType":"auto","parentSelectors":["_root","pagination"],"selector":"a.KwPagination-next","type":"SelectorPagination"},{"id":"wrapper","multiple":true,"parentSelectors":["pagination"],"selector":"div.AvatarCard","type":"SelectorElement"},{"id":"name","multiple":false,"parentSelectors":["wrapper"],"regex":"","selector":"div.AvatarCard-bio-full-name","type":"SelectorText"},{"id":"link","linkType":"linkFromHref","multiple":false,"parentSelectors":["wrapper"],"selector":"a.AvatarCard-link","type":"SelectorLink"},{"id":"phone","multiple":false,"parentSelectors":["wrapper"],"regex":"","selector":"kw-intl-phone-format","type":"SelectorText"},{"id":"e-mail","multiple":false,"parentSelectors":["wrapper"],"regex":"","selector":"a[data-link-type='email']","type":"SelectorText"}]}
1 Like
#3

OMG! That worked like a charm! thanks so much! I wasn't aware the selectors had to be nested under the pagination. Thanks for clarifying and replying so dang fast.

1 Like