Losing my mind - can't figure out how to get data on an individual page via ajax next button?

How to?
#1

Can't seem for the life of me to get the data of each person's entry scraped across both 2 pages of entries. It seems to work backwards and only go into items on page 2 and not check the entries on page 1.

https://experts.shopify.com/services/visual-content-and-branding/create-3d-models-ar

{"_id":"experts_3d_models_shopify","startUrl":["https://experts.shopify.com/services/visual-content-and-branding/create-3d-models-ar"],"selectors":[{"id":"person-wrapper","type":"SelectorElementClick","parentSelectors":["_root","person_wrapper"],"selector":"div._3PwHx","multiple":true,"delay":"2000","clickElementSelector":"button.Polaris-Pagination__NextButton_13336","clickType":"clickMore","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueHTML"},{"id":"person link","type":"SelectorLink","parentSelectors":["_root"],"selector":"._2nyp3 a","multiple":true,"delay":0},{"id":"expert since","type":"SelectorText","parentSelectors":["person link"],"selector":"div.Polaris-Stack__Item_yiyol:nth-of-type(1) p span.Polaris-TextStyle--variationSubdued_1segu","multiple":false,"regex":"","delay":0}]}

#2

Hi @theakx

You can easily use the page range definition for the starting URL.
[1-2]

https://experts.shopify.com/services/visual-content-and-branding/create-3d-models-ar?page=[1-2]

Sitemap example:
{"_id":"experts_3d_models_shopify","startUrl":["https://experts.shopify.com/services/visual-content-and-branding/create-3d-models-ar?page=[1-2]"],"selectors":[{"id":"person link","type":"SelectorLink","parentSelectors":["_root"],"selector":"div[aria-live=\"polite\"] > div:has(h2) a","multiple":true,"delay":0},{"id":"expert since","type":"SelectorText","parentSelectors":["person link"],"selector":"div.Polaris-Stack__Item_yiyol:nth-of-type(1) p span.Polaris-TextStyle--variationSubdued_1segu","multiple":false,"regex":"","delay":0}]}

#3

Thanks for the suggestion....but there are other categories I'm wanting to do. I'm just wanting to solve this smaller dataset issue so I can scrape the entirety of the directory.

#4

@theakx Sure. So add multiple starting URLs for each of the categories with a page range definition in it.

#5

Wouldn't that require a sitemap for each? There's something like 6 parent categories with probably on average up to 8 child categories each. Unless I'm missing something I'd have to make like 48 sitemaps?

#6

Does anyone have any idea on how this can be done?

#7

@theakx I would just use each of the subcategory links as a starting URL with a page range definition in it.