How to set up individual child pages on pages with more button (click selector)

How to?
#1

Hi, my goal is this:

  1. Go to a main page
  2. Click on more button infinitely to see all items available (this does not generate unique URLs)
  3. Click on URL of items on the page
  4. From the individual URLs, scrape fields from this

I experimented with this, but it does not work for me as I don’t want to extract items on this page, I want to go to individual URLs (click through), then extract details in the individual page.

https://webscraper.io/how-to-video/element-click-pagination-next

When I try to build a child page from the elementary click, it doesn’t recognize the parent.

Help please!

#2

@scrapeman Hi! Would you be able to provide the starting URL as well as the data points you are looking to extract?

#3

Hi Viesturs, I have several separate pages with this need (on click more, or on scroll down).

As an example,

Here is a main page

https://appsumo.com/browse/

And clicking through the individual pages, to extract details like title, subtitle, rating, plans and features.

Thanks!

#4

@scrapeman Hi. I checked out the site and experimented with different delays and amount of clicks but it seems that the issue lies in the website it self cause sometimes i got only till 8 th page and later up to 100 page but it failed to load to new elements or the show more button itself, the only solution would be to either zoom out the page to the maximum or use extremely high delay and request interval values, like 8000 and more but that's definitely the websites fault.

My sitemap example:

{"_id":"appsumo-com","startUrl":["https://appsumo.com/browse"],"selectors":[{"id":"wrapper-click-more","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"div.appsumo-deal-tile","multiple":true,"delay":"2000","clickElementSelector":"a.btn:contains(\"Show More Results\")","clickType":"clickMore","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueCSSSelector"},{"id":"link","type":"SelectorLink","parentSelectors":["click-moooore"],"selector":"a.stretched-link","multiple":false,"delay":0},{"id":"card","type":"SelectorElement","parentSelectors":["link"],"selector":"body:has(h2.sumo-product-name)","multiple":true,"delay":0},{"id":"title","type":"SelectorText","parentSelectors":["card"],"selector":"h2.sumo-product-name","multiple":false,"regex":"","delay":0},{"id":"description","type":"SelectorText","parentSelectors":["card"],"selector":"div.pt-4","multiple":false,"regex":"","delay":0},{"id":"price","type":"SelectorText","parentSelectors":["card"],"selector":".sumo-product-sticky-inner-desktop div.sumo-price","multiple":false,"regex":"","delay":0},{"id":"scroll","type":"SelectorElementScroll","parentSelectors":["_root"],"selector":"div.appsumo-deal-tile","multiple":true,"delay":2000},{"id":"click-moooore","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"div.appsumo-deal-tile","multiple":true,"delay":2000,"clickElementSelector":"a.btn:contains(\"Show More Results\")","clickType":"clickMore","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueCSSSelector"}]}

#5

thank you @ViestursWS I appreciate it

1 Like