Pagnation not waiting for Element click

How to?
#1

Hi - I am trying to scrape some details on endangered species. I want the Element Click Selector to select "Europe" and "Critically Endangered", then the Paginator to click the "show more" button, and then the scraping.

It is working fine apart from one detail: The Element Click Selector only clicks on one button (just "Critically Endangered", but NOT "Europe"), then the pagination kicks in. That way I am ending up with wayyy too much data. How do I get it to fist click all the buttons before pagination?

I tried to fiddle with delay times but it hasn't done much so far... though I might just not have found the perfect combination yet?

Url: https://www.iucnredlist.org/search

Sitemap:
{"_id":"RedList","startUrl":["https://www.iucnredlist.org/search"],"selectors":[{"clickActionType":"real","clickElementSelector":"label[for='landRegions-912D59CDF1D3F551FAE21F6F062258F'], label[for='redListCategory-cr']","clickElementUniquenessType":"uniqueCSSSelector","clickType":"clickOnce","delay":3000,"discardInitialElements":"discard-when-click-element-exists","id":"Link_all_species","multiple":true,"parentSelectors":["_root"],"selector":"div.layout-page","type":"SelectorElementClick"},{"id":"PaginatorNew","paginationType":"clickMore","parentSelectors":["Link_all_species","PaginatorNew"],"selector":"a.section__link-out","type":"SelectorPagination"},{"id":"Links_species","linkType":"linkFromHref","multiple":true,"parentSelectors":["PaginatorNew"],"selector":".section div .cards a.link--faux","type":"SelectorLink"},{"id":"Latin","multiple":false,"parentSelectors":["Links_species"],"regex":"","selector":".text-body p span i:nth-of-type(1)","type":"SelectorText"},{"id":"Extinction_Level","multiple":false,"parentSelectors":["Links_species"],"regex":"","selector":".layout-card--split__minor p a strong","type":"SelectorText"},{"id":"Common","multiple":false,"parentSelectors":["Links_species"],"regex":"","selector":".layout-container div h1","type":"SelectorText"},{"id":"Assessment_scope","multiple":false,"parentSelectors":["Links_species"],"regex":"","selector":"div.panel:nth-of-type(2) p.panel__data--std","type":"SelectorText"}]}

#2

Hi,

Try placing the pagination selector below the click selector and not as a child, like this:

{"_id":"RedList","startUrl":["https://www.iucnredlist.org/search"],"selectors":[{"id":"Link_all_species","parentSelectors":["_root"],"type":"SelectorElementClick","clickActionType":"real","clickElementSelector":"label[for='landRegions-912D59CDF1D3F551FAE21F6F062258F'], label[for='redListCategory-cr']","clickElementUniquenessType":"uniqueCSSSelector","clickType":"clickOnce","delay":3000,"discardInitialElements":"discard-when-click-element-exists","multiple":true,"selector":"div.layout-page"},{"id":"PaginatorNew","parentSelectors":["_root","PaginatorNew"],"paginationType":"clickMore","type":"SelectorPagination","selector":"a.section__link-out"},{"id":"Links_species","parentSelectors":["PaginatorNew"],"type":"SelectorLink","selector":".section div .cards a.link--faux","multiple":true,"linkType":"linkFromHref"},{"id":"Latin","parentSelectors":["Links_species"],"type":"SelectorText","selector":".text-body p span i:nth-of-type(1)","multiple":false,"regex":""},{"id":"Extinction_Level","parentSelectors":["Links_species"],"type":"SelectorText","selector":".layout-card--split__minor p a strong","multiple":false,"regex":""},{"id":"Common","parentSelectors":["Links_species"],"type":"SelectorText","selector":".layout-container div h1","multiple":false,"regex":""},{"id":"Assessment_scope","parentSelectors":["Links_species"],"type":"SelectorText","selector":"div.panel:nth-of-type(2) p.panel__data--std","multiple":false,"regex":""}]}
1 Like
#3

I think this is working! Thank you so much :slight_smile:

1 Like