I have a page with a "click for more results" scenario. The page would have effectively infinite results, so I can't use the click more
type in SelectorElementClick
. I would like to have the scraper "click more", then grab the new links, then "click more" again and grab the new links, ad infinitum. I tried various things. Below, I have the link scraper as a child of the SelectorElementClick
, with type set to "click once" and multiple
set to true. This seems to only click more once, grab the links and end.
Thank you.
Url: https://www.fragrantica.com/search/?spol=male~unisex
Sitemap:
{
"_id": "fragrantica_from_rating",
"startUrl": [
"https://www.fragrantica.com/search/?spol=male~unisex"
],
"selectors": [
{
"id": "select_rating",
"type": "SelectorElementClick",
"parentSelectors": [
"_root"
],
"selector": "div.off-canvas-content",
"multiple": false,
"delay": "3000",
"clickElementSelector": "#offCanvasLeftOverlap1 > div > div > div:nth-child(2) > div.ais-Panel-body > p > div > select > option:nth-child(2)",
"clickType": "clickOnce",
"discardInitialElements": "do-not-discard",
"clickElementUniquenessType": "uniqueText"
},
{
"id": "show_more",
"type": "SelectorElementClick",
"parentSelectors": [
"select_rating"
],
"selector": "span.grid-x",
"multiple": true,
"delay": "3000",
"clickElementSelector": ".grid-margin-x .small-12 button.button",
"clickType": "clickOnce",
"discardInitialElements": "do-not-discard",
"clickElementUniquenessType": "uniqueHTMLText"
},
{
"id": "links",
"type": "SelectorLink",
"parentSelectors": [
"show_more"
],
"selector": "a",
"multiple": true,
"delay": 0
}
]
}