Hi. I am trying to scrape a software category list on capterra.com. The list has a "show more" button at the bottom of the page. I have the following sitemap that navigates through the list using the "show more" and displays the software name from the list. This works perfectly:
{"_id":"capterralist","startUrl":["https://www.capterra.com/auction-software/"],"selectors":[{"id":"searchpage","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"div[class|=ProductCard__Root]","multiple":true,"delay":2000,"clickElementSelector":"button[class|=Button__StyledButton]","clickType":"clickMore","discardInitialElements":"discard","clickElementUniquenessType":"uniqueCSSSelector"},{"id":"softwarename","type":"SelectorText","parentSelectors":["searchpage"],"selector":"h2[class|=ProductHeaderSection__ProductHeader]","multiple":false,"regex":"","delay":0}]}
Note I have already discovered the issue with weird random characters in class names that change, hence the use of the CSS selector "class|={value}" to only pick up the first part of the class name and ignore all the random stuff at the end.
Next, I am trying to go the next step and follow a link from each entry in the list to a separate software profile page to get more information from that. Here is that sitemap:
{"_id":"capterra","startUrl":["https://www.capterra.com/auction-software/"],"selectors":[{"id":"searchpage","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"div[class|=ProductCard__Root]","multiple":true,"delay":"2000","clickElementSelector":"button[class|=Button__StyledButton]","clickType":"clickMore","discardInitialElements":"discard","clickElementUniquenessType":"uniqueCSSSelector"},{"id":"softwarelink","type":"SelectorLink","parentSelectors":["searchpage"],"selector":"a[class|=Thumbnail__A]","multiple":false,"delay":0},{"id":"Product","type":"SelectorText","parentSelectors":["softwarelink"],"selector":"h1[class|=DesktopProductHeader__ProductHeading]","multiple":false,"regex":"","delay":0},{"id":"Company","type":"SelectorText","parentSelectors":["softwarelink"],"selector":"h2[class|=DesktopProductHeader__VendorSubheading]","multiple":false,"regex":"","delay":0}]}
This one navigates through the list, and follows the link to the profile for the first 5 entries, then it quits finding the link. The rest of the entries show a null link. I read somewhere on this site to try going to the page and "shrinking" the page (chrome control, go from 100% down to 25%), so I tried that and then it works for 7 entries. Go back to 100% and I get 5 entries again. Totally consistent.
So, why is it failing after 5 entries? And what does the size of the text have to do with any of this? Any help is appreciated.
