Hi all,
i'm creating a simple webscraper with all links on a singel page.
there are arround 80 links on that page. The problem is that after 26 links it stops.
i already adjusted Request interval (ms) and Page load delay (ms) but is seems to stop
at the 26 page every time.
Please post the sitemap here. We cannot replicate the bug without the sitemap.
Thanks for the reply. Here is the sitemap:
{"_id":"leukerecepten","startUrl":"https://www.leukerecepten.nl/?s=&taxonomy[course]=hoofdgerechten&taxonomy[tijd]=&taxonomy[soort]=&taxonomy[cuisine]=&taxonomy[thema]=&taxonomy[moeilijkheid]=","selectors":[{"id":"links","type":"SelectorLink","selector":"div.stream-card--list:nth-of-type(n+14) a.full-link","parentSelectors":["_root"],"multiple":true,"delay":""},{"id":"ingredienten","type":"SelectorText","selector":"div.page-content__ingredients-list","parentSelectors":["links"],"multiple":true,"regex":"","delay":""},{"id":"bereding","type":"SelectorText","selector":"div.column.page-content__recipe","parentSelectors":["links"],"multiple":true,"regex":"","delay":""}]}
I run the sitemap with web scraper 0.3.3. The "links" selector finds 17 links in the start url. The same amount of pages it visits. Maybe the initial links selector was incorrect.
The text selectors in recepie page shouldn't have multiple option checked because you are extracting single record per page.