I´m trying to scrape a website with long posts with HTML element to obtain h3 , p, etc. But the post are scrape incomplete. It is like the scraper stop in the middle of the posts. I try to increase request interval and page load delay to 5s but the result it is the same.
Url: https://www.seriouseats.com/how-tos/equipment
Sitemap:
{"_id":"seriouseats","startUrl":["https://www.seriouseats.com/how-tos/equipment?page=[1-3]"],"selectors":[{"id":"link","type":"SelectorLink","parentSelectors":["_root"],"selector":".metadata a","multiple":true,"delay":0},{"id":"content","type":"SelectorHTML","parentSelectors":["link"],"selector":"div.entry-body","multiple":true,"regex":"","delay":0}]}