Hi, I am trying to scrape some pdf links from a website: https://egyankosh.ac.in/handle/123456789/108637
The scraping works well for the first pdf but the scraping stops after that automatically. Can someone help me get it right?
I tried increasing the delay and interval to 8000 as well:
- Request interval (ms) 8000
- Page load delay (ms) 8000
Still fails.
Here is the sitemap
{"_id":"egyankosh-ac-in","startUrl":["https://egyankosh.ac.in/handle/123456789/108637"],"selectors":[{"id":"course-name-link","linkType":"linkFromHref","multiple":false,"parentSelectors":["_root"],"selector":"div.col-md-12:nth-of-type(1) .col-md-12 .list-group-item-heading a","type":"SelectorLink","version":2},{"id":"block-name-link","linkType":"linkFromHref","multiple":false,"parentSelectors":["course-name-link"],"selector":".list-group-item-heading a","type":"SelectorLink","version":2},{"id":"block-and-unit-name-link","linkType":"linkFromHref","multiple":false,"parentSelectors":["block-name-link"],"selector":"strong a","type":"SelectorLink","version":2},{"id":"pdf-link","linkType":"linkFromHref","multiple":false,"parentSelectors":["block-and-unit-name-link"],"selector":".break-all a","type":"SelectorLink","version":2}]}