The website I scrapped is a very slow-loading page
I tried to scrap many links at one time but it finished and only scraped about 28 links each time I tried.
I have tried to set the
Request interval (ms) 3000
Page load delay (ms) 3000
Still the same
Here is the sitemap for example of two links
Please anyone help
Sitemap:
{"_id":"aaatuckertesting","startUrl":["Tucker Item Detail td:nth-of-type(7)","multiple":false,"regex":""},{"id":"pkg","parentSelectors":["wrapper"],"type":"SelectorText","selector":"#trSearchResults_1 td:nth-of-type(8)","multiple":false,"regex":""},{"id":"retail","parentSelectors":["wrapper"],"type":"SelectorText","selector":"#trSearchResults_1 td:nth-of-type(9)","multiple":false,"regex":""},{"id":"Dlr cost","parentSelectors":["wrapper"],"type":"SelectorText","selector":"#trSearchResults_1 td:nth-of-type(10)","multiple":false,"regex":""},{"id":"mycost","parentSelectors":["wrapper"],"type":"SelectorText","selector":"#trSearchResults_1 td:nth-of-type(11)","multiple":false,"regex":""},{"id":"percent","parentSelectors":["wrapper"],"type":"SelectorText","selector":"#trSearchResults_1 td:nth-of-type(12)","multiple":false,"regex":""},{"id":"des","parentSelectors":["wrapper"],"type":"SelectorText","selector":"p.mt-1","multiple":false,"regex":""},{"id":"other","parentSelectors":["wrapper"],"type":"SelectorText","selector":".col-lg-3 div.col-lg-12:nth-of-type(2)","multiple":false,"regex":""},{"id":"des2","parentSelectors":["wrapper"],"type":"SelectorText","selector":"td.itemInquiryDescriptionLine1.sticky-column_1","multiple":false,"regex":""},{"id":"img","parentSelectors":["wrapper"],"type":"SelectorImage","selector":"img.zoomImg","multiple":false}]}