I'm totaly new here. I have been testing for a couple of hours now but it just wont work as I want it to.
My goal is it to collect data from one site, going to the next page and collecting the data on the new page, and so on.
Sometimes it runs but I dont get data collected (output is null) or it just crashes after the first page.
When the scraper runs it stops after a certain amount of pages, but I guess its a kind of ddos protection.
Url: https://www.bustabit.com/game/1
there are close to 3,000,000 pages to collect data from
Sitemap:
{"_id":"bustabit","startUrl":["https://www.bustabit.com/game/1"],"selectors":[{"id":"Bust At","type":"SelectorText","parentSelectors":["_root","nextSite"],"selector":"h5:nth-of-type(1) span.bold","multiple":false,"regex":"","delay":0},{"id":"nextSite","type":"SelectorLink","parentSelectors":["_root","nextSite"],"selector":".next a","multiple":false,"delay":0}]}
Ideal would be 'Game#' 'Bust At' and 'Date'. But I didnt manage to get it working, therefor only the 'Bust At'. I tried Element click aswell, didnt work either