Scraping 1 Page takes exactly 1 minute... Why?

Sorry for my English :confused: Im trying to Scrape a site, but it takes exactly 60 second pro site... Here is my code:

{"_id":"secondchance","startUrl":["https://danbooru.donmai.us/posts/[1000001-3156001]"],"selectors":[{"id":"image","type":"SelectorImage","selector":"img","parentSelectors":["_root"],"multiple":false,"delay":0},{"id":"Artist","type":"SelectorText","selector":"li.category-1 a.search-tag","parentSelectors":["_root"],"multiple":true,"regex":"","delay":0},{"id":"Tags","type":"SelectorText","selector":"li.category-0 a.search-tag","parentSelectors":["_root"],"multiple":true,"regex":"","delay":0},{"id":"Score","type":"SelectorText","selector":"section#post-information li:nth-of-type(7)","parentSelectors":["_root"],"multiple":false,"regex":"","delay":0},{"id":"Rating","type":"SelectorText","selector":"section#post-information li:nth-of-type(6)","parentSelectors":["_root"],"multiple":false,"regex":"","delay":0},{"id":"Character","type":"SelectorText","selector":"li.category-4 a.search-tag","parentSelectors":["_root"],"multiple":true,"regex":"","delay":0},{"id":"Favorites","type":"SelectorText","selector":"section#post-information li:nth-of-type(8)","parentSelectors":["_root"],"multiple":false,"regex":"","delay":0},{"id":"Source","type":"SelectorText","selector":"section#post-information li:nth-of-type(5)","parentSelectors":["_root"],"multiple":false,"regex":"","delay":0},{"id":"Copyrights","type":"SelectorText","selector":"li.category-3 a.search-tag","parentSelectors":["_root"],"multiple":true,"regex":"","delay":0},{"id":"Tittle(Artist´s Commentary)","type":"SelectorText","selector":"section#translated-artist-commentary h3","parentSelectors":["_root"],"multiple":false,"regex":"","delay":0},{"id":"Meta","type":"SelectorText","selector":"li.category-5 a.search-tag","parentSelectors":["_root"],"multiple":true,"regex":"","delay":0},{"id":"Users Commenting","type":"SelectorText","selector":"a.user-member","parentSelectors":["_root"],"multiple":true,"regex":"","delay":0},{"id":"Users Comments","type":"SelectorText","selector":"div.body > p","parentSelectors":["_root"],"multiple":true,"regex":"","delay":0},{"id":"All Comments","type":"SelectorText","selector":"div.comments-for-post","parentSelectors":["_root"],"multiple":false,"regex":"","delay":0},{"id":"Something, Favorites?","type":"SelectorText","selector":"section#post-information li:nth-of-type(9)","parentSelectors":["_root"],"multiple":false,"regex":"","delay":0},{"id":"Tagger","type":"SelectorText","selector":"section#post-information li:nth-of-type(2)","parentSelectors":["_root"],"multiple":false,"regex":"","delay":0},{"id":"Approver","type":"SelectorText","selector":"section#post-information li:nth-of-type(4)","parentSelectors":["_root"],"multiple":false,"regex":"","delay":0},{"id":"More Data!","type":"SelectorText","selector":"section#post-information li:nth-of-type(1)","parentSelectors":["_root"],"multiple":false,"regex":"","delay":0},{"id":"Much More!!!","type":"SelectorText","selector":"section#post-information li:nth-of-type(3)","parentSelectors":["_root"],"multiple":false,"regex":"","delay":0},{"id":"Take Me Home","type":"SelectorText","selector":"section#post-information li:nth-of-type(10)","parentSelectors":["_root"],"multiple":false,"regex":"","delay":0},{"id":"io k ce","type":"SelectorText","selector":"section#post-information li:nth-of-type(12)","parentSelectors":["_root"],"multiple":false,"regex":"","delay":0},{"id":"Por si las moscas","type":"SelectorText","selector":"section#post-information li:nth-of-type(12)","parentSelectors":["_root"],"multiple":false,"regex":"","delay":0},{"id":"Tag2","type":"SelectorText","selector":"li.category-2 a.search-tag","parentSelectors":["_root"],"multiple":true,"regex":"","delay":0},{"id":"Tag6","type":"SelectorText","selector":"li.category-6 a.search-tag","parentSelectors":["_root"],"multiple":true,"regex":"","delay":0},{"id":"Tag7","type":"SelectorText","selector":"li.category-7 a.search-tag","parentSelectors":["_root"],"multiple":true,"regex":"","delay":0}]}

My request interval is 2000 and my Page load delay is 500, According to "Network" the page loads in 2 seconds... And i'm using Chromium. Help please! Now i dont now what to do :stuck_out_tongue:

You have 2m start URLs. Web scraper at the beginning of the scraping job queues them all and chrome probably is starting to run out of memory. Split your sitemap in smaller parts, preferably 5k URLs per sitemap and scraper should run in regular speed.

1 Like

Oh, i just realized that... Thanks for your answer! I will try it later so i can post it if it works...

Edit: It solved my problem! Thanks for your help :slight_smile: