Extension too slow

#1

Hi and thank you for the Web Scraper Firefox extension.

I'm scraping only links / urls from a simple social media platform - 20 per page. That's it.
But each page takes 40s - a 5h scrape in total. I've set the delays at the minimum but it seems to be 2s per link.
Is this a limitation of the free extension version?

Please advise :man_shrugging:

Many thanks in advance :100::100:

#2

Hi,

No, there are no built-in limitations for the browser extension. Probably has to do with how the website loads elements.

#3

Hi, thanks for the support :100:

No, the pages load pretty much instantly - yet 40s works out to be 2000ms per link. Does this not come from the request interval? 2000 is the minimum :man_shrugging:

Another question please - I used a root URL using the range [1-400] at the end. It works but it counts backwards - 400 down to 1. This is surely a bug?

Thank you

#4

Hello, any input please about the slowness / delay problem?

Thanks

#5

Hi, please share the sitemap so I can test this, thanks

#6

It's a membership social platform :cat:

Instead, can you think of a simple site for me to try as an example / exercise? Simple 'next' link to page2 type of site? :100:

#7

Good idea, you can take a generic site and set up a somewhat similar sitemap and see how it performs.

#8

Exactly. Can you think of an example perhaps?

I'll try download.com