Scrape from a range non-consecutively

How to?
#1

Hello - I'm trying to set up a scrape that can take ID's at the end of a URL and scrape through them in non-consecutive order.

Example: The tutorial clearly shows how to set a range to scrape from a website I.e. www.website.com/user/id=[10000-20000] would scrape from id=10000-20000 in consecutive order I.e. 10001,10002,10003...

I'm wondering if it's instead possible to scrape by 5's, 10's, etc, I.e. from id=10000, to 10005, to 10010 and so on. Or even randomized within a range, ideally! Is this (non-consecutive or random page selection) possible?

#2

You can use something like www.website.com/user/id=[10000-20000:10] which would increment by 10.

So, it would be 10000, 10010, 10020....

Ref: Specify multiple urls with ranges

#3

Ok great I'll give that a shot, thanks so much!

#4

@leemeng do you know if it's possible to go the other direction, I.e. 10020,10010,10000? I tried www.website.com/user/id=[20000-10000:-10] but this didn't work; is there syntax the program is able to understand to count down instead of up?

#5

Interesting case. Here's how I'd do it.

  1. Generate randomized sequences of integers. Alternatively, you can generate list of sequential numbers.
  2. Prefix the URL to the generated numbers
  3. Mass add the URLs to your sitemap (a few ways to do this)

WS does indeed work backwards thru URL ranges. I dunno why it was designed this way.