Edit: I can get it to scrape fully with auto pagination, but this has proven to be unreliable for me in the past (which admittedly was probably my own fault). Still wondering why it won't scrape properly when defining a range or multiple pages in the metadata (like below, i.e. no pagination in the scrape script)
Describe the problem:
Scraper won't scrape more than 1 page. Below has worked for me in the past, but now both below options in site metadata "Start URL" are not working:
-
All start pages in one with [x-y]:
https://www.soundimports.eu/en/audio-components/woofers/?hr-page=%7B%22page%22%3A[1-39]%7D -
Separate start pages:
https://www.soundimports.eu/en/audio-components/woofers/?hr-page=%7B%22page%22%3A1%7D, https://www.soundimports.eu/en/audio-components/woofers/?hr-page=%7B%22page%22%3A2%7Detc.
Both definitions will only result in one page being crawled. It will visit all the "main" pages but only "click through" on the first one (if that makes any sense).
Url: http://www.soundimports.eu
Sitemap:
{"_id":"soundimports_eu_woofers3_test","startUrl":["https://www.soundimports.eu/en/audio-components/woofers/?hr-page=%7B%22page%22%3A[1-39]%7D"],"selectors":[{"id":"Product_title","multiple":false,"parentSelectors":["Link_to_product_page"],"regex":"","selector":"h1","type":"SelectorText"},{"id":"Brand","multiple":false,"parentSelectors":["Link_to_product_page"],"regex":"","selector":".brand a","type":"SelectorText"},{"id":"Artikelcode","multiple":false,"parentSelectors":["Link_to_product_page"],"regex":"","selector":"dt:contains('Article number\n ') + dd","type":"SelectorText"},{"id":"Woofer series","multiple":false,"parentSelectors":["Link_to_product_page"],"regex":"","selector":"dt:contains('Woofer Series ') + dd","type":"SelectorText"},{"id":"Woofer type","multiple":false,"parentSelectors":["Link_to_product_page"],"regex":"","selector":"dt:contains('Woofer Type ') + dd","type":"SelectorText"},{"id":"Item_element_selector","multiple":true,"parentSelectors":["_root"],"selector":"div.product","type":"SelectorElement"},{"id":"Link_to_product_page","linkType":"linkFromHref","multiple":false,"parentSelectors":["Item_element_selector"],"selector":".hide-575 a","type":"SelectorLink"},{"id":"Nominal diameter","multiple":false,"parentSelectors":["Link_to_product_page"],"regex":"","selector":"dt:contains('Nominal Diameter ') + dd","type":"SelectorText"},{"id":"Power RMS","multiple":false,"parentSelectors":["Link_to_product_page"],"regex":"","selector":"dt:contains('Power Handling (RMS) ') + dd","type":"SelectorText"},{"id":"Price","multiple":false,"parentSelectors":["Item_element_selector"],"regex":"(?<=€).*","selector":"div.current","type":"SelectorText"}]}