Web Scraper endless loop - pagination never finishes

Bug Reports
#1

Hi, trying to scrape some audible.de site and the scraping goes on forever. Not sure what I'm doing wrong here. I've tried to make it reproduceable without need for any login. Would be great if some could have a look, maybe something obvious that I'm missing.

{"_id":"audible-de-2Fuer1-KrimisThriller","startUrl":["https://www.audible.de/search?node=16209910031&sort=popularity-rank&ref_pageloadid=HZIn2Br9BrdiaKhO&ref=a_hp_c6_adblp13nmpxx_showmore&pf_rd_p=2fee4f1e-c97f-47aa-a770-c795ec1bc2cc&pf_rd_r=58MPY0MA2B1ZKVKY3ST1&pageLoadId=ebV6Klv2n9bACOgK&ref_plink=not_applicable&creativeId=cbb01799-9edf-480a-bf4d-af650ab19278"],"selectors":[{"id":"Book","parentSelectors":["_root","Next"],"type":"SelectorElement","selector":".bc-col-responsive.bc-spacing-top-none > div.bc-row-responsive","multiple":true},{"id":"Title","parentSelectors":["Book"],"type":"SelectorText","selector":".bc-heading a","multiple":false,"regex":""},{"id":"Author","parentSelectors":["Book"],"type":"SelectorText","selector":".authorLabel span","multiple":false,"regex":""},{"id":"Narrator","parentSelectors":["Book"],"type":"SelectorText","selector":".narratorLabel span","multiple":false,"regex":""},{"id":"Dauer","parentSelectors":["Book"],"type":"SelectorText","selector":".runtimeLabel span","multiple":false,"regex":""},{"id":"Sprache","parentSelectors":["Book"],"type":"SelectorText","selector":".languageLabel span","multiple":false,"regex":""},{"id":"RatingLabel","parentSelectors":["Book"],"type":"SelectorText","selector":"li.ratingsLabel","multiple":false,"regex":""},{"id":"Image","parentSelectors":["Book"],"type":"SelectorImage","selector":"img.bc-pub-block","multiple":false},{"id":"Next","parentSelectors":["_root","Next"],"paginationType":"auto","type":"SelectorPagination","selector":".nextButton span.bc-button-text-inner"}]}
#2

Hi, I placed the Book selector under the pagination selector and updated it, should work smoother now.

{"_id":"audible-de-2Fuer1-KrimisThriller","startUrl":["https://www.audible.de/search?node=16209910031&sort=popularity-rank&ref_pageloadid=HZIn2Br9BrdiaKhO&ref=a_hp_c6_adblp13nmpxx_showmore&pf_rd_p=2fee4f1e-c97f-47aa-a770-c795ec1bc2cc&pf_rd_r=58MPY0MA2B1ZKVKY3ST1&pageLoadId=ebV6Klv2n9bACOgK&ref_plink=not_applicable&creativeId=cbb01799-9edf-480a-bf4d-af650ab19278"],"selectors":[{"id":"Next","paginationType":"auto","parentSelectors":["_root","Next"],"selector":"[rel=\"next\"]","type":"SelectorPagination"},{"id":"Book","multiple":true,"parentSelectors":["Next"],"selector":".bc-col-responsive.bc-spacing-top-none > div.bc-row-responsive","type":"SelectorElement"},{"id":"Title","multiple":false,"parentSelectors":["Book"],"regex":"","selector":".bc-heading a","type":"SelectorText"},{"id":"Author","multiple":false,"parentSelectors":["Book"],"regex":"","selector":".authorLabel span","type":"SelectorText"},{"id":"Narrator","multiple":false,"parentSelectors":["Book"],"regex":"","selector":".narratorLabel span","type":"SelectorText"},{"id":"Dauer","multiple":false,"parentSelectors":["Book"],"regex":"","selector":".runtimeLabel span","type":"SelectorText"},{"id":"Sprache","multiple":false,"parentSelectors":["Book"],"regex":"","selector":".languageLabel span","type":"SelectorText"},{"id":"RatingLabel","multiple":false,"parentSelectors":["Book"],"regex":"","selector":"li.ratingsLabel","type":"SelectorText"},{"id":"Image","multiple":false,"parentSelectors":["Book"],"selector":"img.bc-pub-block","type":"SelectorImage"}]}
#3

@JanAp Excellent, thank you. It works on this URL but strangely on some others (which are accessible only with login details) it doesn't. I cannot see any differences in the HTML code. Weird.

How did you get to the [rel="next"] setting, I don't get this set when I choose with the selection button?

Very much appreciate your help on this!

#4

Hi, not every page has this attribute. It is the best way to set up pagination for a clean and fast result. You can find it in the HTML head. If it is not there, pagination needs to be set up in an alternate way.

image
image931×157 42.2 KB

#5

This link setting (link rel="next") is not available on all of the audible sites and so the pagination does not work for them. Is there an alternative to that?

#6

For this site, the URL does seem to change along with the pages, so you could use the "range url with increment" method listed under Specify multiple urls with ranges. That is, no paginator needed.

In your URL, adding &page=XX would load that page directly. For example. this will load page 21:

https://www.audible.de/search?node=16209910031&sort=popularity-rank&page=21&ref_pageloadid=ebV6Klv2n9bACOgK&ref=a_search_c4_pageNum_1&pf_rd_p=cc935d10-7a4f-4b11-bbdf-80ec5c6000ab&pf_rd_r=YW2BCF3Y8QQDQ4XBR4HD&pageLoadId=uqW6xz262B7zImAG&ref_plink=not_applicable&creativeId=ef56eba4-2dfe-4ca8-9612-8dd421243848

So you could try inserting &page=[1-25] into your Start URL.