Paging not working, though element selector seems ok

How to?
#1

Yet another paging question for the forum! I am trying to get paging working for the Library Thing MDS records: MDS: 111 | LibraryThing

I have a very simple site map (to start with, at least). The paging selector seems to be getting the "next" button ok, but I only ever get the first page of data for each URL.

Sitemap:
{"_id":"LibraryThing","startUrl":["MDS: n/a | LibraryThing > div a:last-child","type":"SelectorPagination"},{"id":"titles","multiple":true,"parentSelectors":["paging"],"regex":"","selector":"a.popup_registered","type":"SelectorText"},{"id":"MDS_number","multiple":false,"parentSelectors":["paging"],"regex":"","selector":".shelforlist h2","type":"SelectorText"}]}

#2

Hi,

Here is a reference for a pagination setup:

{"_id":"librarything","startUrl":["https://www.librarything.com/mds/111"],"selectors":[{"id":"pagination","paginationType":"auto","parentSelectors":["_root","pagination"],"selector":"#ddc > div a:contains(\"next\")","type":"SelectorPagination"},{"id":"listing","multiple":true,"parentSelectors":["pagination"],"regex":"","selector":".lt2_columnar_item li","type":"SelectorText"}]}

Note, that results will only be returned after the pagination is completed.

#3

Thanks JanAp. However, that sitemap gives me the same problem (and it looks pretty much the same as what I had, other than a slightly different selector for the paging button). I still am only getting the first page of data (50 titles) when I choose the Scrape command. Maybe I'm somehow doing that wrong? Did that sitemap give you all the pages of data for that URL?

#4

My apologies, I tested this with a different version of the extension. With the current version, the pagination indeed does not work.

The version update is planned for next week. Please update the extension when the new version is out and test the sitemap.

#5

Aha! Will do - thanks again!

#6

Hi,

The new version of the extension is live, you can update now and test the sitemap!

#7

Got it and it works! (Well, when that site works - the paging is flaky but I'm 99% sure it's the site and not webscraper.) Thanks!