Pagination Won't extend past onscreen links

How to?
#1

Having trouble with pagination past page 10. There are "First page," "last page," "next page," and "final page" buttons which are also scraped (somehow). I already know the pages extend into the hundreds but I can't seem to get past 10. I suspect it is these other buttons that are interfering with the process. Is there a way to exclude them while still incorporating automatic pagination?

(Is it relevant that the pages seem to be search results? This forum is organised in an odd way.)

Url: https://tieba.baidu.com/f?kw=黑人&ie=utf-8&pn=0

Sitemap:

{"_id":"bpf_sitemap","startUrl":["https://tieba.baidu.com/f?ie=utf-8&kw=黑人"],"selectors":[{"id":"bpf_threads","type":"SelectorLink","parentSelectors":["_root","pagination"],"selector":"ul.thread_top_list li.j_thread_list:nth-of-type(2) a.j_th_tit, ul.threadlist_bright > li.j_thread_list:nth-of-type(n+2) a.j_th_tit","multiple":true,"delay":"3000"},{"id":"pagination","type":"SelectorLink","parentSelectors":["_root"],"selector":"a.pagination-item","multiple":true,"delay":"7000"}]}

Many thanks in advance!

#2

pagination needs to be a parent of pagination

{"_id":"bpf_sitemap","startUrl":["https://tieba.baidu.com/f?ie=utf-8&kw=黑人"],"selectors":[{"id":"bpf_threads","type":"SelectorLink","parentSelectors":["_root","pagination"],"selector":"ul.thread_top_list li.j_thread_list:nth-of-type(2) a.j_th_tit, ul.threadlist_bright > li.j_thread_list:nth-of-type(n+2) a.j_th_tit","multiple":true,"delay":"3000"},{"id":"pagination","type":"SelectorLink","parentSelectors":["_root","pagination"],"selector":"a.next","multiple":false,"delay":"7000"}]}

I also changed the pagination link to just the "next>" link but I don't think that makes too much of a difference other than my preference in how to do it.

#3

Thank you so much. I'll give it a go and see what I can get. Cheers!

#4

I've just tested it and I'm afraid it still won't go past page 10. Is there anything else I can do?

#5

@NetworkReject

Hi! You didn't need to set thread links parent of it self and it seems you just forgotten to tick multiple in pagination selector so it won't click next more than once :slight_smile:

@mlk3589

There are two options to click pagination buttons – either use Link selector or Element Click in case Link selector works improperly. I've tested this website and it seems Link selector won't work properly on some pages (it will just stop and won't go further – maybe it's just me).

I've set up an Element Click selector for you, it does click Next page button properly (without the weird delay), and I've also set two selectors inside to pick the thread link and it's title.

Your sitemap:

{"_id":"bpf_sitemap-test","startUrl":["https://tieba.baidu.com/f?kw=黑人&ie=utf-8&pn=0"],"selectors":[{"id":"next_button","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"a.j_th_tit","multiple":true,"delay":"3000","clickElementSelector":"a.next","clickType":"clickMore","discardInitialElements":false,"clickElementUniquenessType":"uniqueText"},{"id":"thread_link","type":"SelectorElementAttribute","parentSelectors":["next_button"],"selector":"_parent_","multiple":false,"extractAttribute":"href","delay":0},{"id":"topic","type":"SelectorText","parentSelectors":["next_button"],"selector":"_parent_","multiple":false,"regex":"","delay":0}]}
#6

Thanks so much for your prompt and speedy reply. I'll try this out as soon as I get the chance.