Pagination seems to work, but won't get more than 21 lines of data out of 22,215

How to?
#1

Pagination seems to be working, but I can't collect more than 21 lines of data out of 22,215. Thank you in advance for any help someone can provide to get this working.

Url: https://www.hankooktire.com/us/services-tips/find-a-dealer/view-all.html

Sitemap:
{"_id":"hankook","startUrl":["https://www.hankooktire.com/us/services-tips/find-a-dealer/find-by-location.html"],"selectors":[{"id":"view all","type":"SelectorLink","selector":"ul.invest_navi_new li:nth-of-type(2) a","parentSelectors":["_root"],"multiple":false,"delay":"2000"},{"id":"dealers","type":"SelectorTable","selector":"div.data_table table","parentSelectors":["view all","other pages"],"multiple":true,"columns":[{"header":"Name","name":"Name","extract":true},{"header":"Address","name":"Address","extract":true},{"header":"Telephone","name":"Telephone","extract":true},{"header":"Type","name":"Type","extract":true}],"delay":0,"tableDataRowSelector":"tbody tr","tableHeaderRowSelector":"thead tr"},{"id":"other pages","type":"SelectorElementClick","selector":"div.pagination a","parentSelectors":["view all"],"multiple":false,"delay":"0","clickElementSelector":"div.pagination a","clickType":"clickMore","discardInitialElements":false,"clickElementUniquenessType":"uniqueHTML"}]}

#2

I also tried navigating the pagination links a different way, only testing with the first 2 plus the next page button. It navigates, but doesn't get the data from all the pages and I don't know why. I also get data duplicate times. I've tried every combination of using click types, multiple toggled on and off, etc, but can't get this to work.

{"_id":"hanook2","startUrl":["https://www.hankooktire.com/us/services-tips/find-a-dealer/view-all.html"],"selectors":[{"id":"get dealer info","type":"SelectorTable","selector":"div.data_table table","parentSelectors":["_root","2nd box","3rd box","next page box"],"multiple":true,"columns":[{"header":"Name","name":"Name","extract":true},{"header":"Address","name":"Address","extract":true},{"header":"Telephone","name":"Telephone","extract":true},{"header":"Type","name":"Type","extract":false}],"delay":"1000","tableDataRowSelector":"tbody tr","tableHeaderRowSelector":"thead tr"},{"id":"2nd box","type":"SelectorElementClick","selector":"div.pagination a:nth-of-type(1)","parentSelectors":["_root"],"multiple":false,"delay":0,"clickElementSelector":"div.pagination a:nth-of-type(1)","clickType":"clickOnce","discardInitialElements":false,"clickElementUniquenessType":"uniqueText"},{"id":"3rd box","type":"SelectorElementClick","selector":"div.pagination a:nth-of-type(2)","parentSelectors":["_root"],"multiple":false,"delay":0,"clickElementSelector":"div.pagination a:nth-of-type(2)","clickType":"clickOnce","discardInitialElements":false,"clickElementUniquenessType":"uniqueText"},{"id":"next page box","type":"SelectorElementClick","selector":"a.next","parentSelectors":["2nd box","3rd box"],"multiple":false,"delay":0,"clickElementSelector":"a.next","clickType":"clickOnce","discardInitialElements":false,"clickElementUniquenessType":"uniqueText"}]}

#3

Hi there!

This website uses JavaScript for pagination, so only type of selector you can use is Element Click.

Unfortunately i did not analyze your sitemap (will do later), but quickly built you an example sitemap that will pick all results (including page number to have it both tested and ordered afterwards if you're not using CouchDB).

Here's the sitemap:
{"_id":"hankooktire","startUrl":["https://www.hankooktire.com/us/services-tips/find-a-dealer/view-all.html"],"selectors":[{"id":"pagination","type":"SelectorElementClick","selector":"div.comm_wrap","parentSelectors":["_root"],"multiple":true,"delay":"1000","clickElementSelector":"a.next","clickType":"clickMore","discardInitialElements":false,"clickElementUniquenessType":"uniqueText"},{"id":"grouping","type":"SelectorElement","selector":"div.data_table tbody tr","parentSelectors":["pagination"],"multiple":true,"delay":0},{"id":"Name","type":"SelectorText","selector":"td.left:nth-of-type(1)","parentSelectors":["grouping"],"multiple":false,"regex":"","delay":0},{"id":"Address","type":"SelectorText","selector":"td.left:nth-of-type(2)","parentSelectors":["grouping"],"multiple":false,"regex":"","delay":0},{"id":"Phone","type":"SelectorText","selector":"td.left:nth-of-type(3)","parentSelectors":["grouping"],"multiple":false,"regex":"","delay":0},{"id":"pgnum","type":"SelectorText","selector":"div.pagination strong","parentSelectors":["pagination"],"multiple":false,"regex":"","delay":0}]}

#4

Thank you iconoclast! Looking how you formated this helped, but I'm still getting stuck. Running your sitemap, I get only the first page of each group of 10 pages because it's clicking the next button that loads the next 10 pages. It's not going through pages 2 thru 10.

It looks like I need to get it to scrape the data from the first page, then click the div.pagination a:nth-of-type(1) link, scrape, click div.pagination a:nth-of-type(2), scrape, and repeat until it gets to div.pagination a:nth-of-type(11), then click a.next, scrape, and repeat cycling through all the pages again until it gets to page 2222. I don't know how to setup this format.

I tried with the following sitemap doing just the 2nd and 3rd box, then telling it to load the next 10 pages and repeat. However, I still only get the first page out of the group of 10.

Thank you in advance for any more assistance.

{"_id":"hankookimport","startUrl":["https://www.hankooktire.com/us/services-tips/find-a-dealer/view-all.html"],"selectors":[{"id":"2nd box","type":"SelectorElementClick","selector":"div.comm_wrap","parentSelectors":["_root"],"multiple":true,"delay":"1000","clickElementSelector":"div.pagination a:nth-of-type(1)","clickType":"clickMore","discardInitialElements":false,"clickElementUniquenessType":"uniqueText"},{"id":"grouping","type":"SelectorElement","selector":"div.data_table tbody tr","parentSelectors":["2nd box","3rd box","next 10"],"multiple":true,"delay":0},{"id":"Name","type":"SelectorText","selector":"td.left:nth-of-type(1)","parentSelectors":["grouping"],"multiple":false,"regex":"","delay":0},{"id":"Address","type":"SelectorText","selector":"td.left:nth-of-type(2)","parentSelectors":["grouping"],"multiple":false,"regex":"","delay":0},{"id":"Phone","type":"SelectorText","selector":"td.left:nth-of-type(3)","parentSelectors":["grouping"],"multiple":false,"regex":"","delay":0},{"id":"pgnum","type":"SelectorText","selector":"div.pagination strong","parentSelectors":["2nd box","3rd box","next 10"],"multiple":false,"regex":"","delay":0},{"id":"3rd box","type":"SelectorElementClick","selector":"div.comm_wrap","parentSelectors":["_root"],"multiple":true,"delay":"1000","clickElementSelector":"div.pagination a:nth-of-type(2)","clickType":"clickMore","discardInitialElements":false,"clickElementUniquenessType":"uniqueText"},{"id":"next 10","type":"SelectorElementClick","selector":"div.comm_wrap","parentSelectors":["_root"],"multiple":true,"delay":0,"clickElementSelector":"a.next","clickType":"clickMore","discardInitialElements":false,"clickElementUniquenessType":"uniqueText"}]}

#5

Well, this one gets me only the page 3 info from the groups of 10. LOL...

{"_id":"hankookimport","startUrl":["https://www.hankooktire.com/us/services-tips/find-a-dealer/view-all.html"],"selectors":[{"id":"2nd box","type":"SelectorElementClick","selector":"div.comm_wrap","parentSelectors":["_root"],"multiple":true,"delay":"1000","clickElementSelector":"div.pagination a:nth-of-type(1)","clickType":"clickMore","discardInitialElements":false,"clickElementUniquenessType":"uniqueText"},{"id":"grouping","type":"SelectorElement","selector":"div.data_table tbody tr","parentSelectors":["2nd box","3rd box","next 10"],"multiple":true,"delay":0},{"id":"Name","type":"SelectorText","selector":"td.left:nth-of-type(1)","parentSelectors":["grouping"],"multiple":false,"regex":"","delay":0},{"id":"Address","type":"SelectorText","selector":"td.left:nth-of-type(2)","parentSelectors":["grouping"],"multiple":false,"regex":"","delay":0},{"id":"Phone","type":"SelectorText","selector":"td.left:nth-of-type(3)","parentSelectors":["grouping"],"multiple":false,"regex":"","delay":0},{"id":"pgnum","type":"SelectorText","selector":"div.pagination strong","parentSelectors":["2nd box","3rd box","next 10"],"multiple":false,"regex":"","delay":0},{"id":"3rd box","type":"SelectorElementClick","selector":"div.comm_wrap","parentSelectors":["2nd box"],"multiple":true,"delay":"1000","clickElementSelector":"div.pagination a:nth-of-type(2)","clickType":"clickMore","discardInitialElements":false,"clickElementUniquenessType":"uniqueText"},{"id":"next 10","type":"SelectorElementClick","selector":"div.comm_wrap","parentSelectors":["_root"],"multiple":true,"delay":0,"clickElementSelector":"a.next","clickType":"clickMore","discardInitialElements":false,"clickElementUniquenessType":"uniqueText"}]}

#6

As I continue to brute force my way thru trying to figure this out, this sitemap gives me only the page 2 info from all the groups of 10.

{"_id":"hankookimport","startUrl":["https://www.hankooktire.com/us/services-tips/find-a-dealer/view-all.html"],"selectors":[{"id":"2nd box","type":"SelectorElementClick","selector":"div.comm_wrap","parentSelectors":["_root"],"multiple":true,"delay":"1000","clickElementSelector":"div.pagination a:nth-of-type(1)","clickType":"clickMore","discardInitialElements":false,"clickElementUniquenessType":"uniqueText"},{"id":"grouping","type":"SelectorElement","selector":"div.data_table tbody tr","parentSelectors":["2nd box","3rd box","next 10"],"multiple":true,"delay":0},{"id":"Name","type":"SelectorText","selector":"td.left:nth-of-type(1)","parentSelectors":["grouping"],"multiple":false,"regex":"","delay":0},{"id":"Address","type":"SelectorText","selector":"td.left:nth-of-type(2)","parentSelectors":["grouping"],"multiple":false,"regex":"","delay":0},{"id":"Phone","type":"SelectorText","selector":"td.left:nth-of-type(3)","parentSelectors":["grouping"],"multiple":false,"regex":"","delay":0},{"id":"pgnum","type":"SelectorText","selector":"div.pagination strong","parentSelectors":["2nd box","3rd box","next 10"],"multiple":false,"regex":"","delay":0},{"id":"3rd box","type":"SelectorElementClick","selector":"div.comm_wrap","parentSelectors":["2nd box"],"multiple":true,"delay":"1000","clickElementSelector":"div.pagination a:nth-of-type(2)","clickType":"clickMore","discardInitialElements":false,"clickElementUniquenessType":"uniqueText"},{"id":"next 10","type":"SelectorElementClick","selector":"div.comm_wrap","parentSelectors":["_root","2nd box","3rd box"],"multiple":true,"delay":0,"clickElementSelector":"a.next","clickType":"clickMore","discardInitialElements":false,"clickElementUniquenessType":"uniqueText"}]}

#7

Yes it's my mistake, i did not notice it goes from page 1 to page 11 and so on (21, 31).

I've reworked the sitemap, now it uses CSS selectors to limit particular buttons (only page numbers), and 'Next' button. When i came back from shop it was done - all 22215 results were scraped.

Here's the sitemap:
{"_id":"hankooktire","startUrl":["https://www.hankooktire.com/us/services-tips/find-a-dealer/view-all.html"],"selectors":[{"id":"pagination","type":"SelectorElementClick","selector":"div.comm_wrap","parentSelectors":["_root"],"multiple":true,"delay":"1000","clickElementSelector":"div.pagination a:not(.last):not(.pre):not(.first):not(.next), a.next","clickType":"clickMore","discardInitialElements":false,"clickElementUniquenessType":"uniqueText"},{"id":"grouping","type":"SelectorElement","selector":"div.data_table tbody tr","parentSelectors":["pagination"],"multiple":true,"delay":0},{"id":"Name","type":"SelectorText","selector":"td.left:nth-of-type(1)","parentSelectors":["grouping"],"multiple":false,"regex":"","delay":0},{"id":"Address","type":"SelectorText","selector":"td.left:nth-of-type(2)","parentSelectors":["grouping"],"multiple":false,"regex":"","delay":0},{"id":"Phone","type":"SelectorText","selector":"td.left:nth-of-type(3)","parentSelectors":["grouping"],"multiple":false,"regex":"","delay":0},{"id":"pgnum","type":"SelectorText","selector":"div.pagination strong","parentSelectors":["pagination"],"multiple":false,"regex":"","delay":0}]}

I've uploaded CSV file for you as well (it needs to be sorted by page number though):
https://nofile.io/f/unRT40QoCCQ/hankooktire.csv

#8

Thank you! You went above and beyond on getting that working! Awesome. I'll take a closer look at how you did that later. Thanks a ton!