I think I've found a bug with Web Scraper using paging via the Element Click Selector.
I'm scraping a search term that returns multiple pages of items (products). I get duplicate rows for the last page scraped. Not all the rows are duplicated. The first three items on the page are not duplicated, but every item after that is. I have tried many different options for click type, click element uniqueness, and discard initial elements. Note that I've tried several different search terms that return different number of results. I have found the behavior to be consistent in duplicating all but the first three items on the last search page.
Sitemap:
{"id":"la_test18","startUrl":["https://www.liveauctioneers.com/search/?keyword=hank%20aaron%201960&sort=-relevance&status=archive&pageSize=48"],"selectors":[{"id":"paginate","type":"SelectorElementClick","parentSelectors":["root"],"selector":"div.item-card___2GTKi","multiple":true,"delay":2000,"clickElementSelector":".next span","clickType":"clickMore","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueCSSSelector"},{"id":"Item","type":"SelectorElement","parentSelectors":["paginate"],"selector":"div.ItemCard__StyledPrintContainer-sc-14sctpg-1","multiple":true,"delay":0},{"id":"Title","type":"SelectorText","parentSelectors":["Item"],"selector":".link_ div span","multiple":false,"regex":"","delay":0}]}
The reason I think it's a bug with Web Scraper is that when I use a different method of paging (substituting the page number in the URL start [1-3]) it correctly returns all the rows with no duplication. Here's the sitemap using this different method of paging:
{"_id":"latest17","startUrl":["https://www.liveauctioneers.com/search/?keyword=hank%20aaron%201960&pagesize=48&sort=-relevance&status=archive&page=[1-3]"],"selectors":[{"id":"Item","type":"SelectorElement","parentSelectors":["root"],"selector":"div.ItemCard__StyledPrintContainer-sc-14sctpg-1","multiple":true,"delay":0},{"id":"ItemLink","type":"SelectorLink","parentSelectors":["Item"],"selector":"a.item-image-link-container___3rNM","multiple":false,"delay":0}]}
Unfortunately, I can't use this method for all the searches I want to scrape as the website does not generate higher number pages (over 5 I think) without rendering lower number pages first. And, because Web Scraper scrapes from higher number pages to lower number pages, we get incorrect results.
Any help in determining what is causing the duplication is much appreciated!