Script does not scrape all items on page after pagination

Bug Reports
#1

Hi,

I am having an issue where pagination is not working consistently to scrape everything on all the pages. At the time of scraping there is 498 items (40 per page, spread across 13 pages)

However after i scrape, there is only 280 items (exactly 7 pages full), it seems that its working correctly on some pages but not all. I read in other thread that it might be due to javascript on the page that is causing it and not the scraper.

Does anyone have a better idea of what is causing it and how to fix it?

Thank you very much!

Web Scraper version: 0.37
Chrome version: 65.0.3325.181
OS: Windows 10

{"_id":"lazada_overall","startUrl":["https://www.lazada.sg/shop-home-improvement/bosch--black-and-decker--makita--stanley/?location=local"],"selectors":[{"id":"container","type":"SelectorElement","selector":"div.c2prKC","parentSelectors":["_root","pagination"],"multiple":true,"delay":0},{"id":"name","type":"SelectorText","selector":"div.c16H9d a","parentSelectors":["container"],"multiple":false,"regex":"","delay":0},{"id":"original price","type":"SelectorText","selector":"del.c13VH6","parentSelectors":["container"],"multiple":false,"regex":"","delay":0},{"id":"sale price","type":"SelectorText","selector":"span.c13VH6","parentSelectors":["container"],"multiple":false,"regex":"","delay":0},{"id":"units_sold","type":"SelectorText","selector":"span.c54ki6 span:nth-of-type(1)","parentSelectors":["container"],"multiple":false,"regex":"","delay":0},{"id":"country","type":"SelectorText","selector":"span.c2i43-","parentSelectors":["container"],"multiple":false,"regex":"","delay":0},{"id":"pagination","type":"SelectorLink","selector":"li.ant-pagination-item a","parentSelectors":["_root","pagination"],"multiple":true,"delay":0}]}

Sitemap:

{"_id":"lazada_overall","startUrl":["https://www.lazada.sg/shop-home-improvement/stanley/?location=local"],"selectors":[{"id":"container","type":"SelectorElement","selector":"div.c2prKC","parentSelectors":["_root","pagination"],"multiple":true,"delay":0},{"id":"name","type":"SelectorText","selector":"div.c16H9d a","parentSelectors":["container"],"multiple":false,"regex":"","delay":0},{"id":"original price","type":"SelectorText","selector":"del.c13VH6","parentSelectors":["container"],"multiple":false,"regex":"","delay":0},{"id":"sale price","type":"SelectorText","selector":"span.c13VH6","parentSelectors":["container"],"multiple":false,"regex":"","delay":0},{"id":"units_sold","type":"SelectorText","selector":"span.c54ki6 span:nth-of-type(1)","parentSelectors":["container"],"multiple":false,"regex":"","delay":0},{"id":"country","type":"SelectorText","selector":"span.c2i43-","parentSelectors":["container"],"multiple":false,"regex":"","delay":0},{"id":"pagination","type":"SelectorLink","selector":"li.ant-pagination-item a","parentSelectors":["_root","pagination"],"multiple":true,"delay":0}]}
#2

Hi,

give oxygen to the extension increasing the delays
I also changed the name of the selector for pagination.

For my test, I change the selection to reduce the number of pages.
4 pages * 10 items = 160 items --> I scaped all these 160 items regarding the number of lines obtained under excel.

PS - can't explain while the site display with my example 132 items found ....

This is my scrapesite:

{"_id":"test2","startUrl":["https://www.lazada.sg/shop-home-improvement/makita/?location=local"],"selectors":[{"id":"container","type":"SelectorElement","selector":"div.c2prKC","parentSelectors":["_root","pagination"],"multiple":true,"delay":"2000"},{"id":"name","type":"SelectorText","selector":"div.c16H9d a","parentSelectors":["container"],"multiple":false,"regex":"","delay":0},{"id":"original price","type":"SelectorText","selector":"del.c13VH6","parentSelectors":["container"],"multiple":false,"regex":"","delay":0},{"id":"sale price","type":"SelectorText","selector":"span.c13VH6","parentSelectors":["container"],"multiple":false,"regex":"","delay":0},{"id":"units_sold","type":"SelectorText","selector":"span.c54ki6 span:nth-of-type(1)","parentSelectors":["container"],"multiple":false,"regex":"","delay":0},{"id":"country","type":"SelectorText","selector":"span.c2i43-","parentSelectors":["container"],"multiple":false,"regex":"","delay":0},{"id":"pagination","type":"SelectorLink","selector":"ul.ant-pagination li:nth-of-type(n+3) a","parentSelectors":["_root","pagination"],"multiple":true,"delay":"2000"}]}