Working fine, almost! but limited results [SOLVED]

How to?
#1

The site returns 25 results per page, there are 9 pages in this example with a total of 212. The final scrape only gets 143. I get all the data as expected from each result I get.

I have tried increasing the 2000 to 5000 and get the same result.

Any help would be appreciated.

Url: https://www.thomsonlocal.com/search/commercial-cleaning-companies/west-midlands

Sitemap:
{"_id":"thomson-evesham-pubs","startUrl":["https://www.thomsonlocal.com/search/commercial-cleaning-companies/west-midlands?page=[1-9]"],"selectors":[{"id":"item-links","type":"SelectorLink","parentSelectors":["_root"],"selector":"a.name","multiple":true,"delay":0},{"id":"business-name","type":"SelectorText","parentSelectors":["item-links"],"selector":"h1","multiple":false,"regex":"","delay":0},{"id":"telephone-number","type":"SelectorText","parentSelectors":["item-links"],"selector":".mobileHide p","multiple":false,"regex":"","delay":0},{"id":"address1","type":"SelectorText","parentSelectors":["item-links"],"selector":"span[itemprop='streetAddress']","multiple":false,"regex":"","delay":0},{"id":"town","type":"SelectorText","parentSelectors":["item-links"],"selector":"span[itemprop='addressLocality']","multiple":false,"regex":"","delay":0},{"id":"postcode","type":"SelectorText","parentSelectors":["item-links"],"selector":"span[itemprop='postalCode']","multiple":false,"regex":"","delay":0},{"id":"alternate-phone","type":"SelectorText","parentSelectors":["item-links"],"selector":".additionalPhonesBlock span","multiple":false,"regex":"","delay":0},{"id":"website","type":"SelectorText","parentSelectors":["item-links"],"selector":".websiteLinksBlock a","multiple":false,"regex":"","delay":0},{"id":"email-address","type":"SelectorText","parentSelectors":["item-links"],"selector":".alternateEmailsBlock a","multiple":true,"regex":"","delay":0},{"id":"websiteurl2","type":"SelectorLink","parentSelectors":["item-links"],"selector":"a[itemprop='sameAs']","multiple":false,"delay":0}]}

#2

The reason for this is that there are duplicate companies in the list. And the 143 is a unique results count, as the scraper will not enter a URL more than once, leaving you with 'incomplete' list.

list
list.png1024×585 188 KB

#3

Thank you, that makes sense, I will check it out manually

#4

I have re-written the query, which gives me the results I need with out having to access each item, this now returns 212 results (with duplicates) which when I de-dupe brings me back to around 142. Also, significantly faster and probably draws less attention.

Thanks for highlighting the duplicates issue.

If any one is interested in the code:
{"_id":"thomson2","startUrl":["https://www.thomsonlocal.com/search/commercial-cleaning-companies/west-midlands?page=[1-9]"],"selectors":[{"id":"name","type":"SelectorText","parentSelectors":["grouping"],"selector":"h2.businessName","multiple":false,"regex":"","delay":0},{"id":"number","type":"SelectorText","parentSelectors":["grouping"],"selector":"div.phoneCont","multiple":false,"regex":"","delay":0},{"id":"site","type":"SelectorElementAttribute","parentSelectors":["grouping"],"selector":"li.listingHeadLink.website a","multiple":false,"extractAttribute":"href","delay":0},{"id":"grouping","type":"SelectorElement","parentSelectors":["_root"],"selector":"li.listing","multiple":true,"delay":0},{"id":"city","type":"SelectorText","parentSelectors":["grouping"],"selector":" span[itemprop='addressLocality']","multiple":false,"regex":"","delay":0},{"id":"street","type":"SelectorText","parentSelectors":["grouping"],"selector":"span[itemprop='streetAddress']","multiple":false,"regex":"","delay":0},{"id":"postcode","type":"SelectorText","parentSelectors":["grouping"],"selector":"span[itemprop='postalCode']","multiple":false,"regex":"","delay":0},{"id":"business-type","type":"SelectorText","parentSelectors":["_root"],"selector":".leftColumn > h2","multiple":false,"regex":"","delay":0}]}

#5