Stuck always at 120 Scraped Contacts

How to?
#1

Hey team,

I’m trying to scrape contact data for dental clinics in Portugal, but I’ve run into an issue: the scraping process always stops at exactly 120 contacts, no matter which search page I use.

Here’s an example of a URL I’m scraping: Pesquisa - Páginas Amarelas

And here’s the sitemap I’m using: {"_id":"PaiPTscraper","startUrl":["https://www.pai.pt/searches?search%5Bquery%5D=dentist&search%5Bne%5D=42.1541%2C-6.4969&search%5Bsw%5D=32.2099%2C-29.8597&search%5Bcenter%5D=&search%5Bmap%5D=&search%5Blocation_id%5D=1&search%5Bcategory_id%5D=&search%5Btag_id%5D=&search%5Bgroup_id%5D=&search%5Blocation_value%5D=Portugal&search%5Blocation%5D=Portugal&search%5Bopen%5D=0&search%5Bcategories%5D%5B%5D=425&search%5Border%5D=relevance"],"selectors":[{"id":"Paganation","parentSelectors":["_root","Paganation"],"paginationType":"auto","type":"SelectorPagination","selector":".next a"},{"id":"product link","parentSelectors":["Paganation"],"type":"SelectorLink","selector":"a.card-link","multiple":true,"linkType":"linkFromHref"},{"id":"address","parentSelectors":["product link"],"type":"SelectorText","selector":"li:nth-of-type(1) div","multiple":false,"regex":""},{"id":"phone","parentSelectors":["product link"],"type":"SelectorText","selector":"div.desktop","multiple":false,"regex":""},{"id":"website","parentSelectors":["product link"],"type":"SelectorLink","selector":"a[data-trackable-event='visit-webpage']","multiple":false,"linkType":"linkFromHref"},{"id":"email 1","parentSelectors":["product link"],"type":"SelectorHTML","selector":"li.listing-item:nth-of-type(4)","multiple":false,"regex":""},{"id":"email 2","parentSelectors":["product link"],"type":"SelectorText","selector":"li.listing-item:nth-of-type(5)","multiple":false,"regex":""},{"id":"Full details","parentSelectors":["product link"],"type":"SelectorHTML","selector":"ul.listing-details","multiple":false,"regex":""},{"id":"Time","parentSelectors":["product link"],"type":"SelectorText","selector":"div.not-mobile-xl","multiple":false,"regex":""}]}

Any idea how can I fix this?

Thanks in advance!

#2

image
image938×697 124 KB

As you are filtering there total 120 contacts

#3

Captura de ecrã 2025-07-20, às 11.23.49
Captura de ecrã 2025-07-20, às 11.23.491920×1198 109 KB

No bro, even in searches with 200, 300 and 500 results, it scrapes only 120

#4

Try this one with page range

{"_id":"PaiPTscraper_page_range","startUrl":["https://www.pai.pt/searches?search[query]=dentist&search[ne]=41.0801%2C-8.0892&search[sw]=40.2788%2C-8.7833&search[center]=&search[map]=&search[location_id]=20205&search[category_id]=&search[tag_id]=&search[group_id]=&search[location_value]=Aveiro&search[location]=Aveiro&commit=Procurar&page=[1-20]"],"selectors":[{"id":"product link","linkType":"linkFromHref","multiple":true,"parentSelectors":["_root"],"selector":"a.card-link","type":"SelectorLink"},{"id":"address","multiple":false,"parentSelectors":["product link"],"regex":"","selector":"li:nth-of-type(1) div","type":"SelectorText"},{"id":"phone","multiple":false,"parentSelectors":["product link"],"regex":"","selector":"div.desktop","type":"SelectorText"},{"id":"website","linkType":"linkFromHref","multiple":false,"parentSelectors":["product link"],"selector":"a[data-trackable-event='visit-webpage']","type":"SelectorLink"},{"extractAttribute":"href","id":"email 1","multiple":false,"parentSelectors":["product link"],"selector":"[href*=\"mailto\"]","type":"SelectorElementAttribute"},{"id":"email 2","multiple":false,"parentSelectors":["product link"],"regex":"","selector":"li.listing-item:nth-of-type(5)","type":"SelectorText"},{"id":"Full details","multiple":false,"parentSelectors":["product link"],"regex":"","selector":"ul.listing-details","type":"SelectorHTML"},{"id":"Time","multiple":false,"parentSelectors":["product link"],"regex":"","selector":"div.not-mobile-xl","type":"SelectorText"}]}

#5

Thank you for trying to help me, Jakirhasan

I imported the sitemap, ran the scrape, and it managed to get 141 results out of the 199 available

Captura de ecrã 2025-07-20, às 12.03.55
Captura de ecrã 2025-07-20, às 12.03.553338×820 144 KB

So it’s no longer stuck at 120 results, which is great but it still didn’t scrape all the available results

Do you have any idea what might be causing this?

#6

This is due to duplicate links, which are automatically discarded.

#7

Really? So everything is fine?

I am not sure because my friend scrapped the same page but he got way more scrapped contacts

#8

The different results are due to the random order in which the results are loaded on the website. If you go to the second page, then switch back to the first and go to the second again, you will see different listings. This means that the same listings can be displayed again on different pages.

#9

Yeah, you’re right. I didn’t even notice that

Is there any solution for it?

#10

I am afraid no, this is a glitch on the website

#11

Alright, man. Thanks!