Data showing in preview, not in scrape

How to?
#1

When I use "Data Preview" on my wrapper, everything I'm trying to grab looks perfect. But when I run the actual scraper, a lot of data in the "percent" column is missing. This is a multipage site The scraper is correctly paginating, and I've confirmed it's grabbing all the listings. It's just not capturing around 95% of the data in the "percent" child.

Url: https://clutch.co/agencies/media-buying/albuquerque%2523collapseFilterForm%23chart-label-carousel-377762%23chart-label-carousel-642340?page=1

Sitemap:
{"_id":"clutch_page_2_on","startUrl":["https://clutch.co/agencies/media-buying/albuquerque%2523collapseFilterForm%23chart-label-carousel-377762%23chart-label-carousel-642340?page=[1-155]"],"selectors":[{"id":"product-wrapper","type":"SelectorElement","parentSelectors":["_root"],"selector":"div[data-clutch-nid]","multiple":true,"delay":0},{"id":"name","type":"SelectorText","parentSelectors":["product-wrapper"],"selector":".field-content a","multiple":false,"regex":"","delay":0},{"id":"percent","type":"SelectorText","parentSelectors":["product-wrapper"],"selector":".directory-main-bar div.chart-label","multiple":false,"regex":"","delay":0},{"id":"url","type":"SelectorLink","parentSelectors":["product-wrapper"],"selector":"a.sl-ext","multiple":false,"delay":0}]}

#2

Hi your scraper structure is fine. However, this site uses lazy loading for the "percent" info, which means you need to scroll down to reveal all the info. That's why the scraper was only getting a fraction of the data; it could not "see" the rest which was not loaded yet.

So if you change your main selector ("product-wrapper") to a scroller, that should fix the issue:
Type: Element scroll down
Selector: same as before
Multiple: Yes
Delay: 2000 (this is the default, which is fine)

1 Like
#3

Thank you so much! This worked perfectly!