Does not scrape my full selection, stops at 4 out of 64

How to?
#1

Hi,

I have 64 product listings selected in a top selector.
I have set 8 sub-selectors for these 64 listings, such as price, product description, and image…

When I run the "Scrape," it works fine (fetches all the requested information) but systematically stops at the fourth listing. I have tried multiple times, even starting from scratch and redoing the full selector setup, and there are still only 4 out of 64.

To make it fully transparent, I shot this video where you can see my setup and the results.

I hope someone can help me solve this.

Thanks in advance.

#2

For some reason the video does not display correctly here above.
So here is the link to it: https://www.screencast.com/t/4jHsjqfdoFq

#3

Hi,

Please post the sitemap, I will have a look at it.

#4

By posting the sitemap, are you referring to this?

{"_id":"AbstractPrintDownload","startUrl":["Abstract print download - Etsy Links","parentSelectors":["_root"],"type":"SelectorLink","selector":"a.logged","multiple":true,"linkType":"linkFromHref"},{"id":"Title","parentSelectors":["Single Links"],"type":"SelectorText","selector":"h1.wt-line-height-tight","multiple":false,"regex":""},{"id":"List Price","parentSelectors":["Single Links"],"type":"SelectorText","selector":".wt-display-flex-xs .wt-text-gray span.wt-text-strikethrough","multiple":false,"regex":""},{"id":"Discount","parentSelectors":["Single Links"],"type":"SelectorText","selector":".wt-mb-xs-1 p.wt-text-slime","multiple":false,"regex":""},{"id":"SalesPrice","parentSelectors":["Single Links"],"type":"SelectorText","selector":"p.wt-text-title-larger","multiple":false,"regex":""},{"id":"InCart","parentSelectors":["Single Links"],"type":"SelectorText","selector":"p.wt-text-brick","multiple":false,"regex":""},{"id":"Description","parentSelectors":["Single Links"],"type":"SelectorHTML","selector":"p.wt-text-body-01.wt-break-word","multiple":false,"regex":""},{"id":"Picture","parentSelectors":["Single Links"],"type":"SelectorImage","selector":"img.wt-horizontal-center[alt='Bold Abstract Wall Art Colorful Print Large Modern Art Print Large Printable Art Contemporary Wall Decor Multicolor Art Digital Abstract Art image 1']","multiple":false},{"id":"Tags","parentSelectors":["Single Links"],"type":"SelectorText","selector":".wt-display-flex-lg div:nth-of-type(8) div:nth-of-type(1)","multiple":false,"regex":""}]}

#5

Yes, but please post it as preformatted text, otherwise the code is broken.

code

#6

There you are!

{"_id":"AbstractPrintDownload","startUrl":["https://www.etsy.com/search?q=abstract+print+download&ref=auto-1&as_prefix=abstract+print+d"],"selectors":[{"id":"Single Links","parentSelectors":["_root"],"type":"SelectorLink","selector":"a.logged","multiple":true,"linkType":"linkFromHref"},{"id":"Title","parentSelectors":["Single Links"],"type":"SelectorText","selector":"h1.wt-line-height-tight","multiple":false,"regex":""},{"id":"List Price","parentSelectors":["Single Links"],"type":"SelectorText","selector":".wt-display-flex-xs .wt-text-gray span.wt-text-strikethrough","multiple":false,"regex":""},{"id":"Discount","parentSelectors":["Single Links"],"type":"SelectorText","selector":".wt-mb-xs-1 p.wt-text-slime","multiple":false,"regex":""},{"id":"SalesPrice","parentSelectors":["Single Links"],"type":"SelectorText","selector":"p.wt-text-title-larger","multiple":false,"regex":""},{"id":"InCart","parentSelectors":["Single Links"],"type":"SelectorText","selector":"p.wt-text-brick","multiple":false,"regex":""},{"id":"Description","parentSelectors":["Single Links"],"type":"SelectorHTML","selector":"p.wt-text-body-01.wt-break-word","multiple":false,"regex":""},{"id":"Picture","parentSelectors":["Single Links"],"type":"SelectorImage","selector":"img.wt-horizontal-center[alt='Bold Abstract Wall Art Colorful Print Large Modern Art Print Large Printable Art Contemporary Wall Decor Multicolor Art Digital Abstract Art image 1']","multiple":false},{"id":"Tags","parentSelectors":["Single Links"],"type":"SelectorText","selector":".wt-display-flex-lg div:nth-of-type(8) div:nth-of-type(1)","multiple":false,"regex":""}]}
#7

The issue occurs because the listings are only rendered after a scroll down. You check this by refreshing the page and clicking on the element preview for the links selector.

Here is a reference sitemap with an element scroll setup:

{"_id":"AbstractPrintDownload","startUrl":["https://www.etsy.com/search?q=abstract+print+download&ref=auto-1&as_prefix=abstract+print+d"],"selectors":[{"id":"Single Links","linkType":"linkFromHref","multiple":true,"parentSelectors":["listing-wrapper"],"selector":"a.logged","type":"SelectorLink"},{"id":"Title","multiple":false,"parentSelectors":["Single Links"],"regex":"","selector":"h1.wt-line-height-tight","type":"SelectorText"},{"id":"List Price","multiple":false,"parentSelectors":["Single Links"],"regex":"","selector":".wt-display-flex-xs .wt-text-gray span.wt-text-strikethrough","type":"SelectorText"},{"id":"Discount","multiple":false,"parentSelectors":["Single Links"],"regex":"","selector":".wt-mb-xs-1 p.wt-text-slime","type":"SelectorText"},{"id":"SalesPrice","multiple":false,"parentSelectors":["Single Links"],"regex":"","selector":"p.wt-text-title-larger","type":"SelectorText"},{"id":"InCart","multiple":false,"parentSelectors":["Single Links"],"regex":"","selector":"p.wt-text-brick","type":"SelectorText"},{"id":"Description","multiple":false,"parentSelectors":["Single Links"],"regex":"","selector":"p.wt-text-body-01.wt-break-word","type":"SelectorHTML"},{"id":"Picture","multiple":false,"parentSelectors":["Single Links"],"selector":"img.wt-horizontal-center[alt='Bold Abstract Wall Art Colorful Print Large Modern Art Print Large Printable Art Contemporary Wall Decor Multicolor Art Digital Abstract Art image 1']","type":"SelectorImage"},{"id":"Tags","multiple":false,"parentSelectors":["Single Links"],"regex":"","selector":".wt-display-flex-lg div:nth-of-type(8) div:nth-of-type(1)","type":"SelectorText"},{"delay":0,"elementLimit":500,"id":"listing-wrapper","multiple":true,"parentSelectors":["_root"],"selector":".tab-reorder-container li","type":"SelectorElementScroll"}]}
#8

Hi Jan,

When I check the element preview, it shows 64 selected items but it ends up scraping only 4!!
Please see it live here 2024-04-10_12-38-21

How should I set it to scroll?

#9 
video link here : https://www.screencast.com/t/iY8TStHZ0yX
#10

please check the sitemap in my previous post

#11

I saw the sitemap in your previous post and I see it has some parts mine does not have but I do not understand what to do with it nor how to generate one with the same setup as yours.

Could you be a bit more specific as to the steps to take to remedy the situation?
Thanks in advance.

#12

You have to import the sitemap in the Web Scraper extension:

image

#13

Thanks Jan. It worked fine.

So just to recap what I learned from this.
If I want to create another fully sitemap for another scrape.
I need to set up the structure as root\listing-wrapper\Single Links, correct?

#14

For this site, sure, you can use this as a base. Any other website will have a different structure of course.

Also, you might want to add pagination to scrape all items, not just the current page.

{"_id":"AbstractPrintDownload-pagination","startUrl":["https://www.etsy.com/search?q=abstract+print+download&ref=auto-1&as_prefix=abstract+print+d"],"selectors":[{"id":"Single Links","linkType":"linkFromHref","multiple":true,"parentSelectors":["listing-wrapper"],"selector":"a.logged","type":"SelectorLink"},{"id":"Title","multiple":false,"parentSelectors":["Single Links"],"regex":"","selector":"h1.wt-line-height-tight","type":"SelectorText"},{"id":"List Price","multiple":false,"parentSelectors":["Single Links"],"regex":"","selector":".wt-display-flex-xs .wt-text-gray span.wt-text-strikethrough","type":"SelectorText"},{"id":"Discount","multiple":false,"parentSelectors":["Single Links"],"regex":"","selector":".wt-mb-xs-1 p.wt-text-slime","type":"SelectorText"},{"id":"SalesPrice","multiple":false,"parentSelectors":["Single Links"],"regex":"","selector":"p.wt-text-title-larger","type":"SelectorText"},{"id":"InCart","multiple":false,"parentSelectors":["Single Links"],"regex":"","selector":"p.wt-text-brick","type":"SelectorText"},{"id":"Description","multiple":false,"parentSelectors":["Single Links"],"regex":"","selector":"p.wt-text-body-01.wt-break-word","type":"SelectorHTML"},{"id":"Picture","multiple":false,"parentSelectors":["Single Links"],"selector":"img.wt-horizontal-center[alt='Bold Abstract Wall Art Colorful Print Large Modern Art Print Large Printable Art Contemporary Wall Decor Multicolor Art Digital Abstract Art image 1']","type":"SelectorImage"},{"id":"Tags","multiple":false,"parentSelectors":["Single Links"],"regex":"","selector":".wt-display-flex-lg div:nth-of-type(8) div:nth-of-type(1)","type":"SelectorText"},{"id":"pagination","paginationType":"auto","parentSelectors":["_root","pagination"],"selector":"a:contains(\"Next\"):visible","type":"SelectorPagination"},{"delay":0,"elementLimit":500,"id":"listing-wrapper","multiple":true,"parentSelectors":["pagination"],"selector":".tab-reorder-container li","type":"SelectorElementScroll"}]}
#15

thanks a lot for your help