Two scroll elements

How to?
#1

I'm scraping data from a property website. The problem is that the page has two scroll elements - one "nested" inside the other.

So, what's happening is that the scraper scrolls the "first" scroll element, and scrapes data so loaded. But to add the rest of the data the "nested" scroll doesn't happen - so the data hidden there is not scraped.

How can I fix that?

Url: https://patriotfamilyhomes.guestybookings.com/properties/

Sitemap:
{"_id":"patriotfamhomes","startUrl":["https://patriotfamilyhomes.guestybookings.com/properties/"],"selectors":[{"delay":3000,"elementLimit":500,"id":"property","multiple":true,"parentSelectors":["_root"],"selector":"div.root-0-2-488","type":"SelectorElementScroll"},{"id":"Name","multiple":false,"parentSelectors":["property","property2"],"regex":"","selector":"div.TextField-colorInput-0-2-257","type":"SelectorText"},{"id":"Where","multiple":false,"parentSelectors":["property","property2"],"regex":"","selector":".mt-2 div","type":"SelectorText"},{"id":"Price","multiple":false,"parentSelectors":["property","property2"],"regex":"","selector":".TextField-h2-0-2-246 span","type":"SelectorText"},{"id":"Type","multiple":false,"parentSelectors":["property","property2"],"regex":"","selector":"span[data-qa='property-type']","type":"SelectorText"},{"id":"Guests","multiple":false,"parentSelectors":["property","property2"],"regex":"","selector":"span[data-qa='property-guests-label']","type":"SelectorText"},{"id":"Bed","multiple":false,"parentSelectors":["property","property2"],"regex":"","selector":"span[data-qa='property-bedroom-label']","type":"SelectorText"},{"id":"Bath","multiple":false,"parentSelectors":["property","property2"],"regex":"","selector":"span[data-qa='property-bathroom-label']","type":"SelectorText"},{"id":"Pic","multiple":false,"parentSelectors":["property","property2"],"selector":"a img","type":"SelectorImage"},{"clickActionType":"real","clickElementSelector":"div.propertyListWrapper-0-2-278","clickElementUniquenessType":"uniqueText","clickType":"clickOnce","delay":2000,"discardInitialElements":"discard-when-click-element-exists","id":"click","multiple":false,"parentSelectors":["_root"],"selector":"div.propertyListWrapper-0-2-278","type":"SelectorElementClick"},{"delay":4000,"elementLimit":500,"id":"property2","multiple":true,"parentSelectors":["_root"],"selector":"div.root-0-2-488","type":"SelectorElementScroll"},{"clickActionType":"real","clickElementSelector":"div.propertyListWrapper-0-2-278","clickElementUniquenessType":"uniqueText","clickType":"clickOnce","delay":4500,"discardInitialElements":"discard-when-click-element-exists","id":"click2","multiple":false,"parentSelectors":["_root"],"selector":"div.propertyListWrapper-0-2-278","type":"SelectorElementClick"},{"delay":6000,"elementLimit":500,"id":"property3","multiple":true,"parentSelectors":["_root"],"selector":"div.root-0-2-488","type":"SelectorElementScroll"}]}

Thank you in advance for any help,

Chrislee

#2

This is a bit of a tricky site, with its two scrollers. Let's call them the outer scroller and the inner scroller. To deal with the outer scroller you only need to zoom out your browser view until it disappears. Don't worry about the tiny text; the scraper does not care about font size.

image
image1452×938 129 KB

The inner scroller is trickier and you'll need to use the undocumented/under-documented scrollElementSelector selector. This one needs to be manually added via Json code (you cannott add it with the Gui). Even with this trick, the scroller seems to halt quite quickly. In the example sitemap below, I have added 4 scrollers, and this gets me about 250 results in my tests:

{"_id":"patriotfamilyhomes-test","startUrl":["https://patriotfamilyhomes.guestybookings.com/properties/"],"selectors":[{"id":"Inner scroll 1","parentSelectors":["_root"],"type":"SelectorElementScroll","selector":"div.overflow-y-scroll > div[class^='root']","multiple":false,"delay":4200,"scrollElementSelector":"div.overflow-y-scroll","elementLimit":300},{"id":"Inner scroll 2","parentSelectors":["_root"],"type":"SelectorElementScroll","selector":"div.overflow-y-scroll > div[class^='root']:nth-of-type(n+61)","multiple":false,"delay":4200,"scrollElementSelector":"div.overflow-y-scroll","elementLimit":500},{"id":"Inner scroll 3","parentSelectors":["_root"],"type":"SelectorElementScroll","selector":"div.overflow-y-scroll > div[class^='root']:nth-of-type(n+121)","multiple":false,"delay":4200,"scrollElementSelector":"div.overflow-y-scroll","elementLimit":600},{"id":"Inner scroll 4","parentSelectors":["_root"],"type":"SelectorElementScroll","selector":"div.overflow-y-scroll > div[class^='root']:nth-of-type(n+181)","multiple":false,"delay":4200,"scrollElementSelector":"div.overflow-y-scroll","elementLimit":700},{"id":"Result elements","parentSelectors":["_root"],"type":"SelectorElement","selector":"div.overflow-y-scroll > div[class^='root']","multiple":true},{"id":"Title","parentSelectors":["Result elements"],"type":"SelectorText","selector":"div[class^='titleWrapper']","multiple":false,"regex":""},{"id":"Type","parentSelectors":["Result elements"],"type":"SelectorText","selector":"span[data-qa='property-type']","multiple":false,"regex":""},{"id":"Price","parentSelectors":["Result elements"],"type":"SelectorText","selector":"div.row-wrapper:contains('Per') span:contains('$') ","multiple":false,"regex":""},{"id":"Location","parentSelectors":["Result elements"],"type":"SelectorText","selector":"div[data-qa='property-address']","multiple":false,"regex":""},{"id":"Bedrooms","parentSelectors":["Result elements"],"type":"SelectorText","selector":"span[data-qa='property-bedroom-label']","multiple":false,"regex":""},{"id":"Bathrooms","parentSelectors":["Result elements"],"type":"SelectorText","selector":"span[data-qa='property-bathroom-label']","multiple":false,"regex":""},{"id":"Desc","parentSelectors":["Result elements"],"type":"SelectorText","selector":"div[data-qa='property-summary']","multiple":false,"regex":""},{"id":"Link","parentSelectors":["Result elements"],"type":"SelectorLink","selector":"div[class^='propertyImage'] a","multiple":false,"linkType":"linkFromHref"}]}

Ref: Scroller does not work on certain websites

#3

Is there a way to go about this if u cannot zoom out to get the scroll to disappear?

#4

Hello,

If you send me the site that you are trying to scrape, as well as which scroll you are having trouble with, I can try to help you. Let me know what information you are trying to get to.

#5

Here is the post where i explained the problem.
Link