Multiple products, single page, product details in popup, and load more auto scroll

[jamie]

Hi all,

I've got a use case which is currently beyond my capabilities. I want to scrape product data (title, description, price, image URL) from the following URL, which is a product page containing multiple products, with some details (like description & full size image) only available once clicking on a product image to load a popup.

Click It Local

The page also loads the products via autoscroll, which is another factor to consider.

There's a finally level of complexity (which could be ignored, but ideally I'd scrape it) which is some products that have size/weight selectors which alter the price of the product. I'd like to pull each variety as a separate line item:

I'm pretty sure this can be done with this tool, I just don't know how to create it! I presume via element selector, or element click selector, but I can't work it out

Help appreciated, thanks!
Jamie

[Remona_Ayoub]

Hi Jamie,
have you watched the pagination tutorial, https://www.webscraper.io/tutorials this explains how to scrape multiple products from one or more pages. I just did a test run and it worked for me, happy to walk you through it if you need help.
As for the pop up, after creating the sitemap for the single page, create a mutiple link selector and click on all the products on that one page you want to scrape, then save it.
Then click into that link you've created, also click into the product as well so you can set up all the fields you want to scrape from the product page. Once you've selected all the data from the main page, click on the pop up and scrape the fields you want from that screen as text.
ie. create a text selector for size1, then select 150g, save it. create a second text selector called price1, select that price and save. then on the page click on the second size to bring up the price on the web page, create a selector called size 2, then select 250g and then another selector for price 2, then select that price. if you want the name and description from the pop up you can create separate selectors for those too.

Hope this helps, let me know how you go.

[jamie]

Hi Remona, thanks for getting back to me

I think a walk through would be helpful. The products are all on one page and load more via scrolling down the page, so I think I need to use an element scroll down selector somewhere to get all of the products showing before I can scrape them.

Then unless I'm mistaken I can't use a link selector to select all the products on the page because the products don't click through to new unique URLs for each product - they open the product in an on-page popup on the same URL, which is why I think I need to use element selector/element click - but this I don't understand how to configure properly. To clarify, the products do not have individual pages. Everything is loaded on the one start URL.

I recorded a quick video to highlight this: https://youtu.be/RLU3lhkKzKo

A practical walkthrough would be very helpful!

[Remona_Ayoub]

Hi Jamie,

You are definitely on the right track, I went to the same site and had a play around with it but i'm getting stuck with configuring the child of the parent, when i try to select from the pop up it returns an error msg saying this is not a child of the parent. I think you may have to reach out to their support team for some guidance.

I just had another crack at it and was able to map the below, the scraper showed that it went through the products but it didn't capture any data, but it was opening all the different product pop ups. I got that far, maybe you can have a look and fine tune it. I'm no expert, sorry, hopefully another member can assist.

Export of sitemap: (import it, scrape it and check out what i mean)

{"_id":"url","startUrl":["https://www.clickitlocal.co.uk/cambridge/?Stores=1&Products=0&LocationID=96&CategoryID=0&DeliveryTypeID=0&StartRow=1&OrderBy=&Search="],"selectors":[{"id":"element","type":"SelectorElementScroll","parentSelectors":["_root"],"selector":".product-summary-block img","multiple":true,"delay":2000},{"id":"element-click","type":"SelectorElementClick","parentSelectors":["element-click"],"selector":"parent","multiple":false,"delay":2000,"clickElementSelector":"parent","clickType":"clickOnce","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueText"},{"id":"popup","type":"SelectorPopupLink","parentSelectors":["element"],"selector":"parent","multiple":true,"delay":0},{"id":"title","type":"SelectorText","parentSelectors":["popup"],"selector":"div.product-title","multiple":false,"regex":"","delay":0},{"id":"description","type":"SelectorText","parentSelectors":["popup"],"selector":"div.product-description","multiple":false,"regex":"","delay":0},{"id":"image","type":"SelectorImage","parentSelectors":["popup"],"selector":".no-swiping.swiper-slide-active img","multiple":false,"delay":0},{"id":"size1","type":"SelectorText","parentSelectors":["popup"],"selector":"button.attribute-box-with-2","multiple":false,"regex":"","delay":0},{"id":"price1","type":"SelectorText","parentSelectors":["popup"],"selector":"div.product-price","multiple":false,"regex":"","delay":0},{"id":"size2","type":"SelectorText","parentSelectors":["popup"],"selector":"button.attribute-box-with-4","multiple":false,"regex":"","delay":0},{"id":"price2","type":"SelectorText","parentSelectors":["popup"],"selector":"div.product-price","multiple":false,"regex":"","delay":0},{"id":"delivery","type":"SelectorText","parentSelectors":["popup"],"selector":".product-availability strong","multiple":false,"regex":"","delay":0}]}

[leemeng]

This site is tricky becase the popups are loaded from another section of the page (another branch), hence the error. You will need to use a very broad selector like body. You also need to close the popups each time. Modify sitemap below as needed:

{"_id":"forum-clickitlocal-test","startUrl":["https://www.clickitlocal.co.uk/east-london/?Stores=1&Products=0&LocationID=133&CategoryID=0&DeliveryTypeID=0&StartRow=1&OrderBy&Search"],"selectors":[{"id":"Click only within All products section","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"body","multiple":true,"delay":"3500","clickElementSelector":".product-summary-block img","clickType":"clickOnce","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueCSSSelector"},{"id":"Product","type":"SelectorText","parentSelectors":["Click only within All products section"],"selector":"div.product-title","multiple":false,"regex":"" },{"id":"From","type":"SelectorText","parentSelectors":["Click only within All products section"],"selector":".store-name u","multiple":false,"regex":"" },{"id":"Price","type":"SelectorText","parentSelectors":["Click only within All products section"],"selector":"div.product-price","multiple":false,"regex":"" },{"id":"Close the popup","type":"SelectorElementClick","parentSelectors":["Click only within All products section"],"selector":".show .close img","multiple":false,"delay":"1000","clickElementSelector":".show .close img","clickType":"clickOnce","discardInitialElements":"discard","clickElementUniquenessType":"uniqueText"}]}