Auto Scroll Issue

How to?
#1

Describe the problem.
Hi all, I'm having an issue trying to scrape one of our suppliers websites. We're a furniture retailer and traditionally have to manually copy the data from our suppliers sites and manually add to our site. This took an absolute age.... until i found this app. Honestly, we've added new lines to the site in days instead of weeks, its brilliant.

Anyway, I'm waffling on.... one supplier uses an auto scroll on their site, to load the next page of products. Ive tried the Element Scroll option but just can't get the site to move onto next page unless i manually scroll with mouse.

I've quickly added a map that simply selects the product name from the product page

Any ideas?

Url: https://www.premierhousewares.com

Sitemap:
{"_id":"permier","startUrl":["https://www.premierhousewares.com/furniture.html?id=1465#page=1"],"selectors":[{"id":"scroller","type":"SelectorElementScroll","selector":"div.col-lg-9","parentSelectors":["_root"],"multiple":true,"delay":0},{"id":"product link","type":"SelectorLink","selector":"ul.products-grid.first a.product-image","parentSelectors":["scroller"],"multiple":true,"delay":0},{"id":"header","type":"SelectorText","selector":"div.product-name h2","parentSelectors":["product link"],"multiple":false,"regex":"","delay":0}]}

#2

Hello,
not sure that this test could help you, but try to look at what I found to retrieve your partner's site references:

{"_id":"test","startUrl":["https://www.premierhousewares.com/furniture.html?id=1465#page=[1-4]"],"selectors":[{"id":"item","type":"SelectorElementScroll","selector":"div.col-lg-9","parentSelectors":["_root"],"multiple":true,"delay":"2000"},{"id":"title","type":"SelectorText","selector":"div.amscroll-page h2.product-name a, ul.products-grid.first h2.product-name a","parentSelectors":["item"],"multiple":true,"regex":"","delay":0},{"id":"sku","type":"SelectorText","selector":"ul.products-grid.first div.sku-prod-lis-hr, div.amscroll-page div.sku-prod-lis-hr","parentSelectors":["item"],"multiple":false,"regex":"","delay":0}]}

#3

Hi... adding a delay worked but not quite right. The site scraped which ever page was on screen after the pause.

#4

Adding the page count to end of source worked, cheers buddy

#5

Hello,
what do you mean by "adding the page count ... worked" ???
Could you please send your ultimate sitemaps to see what has changed.
Thank's

#6

{"_id":"premiertest","startUrl":["https://www.premierhousewares.com/furniture.html#page=[1-147]"],"selectors":[{"id":"Main Grid Select","type":"SelectorElementScroll","selector":"div.col-lg-9","parentSelectors":["_root"],"multiple":true,"delay":"2000"},{"id":"Product Link","type":"SelectorLink","selector":"ul.products-grid.first a.product-image, div.amscroll-page:nth-of-type(n+3) a.product-image","parentSelectors":["Main Grid Select"],"multiple":true,"delay":0},{"id":"header","type":"SelectorText","selector":"div.product-name h2","parentSelectors":["Product Link"],"multiple":false,"regex":"","delay":0},{"id":"SKU","type":"SelectorText","selector":"tr.first:contains('SKU') td.data","parentSelectors":["Product Link"],"multiple":false,"regex":"","delay":0},{"id":"Range","type":"SelectorText","selector":"tr.even:contains('Range') td.data","parentSelectors":["Product Link"],"multiple":false,"regex":"","delay":0},{"id":"Weight","type":"SelectorText","selector":"tr.odd:contains('Weight Kg') td.data","parentSelectors":["Product Link"],"multiple":false,"regex":"","delay":0},{"id":"Brand","type":"SelectorText","selector":"tr.even:contains('Brand') td.data","parentSelectors":["Product Link"],"multiple":false,"regex":"","delay":0},{"id":"Dimensions","type":"SelectorText","selector":"tr.even:contains('Product Dimensions') td.data","parentSelectors":["Product Link"],"multiple":false,"regex":"","delay":0},{"id":"Colour","type":"SelectorText","selector":"tr.last:contains('Colour') td.data","parentSelectors":["Product Link"],"multiple":false,"regex":"","delay":0},{"id":"Price","type":"SelectorText","selector":"span.price span.price","parentSelectors":["Product Link"],"multiple":false,"regex":"","delay":0},{"id":"qty","type":"SelectorText","selector":"p.availability span","parentSelectors":["Product Link"],"multiple":false,"regex":"","delay":0},{"id":"Details","type":"SelectorText","selector":"div.resp-tab-content div.std","parentSelectors":["Product Link"],"multiple":false,"regex":"","delay":0},{"id":"IM1","type":"SelectorImage","selector":"img.gallery-image.visible","parentSelectors":["Product Link"],"multiple":false,"delay":0},{"id":"minidesc","type":"SelectorText","selector":"div.short-description div.std","parentSelectors":["Product Link"],"multiple":false,"regex":"","delay":0},{"id":"assembly","type":"SelectorText","selector":"tr.even:contains('Assembly Info') td.data","parentSelectors":["Product Link"],"multiple":false,"regex":"","delay":0}]}

#7

Im now struggling to get the images to download from the popup, but the core data is there now

web-scraper-order web-scraper-start-url Product Link Product Link-href header SKU Range Weight Brand Dimensions Colour Price qty Details IM1-src minidesc assembly
1523321086-3323 https://www.premierhousewares.com/furniture.html#page=147 https://www.premierhousewares.com/bar-stool-2402776.html Bar Stool 2402776 null 13.7 null H94 x W41 x D41cm null £36.50 158 In Stock Sleek lines, smooth contours, the stylish bar stool appears to be influenced by the design of a high end sports car. Made with strong and reliable ABS plastic, there is no slowing down when it come to the design of the stool. The black ABS effortlessly curves creating a back rest that will support your lower body, while the crisp white accentuates the style of the stool. Complementing the black and white is the glistening chrome base and footrest that provides durability and stability for the chair. Knowing not all bars and tables are the same, the height can be adjusted with ease thanks to the gas lift design that also allows 360° rotation. Practical and stylish, the bar stool will complement a modern setting adding its own unique look to your kitchen or living area. https://www.premierhousewares.com/media/catalog/product/cache/1/image/470x420/9df78eab33525d08d6e5fb8d27136e95/2/4/2402776.jpg Sleek lines, smooth contours, the stylish bar stool has appears to be influenced by the design of a high end sports car. Made with strong and reliable ABS plastic, there is no slowing down when it come to the design of the stool. null