Help, only last page is scraped with element click

How to?
#1

I'm trying to scrape a page that uses javascript for pagination. I've used element click to traverse between pages. Then I've used link action to go into each separate pa ge and get some extra info.

It all works good The element click goes trough all pages. But at the end of scraping process only the last page (9th) is scraped for the information.

Am I doing it wrong or is it impossible to scrape?
Does any have any ideas on how to go about it?

Thanks!

Url: https://www.bizi.si/iskanje?q=spletna%20trgovina

Sitemap:
{"_id":"bizi","startUrl":["https://www.bizi.si/iskanje?q=%20trgovina%20živali"],"selectors":[{"id":"pagination","type":"SelectorElementClick","parentSelectors":["_root"],"selector":".bg-full","multiple":true,"delay":"5000","clickElementSelector":"#divTopInfoRow li.clean.right a.pibtn","clickType":"clickMore","discardInitialElements":false,"clickElementUniquenessType":"uniqueText"},{"id":"item","type":"SelectorLink","parentSelectors":["pagination"],"selector":"td.main > a","multiple":true,"delay":0},{"id":"naslov","type":"SelectorText","parentSelectors":["item"],"selector":"h1","multiple":false,"regex":"","delay":0}]}

#2

Anybody has any idea on how to fix this?

#3

Hi there!

Your sitemap runs all fine on my side, I've uploaded CSV file for you here.

Have you tried increasing Page Load Delay and Request Interval up to 3000 ms? I have them at 3000 / 3000 ms by default.

What WebScraper version you're currently using?

#4

Hi.

thanks for the CSV. I need some more data though :
It's really weird cause I've made a couple of other scrapers before and they've worked OK. But this I just can't get this one to work.
I've set the delays even longer than 3s but still the same.

I have 0.3.8 version of WebScraper and 69.0.3497.100 version of Chrome.

Any other ideas maybe :slight_smile:

#5

Post extensions you have running while you scrape, try disabling some of them.

#6

I've tried in incognito mode (enabled WebScraper there) but it was the same.
Running extensions are these:

image
image.png1600×900 143 KB

#7

Please disable your AdBlock (not the Plus version). If you have filters added in both of the AdBlocks, please try grouping them all together into one (the Plus version, for example).

I can recommend you to install Disconnect extension (it's like Ghostery, to protect against tracking, but more effective IMO).

P.S. you can also try running it in WebScraper DEV version here (version 0.3.8.3):

#8

Thanks for all your help. Will try all of it tomorrow and let you know how it goes.

#9

Hi,

thanks for your help the DEV version seems to be working fine :slight_smile:

Cheers!