Please help. Is there any way to limit "load more" click element selector? :)

How to?
#1

Hey guys, I'm trying to scrape e-commerce page with total about 8,000 items. When I tried to scrape it, my browser crash because my laptop spec isn't that good. So is there a way to limit the click? Let's say 50 or 100 clicks. The only option I see is one click, or multiple click until there's no new element.

The website I want to scrape use "load more" button. Can you guys help me please? Thank you

Url: http://eurekabookhouse(dot)co(dot()id/eureka/cari/result?manufacturer=buku-erlangga

Sitemap:
{"_id":"eureka-erlangga","startUrl":["http://eurekabookhouse.co.id/eureka/cari/result?manufacturer=buku-erlangga"],"selectors":[{"id":"items","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"div#postList.row","multiple":true,"delay":0,"clickElementSelector":"button.show_load_more","clickType":"clickMore","discardInitialElements":false,"clickElementUniquenessType":"uniqueText"},{"id":"link","type":"SelectorLink","parentSelectors":["items"],"selector":"div.block2-txt-child1 a.stext-105","multiple":true,"delay":0},{"id":"Judul Buku","type":"SelectorText","parentSelectors":["link"],"selector":"h4.ctext-101","multiple":false,"regex":"","delay":0},{"id":"Kode Buku","type":"SelectorText","parentSelectors":["link"],"selector":"p.stext-101:nth-of-type(1) a.stext-103","multiple":false,"regex":"","delay":0},{"id":"Kategori Eureka","type":"SelectorText","parentSelectors":["link"],"selector":"p.stext-101:nth-of-type(4) a.stext-103","multiple":false,"regex":"","delay":0},{"id":"Harga","type":"SelectorText","parentSelectors":["link"],"selector":"span.ctext-101 small","multiple":false,"regex":"","delay":0},{"id":"Harga Eureka","type":"SelectorText","parentSelectors":["link"],"selector":"span.ctext-101 span.cl13","multiple":false,"regex":"","delay":0},{"id":"Diskon Eureka","type":"SelectorText","parentSelectors":["link"],"selector":"span.badge","multiple":false,"regex":"","delay":0},{"id":"Stok All","type":"SelectorText","parentSelectors":["link"],"selector":"div.size-204","multiple":false,"regex":"","delay":0},{"id":"Stok Kosong","type":"SelectorText","parentSelectors":["link"],"selector":"div.flex-w span.cl13","multiple":false,"regex":"","delay":0},{"id":"Stok Tersedia","type":"SelectorText","parentSelectors":["link"],"selector":"div.size-204 span.cl1","multiple":false,"regex":"","delay":0},{"id":"Detail Produk","type":"SelectorText","parentSelectors":["link"],"selector":"div.ctext-101 p:nth-of-type(2)","multiple":false,"regex":"","delay":0},{"id":"Deskripsi - p1","type":"SelectorText","parentSelectors":["link"],"selector":"div.ctext-101 p:nth-of-type(3)","multiple":false,"regex":"","delay":0},{"id":"All Deskripsi","type":"SelectorText","parentSelectors":["link"],"selector":"div.ctext-101","multiple":false,"regex":"","delay":0},{"id":"All Deskripsi v2","type":"SelectorText","parentSelectors":["link"],"selector":"div.panel-body","multiple":false,"regex":"","delay":0},{"id":"Gambar","type":"SelectorImage","parentSelectors":["link"],"selector":"div.bk-cover img.lazy","multiple":false,"delay":0}]}

Limit "load more" click selector
#2

Hello there!

Unfortunately the number of times to click is not implemented yet, but there is somewhat good solution.

If you look into the website code using Developer tools, you will notice that button has an ID number changing when you click it, and it goes bottom up (numbers go backwards).

image
image.png1173×991 266 KB

What we can do about it is to have CSS selector to catch only range that is suitable for us.
That can be done using :not selector. We also have to calculate the number of items we want, but it's easier to approximate it to hundreds, for example.

You will have to analyze how much items are getting shown once you click Show More button, and also how much does ID value change accordingly.

For example, if you set Element Click selector to button.show_load_more:not([id^=1450]) it will keep clicking Show More button until it's ID does not begin with '1450' in it's o number (think of 1450 as of a text/number mask). I've had to limit it to hundreds value because it's ending both odd and even.

Please note that you will have to tune it for any different page than what you have brought.

2 Likes
#3

After trying it few times. The solution you gave me works like a charm. Thank you so much for taking your time, really appreciate it! :slight_smile:

#4

Wow that's an awesome tip. I'm gonna try it soon.