Element click selector stops

Hi y'all,

for a research regarding gym trends in Germany, I need to scrape the website, phone number and of course the name of gyms from a few sites.
Everything is going great, but with this particular site there is a problem.
There are 1514 gyms on the page, but it always stops at 501 gyms - so I guess there is a problem with the "load more" - button.
I also increased the delay up to 10000ms but it still won't work.
Any help would be so great!

URL: https://qualitrain.net/studios/jobs/?fwp_categories=krafttraining

Sitemap:

{"_id":"studios","startUrl":["https://qualitrain.net/studios/jobs/?fwp_categories=krafttraining"],"selectors":[{"id":"card","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"div.grid__item","multiple":true,"delay":"10000","clickElementSelector":"button.fwp-load-more","clickType":"clickMore","discardInitialElements":true,"clickElementUniquenessType":"uniqueText"},{"id":"phone","type":"SelectorText","parentSelectors":["card"],"selector":"a.listing-contact.listing--phone","multiple":false,"regex":"","delay":0},{"id":"Links","type":"SelectorLink","parentSelectors":["card"],"selector":"a.listing-contact.listing--website","multiple":false,"delay":0}]}

Hello there!

It stops because your browser starts to lag greatly, that lag may take more than 10 seconds for data to appear.

There is an easy solution just to hide listings from website -- so the browser won't have to render them and cause such lag. You have to use either CSS live editor like Stylebot / Stylizer or Tampermonkey to temporarily hide listing items from screen (but not from DOM tree, as the data is being actually loaded).

The CSS code to hide all listings from screen:
div.grid__item {display:none!important;}

Now to the actual sitemap. It seems you've missed the most important part -- Link selector to go through listings to pick Phone and Website, I've fixed it for you.

Your sitemap:

{"_id":"studios-fix","startUrl":["https://qualitrain.net/studios/jobs/?fwp_categories=krafttraining"],"selectors":[{"id":"card","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"div.grid__item","multiple":true,"delay":"2000","clickElementSelector":"button.fwp-load-more","clickType":"clickMore","discardInitialElements":true,"clickElementUniquenessType":"uniqueText"},{"id":"Links","type":"SelectorLink","parentSelectors":["card"],"selector":"a","multiple":true,"delay":0},{"id":"Phone","type":"SelectorText","parentSelectors":["Links"],"selector":"a.listing-contact.listing--phone","multiple":false,"regex":"","delay":0},{"id":"Website","type":"SelectorText","parentSelectors":["Links"],"selector":"a.listing-contact.listing--website","multiple":false,"regex":"","delay":0}]}

@iconoclast
Thank you so so much, and sorry for my lack of knowledge! I truly appreciate your help - great to see that there still are helpful and kind people like you. Best regards from Germany!

1 Like