I have a site that scrapes fine. Some pages have a "load more" button for these I use element click more option. Until yesterday the delay option worked fine. The scraper waited for 2 seconds before the next element click. But now it wait 10-12 seconds even though I did not change the delay options.
It could be a general issue or it is a special problem with my scraper?
Thanks,
Dani
Can you share the sitemap?
The delay could have increased if the site loads some additional data after the click. Basically it would have to load something for at least 10 seconds.
The scraper require Facebook login.
Here it is:
{"_id":"fb","startUrl":["https://www.facebook.com/ufi/reaction/profile/browser/?ft_ent_identifier=1405584952881073"],"selectors":[{"id":"more","type":"SelectorElementClick","selector":"ul#reaction_profile_browser1.uiList div.clearfix","parentSelectors":["_root"],"multiple":true,"delay":"2000","clickElementSelector":"div#reaction_profile_pager1.clearfix a.pam","clickType":"clickMore","discardInitialElements":true,"clickElementUniquenessType":"uniqueText"},{"id":"name","type":"SelectorLink","selector":"div._5j0e a","parentSelectors":["more"],"multiple":false,"delay":0}]}
Is there any idea for fixing this problem?
Your selector for element click selector is invalid. Remove the class from it and it should work fine.