I'm using a web scraper to crawl the information of a social networking site called Jike. I crawled the content of the homepage of this account. It is a page that loads by scrolling down, but only part of the content can be crawled each time. The crawling ends before the rest of the content is loaded. And I am also unable to set the delay or wait attribute of selectors. May I ask how to solve this problem?
Url: 即刻
Sitemap:
{"_id":"Qamar","startUrl":["THE URL ABOVE"],"selectors":[{"id":"Trending","parentSelectors":["_root"],"type":"SelectorElement","selector":"div.jk-1i3egjr","multiple":true,"scroll":true,"elementLimit":null},{"id":"content_short","parentSelectors":["Trending"],"type":"SelectorText","selector":"div.jk-1i93uyw","multiple":false,"regex":""},{"id":"date","parentSelectors":["Trending"],"type":"SelectorText","selector":"div.jk-1pi78hi","multiple":false,"regex":""},{"id":"image_dynamic","parentSelectors":["Trending"],"type":"SelectorImage","selector":"img.live-photo-cover","multiple":true},{"id":"video","parentSelectors":["Trending"],"type":"SelectorImage","selector":"video","multiple":false},{"id":"image_static","parentSelectors":["Trending"],"type":"SelectorImage","selector":"img.jk-17d300y","multiple":true},{"id":"content_long","parentSelectors":["Trending"],"type":"SelectorText","selector":"div.jk-1i93uyw","multiple":false,"regex":""}]}
