Describe the problem.
I'm trying to limit the number of profiles I'm pulling in with the scraper. Let's say I only want to copy data from profiles with 10,000+ views, how would I create that limit?
Url: https://www.behance.net/search?content=users&field=visual%20effects&country=US&sort=views&time=all
Sitemap:
https://www.behance.net/search?content=users&field=visual%20effects&country=US&sort=views&time=all
I don't know how to avoid scraping profiles below that threshold, but you could use a regex to only grab the numbers which meet your threshold. If it does not, then WS will fill in "null" in that field.
The regex is: (\d{2,}k)|(\d{2,}.\dk)
which will grab text like 10k, 11.2k, 32k etc. It will ignore text like 8900, 9k, 8.9k etc. The sitemap below illustrates. I also grabbed the likes and ran it thru the same regex, so you can see more "null" fields. Later you can use Excel or Google Sheets to filter/sort out the "null" ones.
{"_id":"behance_test_views_filter","startUrl":["https://www.behance.net/search?content=users&field=visual%20effects&country=US&sort=views&time=all"],"selectors":[{"id":"people_selector","type":"SelectorElement","parentSelectors":["_root"],"selector":"div.ProfileRow-userDetails-3UN","multiple":true,"delay":0},{"id":"name","type":"SelectorText","parentSelectors":["people_selector"],"selector":"a.ProfileRow-profileLink-2Iv","multiple":false,"regex":"","delay":0},{"id":"location","type":"SelectorText","parentSelectors":["people_selector"],"selector":"p","multiple":false,"regex":"","delay":0},{"id":"skills","type":"SelectorText","parentSelectors":["people_selector"],"selector":"ul","multiple":false,"regex":"","delay":0},{"id":"views","type":"SelectorText","parentSelectors":["people_selector"],"selector":".ProfileRow-views-3OK span","multiple":false,"regex":"(\\d{2,}k)|(\\d{2,}\\.\\dk)","delay":0},{"id":"likes","type":"SelectorText","parentSelectors":["people_selector"],"selector":".ProfileRow-appreciations-1es span","multiple":false,"regex":"(\\d{2,}k)|(\\d{2,}\\.\\dk)","delay":0}]}