Scraper fails to scrape items despite identifying them properly

How to?
#1

Web Scraper version: 0.6.5
Chrome version: 109.0.5414.120 (Build oficial) (64 bits)
OS: Windows 10

This problem is happening to me quite often as of lately. The issue is as follows:

  1. While setting up the sitemap the selectors are working fine. The Element preview button highlights the items I have chosen and the data preview button delivers everything as expected. The sitemap is correctly set up. You can see it in the picture attached.

  2. When I hit scrape, the new window opens and after a while it closes but no data has been scraped at all.

I am providing a sitemap tryng to scrape items from a Medium list, but I have sufffered it in other websites as well. However, this is only happening on some sites and thankfully not always. This seems to be happening only with Link selectors.

I am providing the sitemap as well as screenshot of DevTools, where no error is thrown. I can provide more sitemaps for this issue if you require them.

I have tried this on Firefox and it fails just the same.

Sitemap:

{"_id":"medium","startUrl":["https://medium.com/@bstulberg"],"selectors":[{"id":"title","multiple":true,"parentSelectors":["_root"],"selector":".jj .l a","type":"SelectorLink"}]}

WS1
WS11280×705 92.3 KB

WS2
WS21360×734 294 KB

#2

@Pablo_Gonzalez Hi, it appears that this most likely happens because the selectors are based on auto-generated classes which consist of random letters and numbers. In most cases, they are used as an anti-scraping mechanism and they are regenerated on a daily/weekly basis.

You can test this by clearing the cookies/cache of your browser and selecting the 'Element preview' button which most likely will return 0 value - no selectors matched.

In order to circumvent this issue, you would have to specify the selectors manually using more 'stable' elements.

example
example2143×1137 405 KB

Example:

{"_id":"medium","startUrl":["https://medium.com/@bstulberg"],"selectors":[{"id":"title","multiple":true,"parentSelectors":["_root"],"selector":"a[aria-label=\"Post Preview Title\"]","type":"SelectorLink"}]}