Unable to scrape this site ! Please help, I'm new here 😀

[JuanDav11]

Hello everyone, I'm new here, I'm trying to scrape the name of some companies in a database, the web is relatively "easy" to scrape, but I can't get more names than the first 25, which is the preview, the only thing that What you have to do to load more names is to go down on the web and they will load, but I have not succeeded. Can somebody help me?? In order to obtain all the names in the database, you have to create an account, but you can do it with an email and password, you don't have to enter more information.

Url: https://app.dealroom.co/companies.startups/f/business_models/anyof_manufacturing/company_status/not_closed/company_type/not_service%20provider_government%20nonprofit/growth_stages/not_mature/has_website_url/anyof_yes/industries/anyof_media/job_types/anyof_Marketing%20%26%20Communication_Project%20Management_Finance/tags/not_outside%20tech?showGrid=true

Sitemap:
{"_id":"prueba","startUrl":["https://app.dealroom.co/companies.startups/f/business_models/anyof_manufacturing/company_status/not_closed/company_type/not_service%20provider_government%20nonprofit/growth_stages/not_mature/has_website_url/anyof_yes/industries/anyof_media/job_types/anyof_Marketing%20%26%20Communication_Project%20Management_Finance/tags/not_outside%20tech?showGrid=true"],"selectors":[{"id":"scroll","parentSelectors":["_root"],"type":"SelectorElementScroll","selector":"a.entity-overview","multiple":true,"delay":500,"elementLimit":0},{"id":"scoll","parentSelectors":["scroll"],"type":"SelectorElementScroll","selector":"parent","multiple":true,"delay":500,"elementLimit":0},{"id":"name","parentSelectors":["scoll"],"type":"SelectorText","selector":"h3","multiple":false,"regex":""}]}

[ViestursWS]

@JuanDav11

et more names than the first 25, which is the preview, the only thing that What you have to do to load more names is to go down on the web and they will load, but I have not succeeded. Can somebody help me?? In order to obtain all the names in the data

@JuanDav11 Hello, after inspecting the website it appears that it is set to transform the rendered listings. This is considered a web development practice to improve the performance of the page, however, it also serves as an anti-scraping mechanism. After scrolling down the page the previously rendered items are replaced with newly discovered items.

You can test this by zooming out the page as far as you can and clicking on the 'Element preview' button. The selected element count will never correspond to the actual count. This is something that can be affected by zooming out the page as far as you can, however, that will not guarantee that all of the available listings will be scraped.

[JuanDav11]

Ok perfect thank you very much.
But that applies regardless of the type of view you have on the web page (list or cards). And using "pop up link" would it change something or would that not matter?

[ViestursWS]

@JuanDav11 It seems that you should not experience this issue within the 'Landscape' view mode.

Example:

{"_id":"prueba-landscape","startUrl":["https://app.dealroom.co/companies.startups/f/business_models/anyof_manufacturing/company_status/not_closed/company_type/not_service%20provider_government%20nonprofit/growth_stages/not_mature/has_website_url/anyof_yes/industries/anyof_media/job_types/anyof_Marketing%20%26%20Communication_Project%20Management_Finance/tags/not_outside%20tech?showGrid=true"],"selectors":[{"id":"scroll","multiple":true,"parentSelectors":["_root"],"selector":"a.landscape-entity-icon","type":"SelectorElement"},{"id":"title","multiple":false,"parentSelectors":["scroll"],"regex":"","selector":"_parent_","type":"SelectorText"},{"id":"link","multiple":false,"parentSelectors":["scroll"],"selector":"_parent_","type":"SelectorLink"}]}