Extract Data from Unusual Website Style

#1

Need help with scaping all the exhibitors from the below website:

Url: https://www.agritechnica.com/en/exhibitors-products#/

The sitemap I have now is as below:

{"_id":"agritechnica","startUrl":["[https://www.agritechnica.com/en/exhibitors-products#/"],"selectors":{"id":"Link ","type":"SelectorPopupLink","parentSelectors":["_root"],"selector":"div.LMA5IJ-v-O","multiple":true,"delay":0},{"id":"Company","type":"SelectorText","parentSelectors":["Link "],"selector":"div[itemprop='legalName']","multiple":true,"regex":"","delay":0},{"id":"Address","type":"SelectorText","parentSelectors":["Link "],"selector":"div[itemprop='address']","multiple":true,"regex":"","delay":0},{"id":"Website","type":"SelectorText","parentSelectors":["Link "],"selector":"a[itemprop='url']","multiple":true,"regex":"","delay":0},{"id":"Email","type":"SelectorText","parentSelectors":["Link "],"selector":"div.LMA5IJ-x-mb:nth-of-type(4) a","multiple":true,"regex":"","delay":0},{"id":"Phone","type":"SelectorText","parentSelectors":["Link "],"selector":"div[itemprop='telephone']","multiple":true,"regex":"","delay":0},{"id":"Description","type":"SelectorText","parentSelectors":["Link "],"selector":".LMA5IJ-x-q div.gwt-HTML","multiple":true,"regex":"","delay":0}]}

This is not including the inclusion of a scroll feature to capture the "show more" elements - but even the rudimentary model without it currently not working, which I am confused by.

I have made the main link selector type a "Link popup", the other selectors ie company name address "Text".

Any advise is appreciated.

#2

The page does not require a scroll selector, as even though the elements seem to render only after a scroll if you inspect the page, the only change that appears to happen is the elements change their style and not content itself.

Also, the pop-up's are not really pop-up links, but just regular JavaScript, so they require an Element Click selector to click on each one and render them.

The sitemap should look something like this:

{"_id":"agritechnica-click","startUrl":["https://www.agritechnica.com/en/exhibitors-products#/"],"selectors":[{"id":"show-more","type":"SelectorElementClick","parentSelectors":["_root"],"selector":".listContentContainer > div:has(a)","multiple":true,"delay":"1500","clickElementSelector":".onlineGuideDefaultStyles div div div div div > div:contains(\"Show more\")","clickType":"clickMore","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueCSSSelector"},{"id":"card-click","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"body","multiple":true,"delay":2000,"clickElementSelector":".listContentContainer > div:has(a)","clickType":"clickOnce","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueCSSSelector"},{"id":"name","type":"SelectorText","parentSelectors":["element"],"selector":"div[itemprop='legalName']","multiple":false,"regex":"","delay":0},{"id":"address","type":"SelectorText","parentSelectors":["element"],"selector":"div[itemprop='address']","multiple":false,"regex":"","delay":0},{"id":"element","type":"SelectorElement","parentSelectors":["card-click"],"selector":".popupContent","multiple":true,"delay":0}]}

The only caveat to this sitemap is that there is a whole lot of JavaScript that has to be executed, which might result in your memory filling up and crashing the extension. If this happens, you are probably better off setting each letter as a separate start URL in your sitemap which will considerably cut down on the time the Click Selector has to work continuously.

#3

Thanks so much for your help - really really appreciate it.