How to scrape from popup box?

How to?
#1

Hi everyone,

I am trying to scrape a government website which consists of multiple sections of "div.pr-link". Upon clicking, a popup box that contains more content shows up. I am trying to scrape content in these popup boxes. I need web scraper to click each "div.pr-link", scrape each item inside the popup box, then move on to next box and repeat the process. It seems I need to use both "Element Scroll" and "Element Click" for this task. But I am not sure how. Any advice from the community is greatly appreciated!

Url: Narcotics Division, Security Bureau - Beat Drugs Fund - Funding Exercise - Regular Funding Scheme Project Search Results

#2

Hi,

All the data visible in the popups is already rendered in the HTML, so there is no need to perform any clicks.

See a reference sitemap below:

{"_id":"nd-gov-hk","startUrl":["https://www.nd.gov.hk/en/Project_Detail.php?p=1&S=5"],"selectors":[{"id":"listing-wrapper","multiple":true,"parentSelectors":["pagination"],"selector":".results_list .popup","type":"SelectorElement"},{"id":"title","multiple":false,"parentSelectors":["listing-wrapper"],"regex":"","selector":".item-title","type":"SelectorText"},{"id":"Approved Grant","multiple":false,"parentSelectors":["listing-wrapper"],"regex":"","selector":"tr:nth-of-type(2) td:nth-of-type(2)","type":"SelectorText"},{"id":"Description","multiple":false,"parentSelectors":["listing-wrapper"],"regex":"","selector":"p","type":"SelectorText"},{"id":"pagination","paginationType":"auto","parentSelectors":["_root","pagination"],"selector":"li.active + li a","type":"SelectorPagination"}]}
1 Like
#3

Hi JanAp,

Thank you so much for the reply! I have tried scraping using your reference sitemap, and it works! You see, I am not from an IT background and know nothing about html programming. I am just curious where you got all the selectors "li.active + li a", "p", "tr:nth-of-type(2) td:nth-of-type(2)", ".item-title", ".results_list .popup". As I was doing it, I just followed online tutorials where we were asked to click the select button and then click on the webpage - and not once did I got the selectors above just by clicking. Could you provide more clues, for people with non-IT background, on where to look for the selectors?

Thanks again!!

#4

Yeah, sometimes the point-and-click does not provide the optimal result and it is necessary to inspect the HTML. This will require some basic understanding of HTML and CSS. If you are interested in learning more just google for basic tutorials :+1:

1 Like
#5

Got it! Thank you JanAp :smiling_face_with_three_hearts:. I appreciate your kind help. I will follow your advice and teach myself some basic HTML.