How to scrape data from website that does not allow direct access to the link without executing a click?

How to?
#1

Hi, I have an issue with scraping data from this website, because this website does not allow direct access to the link without executing a click.

I try to use the link selector but the website does not allow it.

here's the URL: https://lpse.lkpp.go.id/eproc4/lelang?kategoriId=&tahun=2022&instansiId=&rekanan=&kontrak_status=&kontrak_tipe=

this is my sitemaps:
{"_id":"LKPP","startUrl":["[https://lpse.lkpp.go.id/eproc4/lelang?kategoriId=&tahun=2022&instansiId=&rekanan=&kontrak_status=&kontrak_tipe="],"selectors":{"id":"pagination","parentSelectors":["_root","pagination"],"paginationType":"clickMore","selector":".next a","type":"SelectorPagination"},{"id":"Nama Paket","parentSelectors":["_root","pagination"],"type":"SelectorLink","selector":"td p a","multiple":true,"linkType":"linkFromHref"},{"id":"Tab Pemenang Berkontrak","parentSelectors":["Nama Paket"],"type":"SelectorLink","selector":"li:nth-of-type(5) a","multiple":false,"linkType":"linkFromHref"},{"id":"Detail Pemenang Berkontrak","parentSelectors":["Tab Pemenang Berkontrak"],"type":"SelectorText","selector":"td","multiple":true,"regex":""}]}

Thank you very much.

#2

This site is tricky but can be scraped if you use some "hacks". There are two problems you need to solve:

  1. The site is aggressive in using popups and WS can't handle those.
  2. The site also checks for "Referer", so you cannot just load the tender details pages.

You can solve problem 1 by getting a list of tender details page URLs from the Nama Paket column. The URLs all look something like
https://lpse.lkpp.go.id/eproc4/lelang/8870119/pengumumanlelang
but at this point, if you paste it into browser or use it as StartURL, it won't work and the site will show an error page.
You will need to solve problem 2, which is lack of referer. You can do this with an extension like Referer Control. You should configure it with these settings:
Site filter: Add https://lpse.lkpp.go.id
Referer setting: Click Custom -> target host

image
image1361×1344 114 KB

After you have installed and configured Referer Control. You should be able to load and scrape the tender details page URLs.