Help me please to scrap data

How to?
#1

Untitled
Untitled.png1337×663 45.4 KB

anyone can help me, before this i can scrap data "pemenang", but when i run by click scrape, data "pemenang" not create. its text format.

i scrap from site http://lpse.karokab.go.id/eproc4/lelang and the trouble run when its collect data "pemenang". and create null data.

its my work before
{"_id":"tes","startUrl":["http://lpse.karokab.go.id/eproc4/lelang"],"selectors":[{"id":"link nama paket","type":"SelectorLink","parentSelectors":["_root"],"selector":"tr:nth-of-type(n+2) p a","multiple":true,"delay":0},{"id":"Kode Tender","type":"SelectorText","parentSelectors":["link nama paket"],"selector":"tr:contains('Kode Tender') strong","multiple":false,"regex":"","delay":0},{"id":"Nama Tender","type":"SelectorText","parentSelectors":["link nama paket"],"selector":"tr:contains('Nama Tender') strong","multiple":false,"regex":"","delay":0},{"id":"Sumber Dana","type":"SelectorText","parentSelectors":["link nama paket"],"selector":"tr:contains('Rencana Umum Pengadaan') td:nth-of-type(3)","multiple":false,"regex":"","delay":0},{"id":"Tahap Tender Saat Ini","type":"SelectorText","parentSelectors":["link nama paket"],"selector":"td a","multiple":false,"regex":"","delay":0},{"id":"Instansi","type":"SelectorText","parentSelectors":["link nama paket"],"selector":"tr:contains('Instansi') td","multiple":false,"regex":"","delay":0},{"id":"Satker","type":"SelectorText","parentSelectors":["link nama paket"],"selector":"tr:contains('Satuan Kerja') td","multiple":false,"regex":"","delay":0},{"id":"Kategori","type":"SelectorText","parentSelectors":["link nama paket"],"selector":"tr:contains('Kategori') td","multiple":false,"regex":"","delay":0},{"id":"Tahun Anggaran","type":"SelectorText","parentSelectors":["link nama paket"],"selector":"tr:contains('Tahun Anggaran') td","multiple":false,"regex":"","delay":0},{"id":"Nilai Pagu Paket","type":"SelectorText","parentSelectors":["link nama paket"],"selector":"tr:nth-of-type(12) td:nth-of-type(1)","multiple":false,"regex":"","delay":0},{"id":"Nilai HPS Paket","type":"SelectorText","parentSelectors":["link nama paket"],"selector":".table-condensed.table-bordered > tbody > tr > td:nth-of-type(2)","multiple":false,"regex":"","delay":0},{"id":"Lokasi Pekerjaan","type":"SelectorText","parentSelectors":["link nama paket"],"selector":"td li","multiple":false,"regex":"","delay":0},{"id":"Kualifikasi Usaha","type":"SelectorText","parentSelectors":["link nama paket"],"selector":"tr:contains('Kualifikasi Usaha') td","multiple":false,"regex":"","delay":0},{"id":"Pemenang","type":"SelectorElementClick","parentSelectors":["link nama paket"],"selector":"li:nth-of-type(4) a","multiple":false,"delay":0,"clickElementSelector":"li:nth-of-type(4) a","clickType":"clickOnce","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueText"},{"id":"Nama Pemenang","type":"SelectorText","parentSelectors":["Pemenang"],"selector":"td td:nth-of-type(1)","multiple":false,"regex":"","delay":0},{"id":"Alamat Pemenang","type":"SelectorText","parentSelectors":["Pemenang"],"selector":"strong","multiple":false,"regex":"","delay":0},{"id":"NPWP","type":"SelectorText","parentSelectors":["Pemenang"],"selector":"td:nth-of-type(3)","multiple":false,"regex":"","delay":0},{"id":"Harga Penawaran","type":"SelectorText","parentSelectors":["Pemenang"],"selector":"td:nth-of-type(4)","multiple":false,"regex":"","delay":0},{"id":"Harga Terkoreksi","type":"SelectorText","parentSelectors":["Pemenang"],"selector":"td:nth-of-type(5)","multiple":false,"regex":"","delay":0},{"id":"Hasil Negosiasi","type":"SelectorText","parentSelectors":["Pemenang"],"selector":"td:nth-of-type(6)","multiple":false,"regex":"","delay":0}]}

please correct for pagination too....

thanks for your help...

#2

Hi, the "Pemenang" button is actually a clickable link (the URL changes) so you can just use Type: Link to make things easier. I've created a basic sitemap from scratch and you can refer to it to modify yours:

{"_id":"lpse_test","startUrl":["http://lpse.karokab.go.id/eproc4/lelang"],"selectors":[{"id":"Click link Nama Paket","type":"SelectorLink","parentSelectors":["_root"],"selector":"tr p a","multiple":true,"delay":0},{"id":"Kode Tender\t","type":"SelectorText","parentSelectors":["Click link Nama Paket"],"selector":"tr:contains('Kode Tender') strong","multiple":false,"regex":"","delay":0},{"id":"Name Tender","type":"SelectorText","parentSelectors":["Click link Nama Paket"],"selector":"tr:contains('Nama Tender') strong","multiple":false,"regex":"","delay":0},{"id":"Click Pemenang","type":"SelectorLink","parentSelectors":["Click link Nama Paket"],"selector":"li:nth-of-type(4) a","multiple":false,"delay":0},{"id":"Nama Pemenang","type":"SelectorText","parentSelectors":["Click Pemenang"],"selector":"td td:nth-of-type(1)","multiple":false,"regex":"","delay":0},{"id":"Alamat","type":"SelectorText","parentSelectors":["Click Pemenang"],"selector":"td:nth-of-type(2)","multiple":false,"regex":"","delay":0},{"id":"NPWP","type":"SelectorText","parentSelectors":["Click Pemenang"],"selector":"td:nth-of-type(3)","multiple":false,"regex":"","delay":0},{"id":"Harga Penawaran\t","type":"SelectorText","parentSelectors":["Click Pemenang"],"selector":"td:nth-of-type(4)","multiple":false,"regex":"","delay":0}]}

For test scrape I used Page load delay (ms): 6000

For pagination, you should refer to:
Web Scraper pagination tutorial

The LPSE site is the third type, "page loads data dynamically from the server" which is around 3:59

1 Like
#3

thanks mr lee... i'll try....

#4

i try with your sitemap use create new sitemap>impor sitemap. but "no data scraped yet". i use 6000ms . it stop when data pemenang not display at web.
example : http://lpse.karokab.go.id/eproc4/evaluasi/2820516/pemenang and http://lpse.karokab.go.id/eproc4/evaluasi/2823516/pemenang

what this aplication can't skip blank data?
for pagination its after or before "Click link Nama Paket" ?