I´m looking to the municipal lower chamber member expenses.
Usually, do sites are not helpful but this one seemed quite straightforward on how to get the data.
Though I cannot make, by any means, for the pagination work properly.
Usually scrapes the first and second page. But the second page not always is page 2, some times is page 3, 5, or 6, depending on the attempt.
I have already tried to use links, element click, and pagination, without success.
So the first thing is this, any idea on how to make the pagination work to scrape all pages?
The second thing is if is possible to get these links:
I just need the URL, but it calls a script that opens the page.
Many thanks for the help!
Url with less pagination: http://transparencia.jundiai.sp.leg.br/portal_transparencia/Pages/Geral/wfDespesa.aspx?id=Des010120202901202002050100000000000000000000
Sitemap:
{"_id":"camara-jundiai","startUrl":["http://transparencia.jundiai.sp.leg.br/portal_transparencia/Pages/Geral/wfDespesa.aspx?id=Des010120202912202002050100000000000000000000"],"selectors":[{"clickElementSelector":"a.dxp-num","clickElementUniquenessType":"uniqueText","clickType":"clickOnce","delay":2000,"discardInitialElements":"do-not-discard","id":"gastos","multiple":true,"parentSelectors":["_root"],"selector":".dxgvTable tr.dxgvDataRow","type":"SelectorElementClick"},{"delay":0,"id":"cnpj/cpf","multiple":false,"parentSelectors":["gastos"],"regex":"","selector":"td:nth-of-type(4)","type":"SelectorText"},{"delay":0,"id":"fornecedor","multiple":false,"parentSelectors":["gastos"],"regex":"","selector":"td:nth-of-type(5)","type":"SelectorText"},{"delay":0,"id":"modalidade_licitacao","multiple":false,"parentSelectors":["gastos"],"regex":"","selector":"td:nth-of-type(6)","type":"SelectorText"},{"delay":0,"id":"valor","multiple":false,"parentSelectors":["gastos"],"regex":"","selector":"td:nth-of-type(9)","type":"SelectorText"},{"delay":0,"id":"id_empenho","multiple":false,"parentSelectors":["gastos"],"regex":"","selector":"td:nth-of-type(3)","type":"SelectorText"}]}