Dropdown is selected with Element Click but does not update the page

Hey,

I need help making the Dropdown selection work.
I´m usiling element click. it changes the selector status but does not reflects on adding more items to its section.

is there anything that I´m missing or that I should try?

If I select the dropdown manually it works, though when loading a new congressman the last section "number of items" dropdown resets.
So doing manually does seem an option since I would have to oversee the whole process to manually select each time it changes to another congressman.

{"_id":"camara_alerj_v3-3","startUrl":["DOCIGP - Descentralização Orçamentária - Alerj tbody tr","clickElementUniquenessType":"uniqueText","clickType":"clickOnce","delay":1000,"discardInitialElements":"discard","multiple":true,"selector":"div#vue-entries"},{"id":"data","parentSelectors":["lancamentos"],"type":"SelectorText","selector":"> div > div:nth-of-type(4) tbody tr.bg-primary-lighter td:nth-of-type(1)","multiple":false,"delay":0,"regex":""},{"id":"natureza","parentSelectors":["lancamentos"],"type":"SelectorText","selector":"> div > div:nth-of-type(4) tbody tr.bg-primary-lighter td:nth-of-type(2)","multiple":false,"delay":0,"regex":""},{"id":"sub-item","parentSelectors":["lancamentos"],"type":"SelectorText","selector":"> div > div:nth-of-type(4) tbody tr.bg-primary-lighter td:nth-of-type(2) small","multiple":false,"delay":0,"regex":""},{"id":"favorecido","parentSelectors":["lancamentos"],"type":"SelectorText","selector":"> div > div:nth-of-type(4) tbody tr.bg-primary-lighter td:nth-of-type(3)","multiple":false,"delay":0,"regex":""},{"id":"cnpj","parentSelectors":["lancamentos"],"type":"SelectorText","selector":"> div > div:nth-of-type(4) tbody tr.bg-primary-lighter td:nth-of-type(3) small","multiple":false,"delay":0,"regex":""},{"id":"valor","parentSelectors":["lancamentos"],"type":"SelectorText","selector":"> div > div:nth-of-type(4) tbody tr.bg-primary-lighter td:nth-of-type(5)","multiple":false,"delay":0,"regex":""},{"id":"clica_parla","parentSelectors":["parla_250"],"type":"SelectorElementClick","clickElementSelector":"div#vue-entries > div > div:nth-of-type(2) tbody tr","clickElementUniquenessType":"uniqueText","clickType":"clickOnce","delay":2000,"discardInitialElements":"discard","multiple":true,"selector":"div#app"},{"id":"clica_comp","parentSelectors":["comp_250"],"type":"SelectorElementClick","clickElementSelector":"div#vue-entries > div > div:nth-of-type(3) tbody tr","clickElementUniquenessType":"uniqueText","clickType":"clickOnce","delay":1000,"discardInitialElements":"discard","multiple":true,"selector":"div:has(> div > div > div#vue-entries)"},{"id":"documento","parentSelectors":["lancamentos"],"type":"SelectorLink","selector":"> div > div:nth-of-type(5) a.btn:first-child","multiple":false,"delay":0},{"id":"parlamentar","parentSelectors":["lancamentos"],"type":"SelectorText","selector":".col-md-9 small","multiple":false,"delay":0,"regex":""},{"id":"lancamento_250","parentSelectors":["clica_comp"],"type":"SelectorElementClick","clickElementSelector":"div#vue-entries > div > div:nth-of-type(4) select option[value="250"]","clickElementUniquenessType":"uniqueHTMLText","clickType":"clickOnce","delay":1500,"discardInitialElements":"discard","multiple":true,"selector":"div:has(> div#vue-entries)"},{"id":"comp_250","parentSelectors":["clica_parla"],"type":"SelectorElementClick","clickElementSelector":"div#vue-entries > div > div:nth-of-type(3) select option[value="250"]","clickElementUniquenessType":"uniqueText","clickType":"clickOnce","delay":2000,"discardInitialElements":"discard","multiple":true,"selector":"main"},{"id":"parla_250","parentSelectors":["_root"],"type":"SelectorElementClick","clickElementSelector":"div#vue-entries > div > div:nth-of-type(2) select option[value="250"]","clickElementUniquenessType":"uniqueText","clickType":"clickOnce","delay":2000,"discardInitialElements":"discard","multiple":true,"selector":"body"},{"id":"q_documentos","parentSelectors":["lancamentos"],"type":"SelectorText","selector":"> div > div:nth-of-type(4) tbody tr.bg-primary-lighter td:nth-of-type(4)","multiple":false,"delay":0,"regex":""}]}

@rcarvalheira Hi, could you, please, re-paste your sitemap(by applying the 'Preformatted text' option) as the JSON of it seem to be invalid?


sorry. sure.

{"_id":"camara_alerj_v3-3","startUrl":["https://docigp.alerj.rj.gov.br/transparencia#/"],"selectors":[{"id":"lancamentos","parentSelectors":["lancamento_250"],"type":"SelectorElementClick","clickElementSelector":"div:nth-of-type(4) tbody tr","clickElementUniquenessType":"uniqueText","clickType":"clickOnce","delay":1000,"discardInitialElements":"discard","multiple":true,"selector":"div#vue-entries"},{"id":"data","parentSelectors":["lancamentos"],"type":"SelectorText","selector":"> div > div:nth-of-type(4) tbody tr.bg-primary-lighter td:nth-of-type(1)","multiple":false,"delay":0,"regex":""},{"id":"natureza","parentSelectors":["lancamentos"],"type":"SelectorText","selector":"> div > div:nth-of-type(4) tbody tr.bg-primary-lighter td:nth-of-type(2)","multiple":false,"delay":0,"regex":""},{"id":"sub-item","parentSelectors":["lancamentos"],"type":"SelectorText","selector":"> div > div:nth-of-type(4) tbody tr.bg-primary-lighter td:nth-of-type(2) small","multiple":false,"delay":0,"regex":""},{"id":"favorecido","parentSelectors":["lancamentos"],"type":"SelectorText","selector":"> div > div:nth-of-type(4) tbody tr.bg-primary-lighter  td:nth-of-type(3)","multiple":false,"delay":0,"regex":""},{"id":"cnpj","parentSelectors":["lancamentos"],"type":"SelectorText","selector":"> div > div:nth-of-type(4) tbody tr.bg-primary-lighter  td:nth-of-type(3) small","multiple":false,"delay":0,"regex":""},{"id":"valor","parentSelectors":["lancamentos"],"type":"SelectorText","selector":"> div > div:nth-of-type(4) tbody tr.bg-primary-lighter  td:nth-of-type(5)","multiple":false,"delay":0,"regex":""},{"id":"clica_parla","parentSelectors":["parla_250"],"type":"SelectorElementClick","clickElementSelector":"div#vue-entries > div > div:nth-of-type(2) tbody tr","clickElementUniquenessType":"uniqueText","clickType":"clickOnce","delay":2000,"discardInitialElements":"discard","multiple":true,"selector":"div#app"},{"id":"clica_comp","parentSelectors":["comp_250"],"type":"SelectorElementClick","clickElementSelector":"div#vue-entries > div > div:nth-of-type(3) tbody tr","clickElementUniquenessType":"uniqueText","clickType":"clickOnce","delay":1000,"discardInitialElements":"discard","multiple":true,"selector":"div:has(> div > div > div#vue-entries)"},{"id":"documento","parentSelectors":["lancamentos"],"type":"SelectorLink","selector":"> div > div:nth-of-type(5) a.btn:first-child","multiple":false,"delay":0},{"id":"parlamentar","parentSelectors":["lancamentos"],"type":"SelectorText","selector":".col-md-9 small","multiple":false,"delay":0,"regex":""},{"id":"lancamento_250","parentSelectors":["clica_comp"],"type":"SelectorElementClick","clickElementSelector":"div#vue-entries > div > div:nth-of-type(4) select option[value=\"250\"]","clickElementUniquenessType":"uniqueHTMLText","clickType":"clickOnce","delay":1500,"discardInitialElements":"discard","multiple":true,"selector":"div:has(> div#vue-entries)"},{"id":"comp_250","parentSelectors":["clica_parla"],"type":"SelectorElementClick","clickElementSelector":"div#vue-entries > div > div:nth-of-type(3) select option[value=\"250\"]","clickElementUniquenessType":"uniqueText","clickType":"clickOnce","delay":2000,"discardInitialElements":"discard","multiple":true,"selector":"main"},{"id":"parla_250","parentSelectors":["_root"],"type":"SelectorElementClick","clickElementSelector":"div#vue-entries > div > div:nth-of-type(2) select option[value=\"250\"]","clickElementUniquenessType":"uniqueText","clickType":"clickOnce","delay":2000,"discardInitialElements":"discard","multiple":true,"selector":"body"},{"id":"q_documentos","parentSelectors":["lancamentos"],"type":"SelectorText","selector":"> div > div:nth-of-type(4) tbody tr.bg-primary-lighter td:nth-of-type(4)","multiple":false,"delay":0,"regex":""}]}

@rcarvalheira This particular case was already covered in this post - Scrap dynamic data with multiple clicks

Thanks.
Amazingly, another person was looking for help at the same site. I guess it is indeed a hard site to scrape.

My use case is a little different. After all, I gave up with the pagination because, since I´m going one level deeper and looking for the receipts, my iterations were taking too long.
So I decided to get rid of paginations by selecting more elements to be shown. Though this hasn't worked, that's the why of this post.

But while seeing your example, I guess I need to understand more about the use of discard and when to use each case in "uniqueness".
Since on my tests, my process was running twice every first page, otherwise was duplicating. While in your sitemap this didn´t happen.

Without running every first page twice, probably is time feasible to use paginations instead of my later approach.

I understand that in your example, for the pagination, you select the element after the active one, and thus use Click More instead of Click Once.

With that in mind, I reviewed my previous sitemap and tried to use the same concept.
Though I´m losing the first run of clicks on the third table (lançamentos / entries/numbers 5 and 6 on the image).
I filtered congressman names by "andr" and limited them to 5 entries so I could run it faster just to validate the solution.

Can you help me on solving the original question (why the "number of listed element" selector does not work) or whit this newer solution that um brought (why the third table pagination drops the first page)

Scraped content began from the second page even with "never discard"


My sitemap for this second question is the following:

{"_id":"camara_alerj_v3-2","startUrl":["https://docigp.alerj.rj.gov.br/transparencia#/"],"selectors":[{"id":"lancamentos","parentSelectors":["pagina_lancamento"],"type":"SelectorElementClick","clickElementSelector":"div:nth-of-type(4) tbody tr","clickElementUniquenessType":"uniqueCSSSelector","clickType":"clickOnce","delay":1000,"discardInitialElements":"do-not-discard","multiple":true,"selector":"div#vue-entries"},{"id":"data","parentSelectors":["lancamentos"],"type":"SelectorText","selector":"> div > div:nth-of-type(4) tbody tr.bg-primary-lighter td:nth-of-type(1)","multiple":false,"delay":0,"regex":""},{"id":"natureza","parentSelectors":["lancamentos"],"type":"SelectorText","selector":"> div > div:nth-of-type(4) tbody tr.bg-primary-lighter td:nth-of-type(2)","multiple":false,"delay":0,"regex":""},{"id":"sub-item","parentSelectors":["lancamentos"],"type":"SelectorText","selector":"> div > div:nth-of-type(4) tbody tr.bg-primary-lighter td:nth-of-type(2) small","multiple":false,"delay":0,"regex":""},{"id":"favorecido","parentSelectors":["lancamentos"],"type":"SelectorText","selector":"> div > div:nth-of-type(4) tbody tr.bg-primary-lighter  td:nth-of-type(3)","multiple":false,"delay":0,"regex":""},{"id":"cnpj","parentSelectors":["lancamentos"],"type":"SelectorText","selector":"> div > div:nth-of-type(4) tbody tr.bg-primary-lighter  td:nth-of-type(3) small","multiple":false,"delay":0,"regex":""},{"id":"valor","parentSelectors":["lancamentos"],"type":"SelectorText","selector":"> div > div:nth-of-type(4) tbody tr.bg-primary-lighter  td:nth-of-type(5)","multiple":false,"delay":0,"regex":""},{"id":"clica_parla","parentSelectors":["pagina_parla"],"type":"SelectorElementClick","clickElementSelector":"div#vue-entries > div > div:nth-of-type(2) tbody tr","clickElementUniquenessType":"uniqueCSSSelector","clickType":"clickOnce","delay":1500,"discardInitialElements":"do-not-discard","multiple":true,"selector":"_parent_"},{"id":"clica_comp","parentSelectors":["pagina_comp"],"type":"SelectorElementClick","clickElementSelector":"div#vue-entries > div > div:nth-of-type(3) tbody tr","clickElementUniquenessType":"uniqueCSSSelector","clickType":"clickOnce","delay":1000,"discardInitialElements":"discard-when-click-element-exists","multiple":true,"selector":"_parent_"},{"id":"documento","parentSelectors":["lancamentos"],"type":"SelectorLink","selector":"> div > div:nth-of-type(5) a.btn:first-child","multiple":false,"delay":0},{"id":"parlamentar","parentSelectors":["lancamentos"],"type":"SelectorText","selector":".col-md-9 small","multiple":false,"delay":0,"regex":""},{"id":"pagina_lancamento","parentSelectors":["clica_comp"],"type":"SelectorElementClick","clickElementSelector":"div.table-responsive:has(th:contains(\"Objeto\")) li.page-item.active + li div:not(:contains(\"»\"))","clickElementUniquenessType":"uniqueCSSSelector","clickType":"clickMore","delay":1500,"discardInitialElements":"do-not-discard","multiple":true,"selector":"div:has(> div > div#vue-entries)"},{"id":"pagina_comp","parentSelectors":["clica_parla"],"type":"SelectorElementClick","clickElementSelector":"div.table-responsive:has(th:contains(\"Ano\")) li.page-item.active + li div:not(:contains(\"»\"))","clickElementUniquenessType":"uniqueText","clickType":"clickMore","delay":1500,"discardInitialElements":"do-not-discard","multiple":true,"selector":"_parent_"},{"id":"pagina_parla","parentSelectors":["_root"],"type":"SelectorElementClick","clickElementSelector":"div.table-responsive:has(th:contains(\"Nome do Parlamentar\")) li.page-item.active + li div:not(:contains(\"»\"))","clickElementUniquenessType":"uniqueCSSSelector","clickType":"clickMore","delay":2000,"discardInitialElements":"do-not-discard","multiple":true,"selector":"body"},{"id":"q_documentos","parentSelectors":["lancamentos"],"type":"SelectorText","selector":"> div > div:nth-of-type(4) tbody tr.bg-primary-lighter td:nth-of-type(4)","multiple":false,"delay":0,"regex":""},{"id":"competencia","parentSelectors":["lancamentos"],"type":"SelectorText","selector":"div.card:has(th:contains(\"Ano\")) .bg-primary-lighter.text-white.bg-primary-lighter.text-white td:nth(0)","multiple":false,"delay":0,"regex":""}]}