Less data in Scraper browser smaller window and data hidden in

[NicV]

While the preview data are OK, the outcome of the scraper is missing a lot of data, as the scraper window size is smaller so the site is showing less data. How can I tell the scraper to use a full size window?

Furthermore I'd like to collect also some hidden data, ,like "Division", T1:Swim-To-Bike" and "T2: Bike-To-Run", that are shown only if you click a button; of course you cannot expand all data as, when you you click on the second button the first one collapse. I saw a similar Topic on "Can't scrape 3 buttons...", but I am afraid this is different, as there are a lot more hidden data here.There are also hidden buttons that give you access to more hidden data. How can I retrieve all these hidden data?

Url: Competitor Labs

Sitemap:
{"_id":"HelpCervia","startUrl":["Competitor Labs span","type":"SelectorText"},{"id":"DivRank","multiple":false,"parentSelectors":["Table"],"regex":"","selector":".column-FinishRankGroup span","type":"SelectorText"},{"id":"OverallRank","multiple":false,"parentSelectors":["Table"],"regex":"","selector":".column-FinishRankOverall span","type":"SelectorText"},{"id":"Swim","multiple":false,"parentSelectors":["Table"],"regex":"","selector":".column-SwimTimeConverted span","type":"SelectorText"},{"id":"Bike","multiple":false,"parentSelectors":["Table"],"regex":"","selector":".column-BikeTimeConverted span","type":"SelectorText"},{"id":"Run","multiple":false,"parentSelectors":["Table"],"regex":"","selector":".column-RunTimeConverted span","type":"SelectorText"},{"id":"TotalTime","multiple":false,"parentSelectors":["Table"],"regex":"","selector":".column-FinishTimeConverted span","type":"SelectorText"},{"id":"Points","multiple":false,"parentSelectors":["Table"],"regex":"","selector":".column-RankPoints span","type":"SelectorText"},{"id":"Athlete","multiple":false,"parentSelectors":["Table"],"regex":"","selector":"td:nth-of-type(2) span","type":"SelectorText"},{"id":"GenderRank","multiple":false,"parentSelectors":["Table"],"regex":"","selector":".column-FinishRankGender span","type":"SelectorText"}]}

[JanAp]

Please post the sitemap as Preformatted text

[NicV]

Sitemap

{"_id":"HelpCervia","startUrl":["https://labs.competitor.com/result/subevent/FEA6DADB-93A9-42C9-BA4E-3DFC6D024B22?filter={}&order=ASC&page=1&perPage=200&sort=FinishRankOverall"],"selectors":[{"id":"Table","parentSelectors":["Pagine"],"type":"SelectorElement","selector":"tr.MuiTableRow-hover","multiple":true},{"id":"Country","parentSelectors":["Table"],"type":"SelectorText","selector":"td:nth-of-type(3) span","multiple":false,"regex":""},{"id":"DivRank","parentSelectors":["Table"],"type":"SelectorText","selector":".column-FinishRankGroup span","multiple":false,"regex":""},{"id":"OverallRank","parentSelectors":["Table"],"type":"SelectorText","selector":".column-FinishRankOverall span","multiple":false,"regex":""},{"id":"Swim","parentSelectors":["Table"],"type":"SelectorText","selector":".column-SwimTimeConverted span","multiple":false,"regex":""},{"id":"Bike","parentSelectors":["Table"],"type":"SelectorText","selector":".column-BikeTimeConverted span","multiple":false,"regex":""},{"id":"Run","parentSelectors":["Table"],"type":"SelectorText","selector":".column-RunTimeConverted span","multiple":false,"regex":""},{"id":"TotalTime","parentSelectors":["Table"],"type":"SelectorText","selector":".column-FinishTimeConverted span","multiple":false,"regex":""},{"id":"Points","parentSelectors":["Table"],"type":"SelectorText","selector":".column-RankPoints span","multiple":false,"regex":""},{"id":"Athlete","parentSelectors":["Table"],"type":"SelectorText","selector":"td:nth-of-type(2) span","multiple":false,"regex":""},{"id":"GenderRank","parentSelectors":["Table"],"type":"SelectorText","selector":".column-FinishRankGender span","multiple":false,"regex":""},{"id":"Pagine","parentSelectors":["_root","Pagine"],"paginationType":"auto","type":"SelectorPagination","selector":"button.MuiPaginationItem-page"}]}

[JanAp]

The hidden data is only available after the click and since the tr element holding the hidden data is at the same level as the original table row, it will not be possible to arrange the data automatically in one row, meaning that you will have to manually move the cells to align the data for each page.

I have removed the pagination for testing purposes, so the scraping finishes faster. You can try the sitemap below:

{"_id":"HelpCervia-with-click","startUrl":["https://labs.competitor.com/result/subevent/FEA6DADB-93A9-42C9-BA4E-3DFC6D024B22?filter={}&order=ASC&page=1&perPage=200&sort=FinishRankOverall"],"selectors":[{"id":"Table","multiple":true,"parentSelectors":["_root"],"selector":"tr.MuiTableRow-hover","type":"SelectorElement"},{"id":"Country","multiple":false,"parentSelectors":["Table"],"regex":"","selector":"td:nth-of-type(3) span","type":"SelectorText"},{"id":"DivRank","multiple":false,"parentSelectors":["Table"],"regex":"","selector":".column-FinishRankGroup span","type":"SelectorText"},{"id":"OverallRank","multiple":false,"parentSelectors":["Table"],"regex":"","selector":".column-FinishRankOverall span","type":"SelectorText"},{"id":"Swim","multiple":false,"parentSelectors":["Table"],"regex":"","selector":".column-SwimTimeConverted span","type":"SelectorText"},{"id":"Bike","multiple":false,"parentSelectors":["Table"],"regex":"","selector":".column-BikeTimeConverted span","type":"SelectorText"},{"id":"Run","multiple":false,"parentSelectors":["Table"],"regex":"","selector":".column-RunTimeConverted span","type":"SelectorText"},{"id":"TotalTime","multiple":false,"parentSelectors":["Table"],"regex":"","selector":".column-FinishTimeConverted span","type":"SelectorText"},{"id":"Points","multiple":false,"parentSelectors":["Table"],"regex":"","selector":".column-RankPoints span","type":"SelectorText"},{"id":"Athlete","multiple":false,"parentSelectors":["Table"],"regex":"","selector":"td:nth-of-type(2) span","type":"SelectorText"},{"id":"GenderRank","multiple":false,"parentSelectors":["Table"],"regex":"","selector":".column-FinishRankGender span","type":"SelectorText"},{"clickActionType":"real","clickElementSelector":"tr.MuiTableRow-hover","clickElementUniquenessType":"uniqueCSSSelector","clickType":"clickOnce","delay":0,"discardInitialElements":"do-not-discard","id":"athlete-click","multiple":true,"parentSelectors":["_root"],"selector":"[id*=\"-expand\"]","type":"SelectorElementClick"},{"id":"Division","multiple":false,"parentSelectors":["athlete-click"],"regex":"","selector":".genInfo .tableFooter div:nth-of-type(2)","type":"SelectorText"},{"id":"Swim-to-Bike","multiple":false,"parentSelectors":["athlete-click"],"regex":"","selector":"#transitions .tableFooter div.text:nth-of-type(1)","type":"SelectorText"},{"id":"Bike-to-Run","multiple":false,"parentSelectors":["athlete-click"],"regex":"","selector":"#transitions .tableFooter div:nth-of-type(2)","type":"SelectorText"}]}

[NicV]

I am still experiencing the problem of the smaller scraper window causing the loss of data and I am struggling to add pagination as when the athlete click comes in after the end of first page the second page is not loaded yet, so the scraper stay indefinetively on the first page. Tried akso to add a 5000 ms delay on athlete click, but I saw no difference. Of course I can solve all problems by copying data from the preview, avoiding the loss, and repeat the preview for all pages, to overcome pagination issue, but this is not confortable.

{"_id":"HelpCervia-with-click-mod","startUrl":["https://labs.competitor.com/result/subevent/FEA6DADB-93A9-42C9-BA4E-3DFC6D024B22?filter={}&order=ASC&page=1&perPage=200&sort=FinishRankOverall"],"selectors":[{"id":"Table","multiple":true,"parentSelectors":["Pagination"],"selector":"tr.MuiTableRow-hover","type":"SelectorElement"},{"id":"Country","multiple":false,"parentSelectors":["Table"],"regex":"","selector":"td:nth-of-type(3) span","type":"SelectorText"},{"id":"DivRank","multiple":false,"parentSelectors":["Table"],"regex":"","selector":".column-FinishRankGroup span","type":"SelectorText"},{"id":"OverallRank","multiple":false,"parentSelectors":["Table"],"regex":"","selector":".column-FinishRankOverall span","type":"SelectorText"},{"id":"Swim","multiple":false,"parentSelectors":["Table"],"regex":"","selector":".column-SwimTimeConverted span","type":"SelectorText"},{"id":"Bike","multiple":false,"parentSelectors":["Table"],"regex":"","selector":".column-BikeTimeConverted span","type":"SelectorText"},{"id":"Run","multiple":false,"parentSelectors":["Table"],"regex":"","selector":".column-RunTimeConverted span","type":"SelectorText"},{"id":"TotalTime","multiple":false,"parentSelectors":["Table"],"regex":"","selector":".column-FinishTimeConverted span","type":"SelectorText"},{"id":"Points","multiple":false,"parentSelectors":["Table"],"regex":"","selector":".column-RankPoints span","type":"SelectorText"},{"id":"Athlete","multiple":false,"parentSelectors":["Table"],"regex":"","selector":"td:nth-of-type(2) span","type":"SelectorText"},{"id":"GenderRank","multiple":false,"parentSelectors":["Table"],"regex":"","selector":".column-FinishRankGender span","type":"SelectorText"},{"clickActionType":"real","clickElementSelector":"tr.MuiTableRow-hover","clickElementUniquenessType":"uniqueCSSSelector","clickType":"clickOnce","delay":0,"discardInitialElements":"do-not-discard","id":"athlete-click","multiple":true,"parentSelectors":["Pagination"],"selector":"[id*=\"-expand\"]","type":"SelectorElementClick"},{"id":"Division","multiple":false,"parentSelectors":["athlete-click"],"regex":"","selector":".genInfo .tableFooter div:nth-of-type(2)","type":"SelectorText"},{"id":"Swim-to-Bike","multiple":false,"parentSelectors":["athlete-click"],"regex":"","selector":"#transitions .tableFooter div.text:nth-of-type(1)","type":"SelectorText"},{"id":"Bike-to-Run","multiple":false,"parentSelectors":["athlete-click"],"regex":"","selector":"#transitions .tableFooter div:nth-of-type(2)","type":"SelectorText"},{"id":"BIB","multiple":false,"parentSelectors":["athlete-click"],"regex":"","selector":".genInfo .tableFooter div.text:nth-of-type(1)","type":"SelectorText"},{"id":"Pagination","paginationType":"auto","parentSelectors":["_root","Pagination"],"selector":"button.MuiPaginationItem-page","type":"SelectorPagination"}]}

[leemeng]

Have you tried the "zoom out before scraping" trick? You can do this before you launch the scraper, and the browser should remember your zoom setting. In this example, I have zoomed out to 25%. You can also click Maximise on the scraper window.

[NicV]

Thanks, this is solving the loss of data problem; still trying to solve pagination and athlete click problem

[JanAp]

Here is a sitemap with a working pagination:

{"_id":"HelpCervia-with-click-mod","startUrl":["https://labs.competitor.com/result/subevent/FEA6DADB-93A9-42C9-BA4E-3DFC6D024B22?filter={}&order=ASC&page=1&perPage=200&sort=FinishRankOverall"],"selectors":[{"id":"Table","multiple":true,"parentSelectors":["Pagination"],"selector":"tr.MuiTableRow-hover","type":"SelectorElement"},{"id":"Country","multiple":false,"parentSelectors":["Table"],"regex":"","selector":"td:nth-of-type(3) span","type":"SelectorText"},{"id":"DivRank","multiple":false,"parentSelectors":["Table"],"regex":"","selector":".column-FinishRankGroup span","type":"SelectorText"},{"id":"OverallRank","multiple":false,"parentSelectors":["Table"],"regex":"","selector":".column-FinishRankOverall span","type":"SelectorText"},{"id":"Swim","multiple":false,"parentSelectors":["Table"],"regex":"","selector":".column-SwimTimeConverted span","type":"SelectorText"},{"id":"Bike","multiple":false,"parentSelectors":["Table"],"regex":"","selector":".column-BikeTimeConverted span","type":"SelectorText"},{"id":"Run","multiple":false,"parentSelectors":["Table"],"regex":"","selector":".column-RunTimeConverted span","type":"SelectorText"},{"id":"TotalTime","multiple":false,"parentSelectors":["Table"],"regex":"","selector":".column-FinishTimeConverted span","type":"SelectorText"},{"id":"Points","multiple":false,"parentSelectors":["Table"],"regex":"","selector":".column-RankPoints span","type":"SelectorText"},{"id":"Athlete","multiple":false,"parentSelectors":["Table"],"regex":"","selector":"td:nth-of-type(2) span","type":"SelectorText"},{"id":"GenderRank","multiple":false,"parentSelectors":["Table"],"regex":"","selector":".column-FinishRankGender span","type":"SelectorText"},{"clickActionType":"real","clickElementSelector":"tr.MuiTableRow-hover","clickElementUniquenessType":"uniqueCSSSelector","clickType":"clickOnce","delay":0,"discardInitialElements":"do-not-discard","id":"athlete-click","multiple":true,"parentSelectors":["Pagination"],"selector":"[id*=\"-expand\"]","type":"SelectorElementClick"},{"id":"Division","multiple":false,"parentSelectors":["athlete-click"],"regex":"","selector":".genInfo .tableFooter div:nth-of-type(2)","type":"SelectorText"},{"id":"Swim-to-Bike","multiple":false,"parentSelectors":["athlete-click"],"regex":"","selector":"#transitions .tableFooter div.text:nth-of-type(1)","type":"SelectorText"},{"id":"Bike-to-Run","multiple":false,"parentSelectors":["athlete-click"],"regex":"","selector":"#transitions .tableFooter div:nth-of-type(2)","type":"SelectorText"},{"id":"BIB","multiple":false,"parentSelectors":["athlete-click"],"regex":"","selector":".genInfo .tableFooter div.text:nth-of-type(1)","type":"SelectorText"},{"id":"Pagination","paginationType":"clickMore","parentSelectors":["_root","Pagination"],"selector":"[aria-label=\"Go to next page\"]","type":"SelectorPagination"}]}

When the scrape window pops up, just click to maximise the window.

[NicV]

Thank you all, now it is pefectly working. If I try to maximise the pop up sometimes it doesn't add data but with the zoom out it works perfectly; I didn't expect the zoom in pop up window was related to the zoom in the main one. Thanks a lot!

[NicV]

No, sorry, that pagination doesn't work: at the end it goes on indefinitively on the last page as if next pahe button is always available. Is there a way to stop the scarping process and collect scraped data?

[JanAp]

Sure, change the pagination selector to:

[aria-label="Go to next page"]:not([disabled])