Scraping stops after 20-30% of the job

How to?
#1

Describe the problem.
Trying to get a list of funds and related ISINs. The job is fairly big, 38000+ records.
The pagination works, but scraping ends after 20-30%, the partial result is possible to download.
Also quite slow even if page load is at 1000ms, 500 is not long enough.

Url: http://example.com

Sitemap:
{"_id":"MS_ISINs","startUrl":["https://www.morningstar.se/se/screener/fund.aspx#?filtersSelectedValue={}&sortField=name&sortOrder=asc"],"selectors":[{"clickActionType":"real","clickElementSelector":"button#ec-screener-view-tabs-tab6","clickElementUniquenessType":"uniqueText","clickType":"clickOnce","delay":2000,"discardInitialElements":"do-not-discard","id":"ovrigt","multiple":false,"parentSelectors":["_root"],"selector":"ec-tabs","type":"SelectorElementClick"},{"clickActionType":"real","clickElementSelector":".ec-table__cell--sticky span.ng-binding","clickElementUniquenessType":"uniqueText","clickType":"clickOnce","delay":2000,"discardInitialElements":"do-not-discard","id":"sort","multiple":false,"parentSelectors":["_root"],"selector":"ec-section.ec-section--table-securities","type":"SelectorElementClick"},{"id":"pag","paginationType":"auto","parentSelectors":["_root","pag"],"selector":".ec-section__section--paginator li:nth-of-type(n+2) a","type":"SelectorPagination"},{"columns":[{"extract":false,"header":"Check-box at table header","name":"Check-box at table header"},{"extract":true,"header":"Namn","name":"Namn"},{"extract":true,"header":"ISIN","name":"ISIN"},{"extract":false,"header":"PPM-nummer","name":"PPM-nummer"}],"id":"tabell","multiple":true,"parentSelectors":["pag"],"selector":"table","tableDataRowSelector":"tr.ec-table__row","tableHeaderRowSelector":"tr.ec-table__row-header","type":"SelectorTable"}]}

#2

Hi,

I adjusted the pagination to target only the next button. I would also recommend splitting the data into smaller batches by defining several start URLs, i.e. by 'Årlig avgift (%)'.

See example below:

{"_id":"MS_ISINs","startUrl":["https://www.morningstar.se/se/screener/fund.aspx#?filtersSelectedValue=%7B%22ongoingCharge%22:%7B%22id%22:%22:LT:0.5%22%7D%7D&sortField=name&sortOrder=asc"],"selectors":[{"clickActionType":"real","clickElementSelector":"button#ec-screener-view-tabs-tab6","clickElementUniquenessType":"uniqueText","clickType":"clickOnce","delay":2000,"discardInitialElements":"do-not-discard","id":"ovrigt","multiple":false,"parentSelectors":["_root"],"selector":"ec-tabs","type":"SelectorElementClick"},{"clickActionType":"real","clickElementSelector":".ec-table__cell--sticky span.ng-binding","clickElementUniquenessType":"uniqueCSSSelector","clickType":"clickOnce","delay":2000,"discardInitialElements":"discard-when-click-element-exists","id":"sort","multiple":false,"parentSelectors":["_root"],"selector":"ec-section.ec-section--table-securities","type":"SelectorElementClick"},{"clickActionType":"real","clickElementSelector":"[id=\"ec-screener-input-page-size-select\"] [value=\"50\"]","clickElementUniquenessType":"uniqueCSSSelector","clickType":"clickOnce","delay":1000,"discardInitialElements":"discard-when-click-element-exists","id":"per-page50","multiple":false,"parentSelectors":["_root"],"selector":"body","type":"SelectorElementClick"},{"id":"pag","paginationType":"auto","parentSelectors":["_root","pag"],"selector":".ec-section__section--paginator .ec-pagination__item--next a","type":"SelectorPagination"},{"columns":[{"extract":false,"header":"Check-box at table header","name":"Check-box at table header"},{"extract":true,"header":"Namn","name":"Namn"},{"extract":true,"header":"ISIN","name":"ISIN"},{"extract":false,"header":"PPM-nummer","name":"PPM-nummer"}],"id":"tabell","multiple":true,"parentSelectors":["pag"],"selector":"table","tableDataRowSelector":"tr.ec-table__row","tableHeaderRowSelector":"tr.ec-table__row-header","type":"SelectorTable"}]}