Link does not work

How to?
#1

When I use the link selector on the secondary page, if the href of the link is the same as the url of other pages, the data cannot be obtained.
What should I do to obtain the data normally?

Sitemap:
{"_id":"Imperial-College-London-ug-new","startUrl":["https://www.imperial.ac.uk/study/courses?courseType=undergraduate&keywords="],"selectors":[{"id":"pagination","paginationType":"clickMore","parentSelectors":["_root","pagination"],"selector":".pagination__list li:contains('Next') a","type":"SelectorPagination"},{"elementLimit":0,"id":"major_list","multiple":true,"parentSelectors":["pagination"],"scroll":false,"selector":"div.course-card","type":"SelectorElement"},{"id":"major_url","linkType":"linkFromHref","multiple":true,"parentSelectors":["major_list"],"selector":".course-card__title a, .course-card__facet a","type":"SelectorLink"},{"id":"major_name","multiple":false,"parentSelectors":["major_url"],"regex":"","selector":".js-course-selector-content[style*='visibility: visible'] h1","type":"SelectorText"},{"id":"degree","multiple":false,"parentSelectors":["major_url"],"regex":"","selector":".js-course-selector-content[style*='visibility: visible'] .course-key-facts__items>li:contains('Qualification') ul","type":"SelectorText"},{"id":"language_require","multiple":false,"parentSelectors":["major_url"],"regex":"","selector":".accordion-tabs section:contains('English language requirement')","type":"SelectorHTML"},{"id":"language_require_link","linkType":"linkFromHref","multiple":false,"parentSelectors":["major_url"],"selector":".accordion-tabs section:contains('English language requirement') a","type":"SelectorLink"},{"id":"IELTS","multiple":false,"parentSelectors":["language_require_link"],"regex":"","selector":".js-tabs-panel:contains('IELTS') .content","type":"SelectorText"},{"id":"TOEFL","multiple":false,"parentSelectors":["language_require_link"],"regex":"","selector":".js-tabs-panel:contains('TOEFL') .content","type":"SelectorText"},{"id":"PTE","multiple":false,"parentSelectors":["language_require_link"],"regex":"","selector":".js-tabs-panel:contains('PTE') .content","type":"SelectorText"}]}

#2

Remove pagination . Then add URL by page range/ This is the main URL : https://www.imperial.ac.uk/study/courses/?courseType=undergraduate&page=[1-6]

#3

I removed the pagination in your way, but it still didn't work and only one piece of data could be obtained.

{"_id":"Imperial-College-London-ug-new","startUrl":["https://www.imperial.ac.uk/study/courses?courseType=undergraduate&keywords="],"selectors":[{"elementLimit":0,"id":"major_list","multiple":true,"parentSelectors":["_root"],"scroll":false,"selector":"div.course-card","type":"SelectorElement"},{"id":"major_url","linkType":"linkFromHref","multiple":true,"parentSelectors":["major_list"],"selector":".course-card__title a, .course-card__facet a","type":"SelectorLink"},{"id":"major_name","multiple":false,"parentSelectors":["major_url"],"regex":"","selector":".js-course-selector-content[style*='visibility: visible'] h1","type":"SelectorText"},{"id":"degree","multiple":false,"parentSelectors":["major_url"],"regex":"","selector":".js-course-selector-content[style*='visibility: visible'] .course-key-facts__items>li:contains('Qualification') ul","type":"SelectorText"},{"id":"language_require","multiple":false,"parentSelectors":["major_url"],"regex":"","selector":".accordion-tabs section:contains('English language requirement')","type":"SelectorHTML"},{"id":"language_require_link","linkType":"linkFromHref","multiple":false,"parentSelectors":["major_url"],"selector":".accordion-tabs section:contains('English language requirement') a","type":"SelectorLink"},{"id":"IELTS","multiple":false,"parentSelectors":["language_require_link"],"regex":"","selector":".js-tabs-panel:contains('IELTS') .content","type":"SelectorText"},{"id":"TOEFL","multiple":false,"parentSelectors":["language_require_link"],"regex":"","selector":".js-tabs-panel:contains('TOEFL') .content","type":"SelectorText"},{"id":"PTE","multiple":false,"parentSelectors":["language_require_link"],"regex":"","selector":".js-tabs-panel:contains('PTE') .content","type":"SelectorText"}]}

#4

Please try this one:
{"_id":"Imperial-College-London-ug-new4","startUrl":["https://www.imperial.ac.uk/study/courses/?courseType=undergraduate&page=[1-6]"],"selectors":[{"elementLimit":0,"id":"major_list","multiple":true,"parentSelectors":["_root"],"scroll":false,"selector":"div.course-card","type":"SelectorElement"},{"id":"major_url","linkType":"linkFromHref","multiple":true,"parentSelectors":["major_list"],"selector":".course-card__title a, .course-card__facet a","type":"SelectorLink"},{"id":"major_name","multiple":false,"parentSelectors":["major_url"],"regex":"","selector":".js-course-selector-content[style*='visibility: visible'] h1","type":"SelectorText"},{"id":"degree","multiple":false,"parentSelectors":["major_url"],"regex":"","selector":".js-course-selector-content[style*='visibility: visible'] .course-key-facts__items>li:contains('Qualification') ul","type":"SelectorText"},{"id":"language_require","multiple":false,"parentSelectors":["major_url"],"regex":"","selector":".accordion-tabs section:contains('English language requirement')","type":"SelectorHTML"},{"extractAttribute":"href","id":"language_require_link","multiple":false,"parentSelectors":["major_url"],"selector":".accordion-tabs section:contains('English language requirement') a","type":"SelectorElementAttribute"},{"id":"TOEFL","multiple":false,"parentSelectors":["language_require_link"],"regex":"","selector":".js-tabs-panel:contains('TOEFL') .content","type":"SelectorText"}]}

#5

I tried your method. Although there are multiple pieces of data now, the fields such as IELTS and TOEFL that I need are missing

#6

Hi,

The issue here is related to the deduplication feature of the scraper, meaning that each unique URL will be visited only once. You should remove the selectors under language_require_link to return all scraping results.

The data returned by the selectors under language_require_link is the same for all records anyway, so there is actually no point rescraping it.

#7

Thank you. I understand what you mean. However, it is still hoped that the scraper will remove this feature, because some special scenarios do indeed need to crawl the same URL