Element scroll down and popup link work, but cannot parse data

How to?
#1

I am trying to scrape the website of the European Court of Human Rights. See: https://hudoc.echr.coe.int/eng#{"documentcollectionid2":["GRANDCHAMBER","CHAMBER"]}

A list of cases is presented first, and it is necessary to scroll down for other cases to show up. Alternatively, it is possible to click on the first one and then use pagination to move forward. The website seems javascript heavy and somehow dynamically generated, and I have not had much luck.

I found that I can use the element scroll down selector to select more cases—although at some point it stops, perhaps because of a timeout or something like that. A more successful techniques would seem that of selecting the first case and then use pagination. However, I although the 'next page' selector is correctly pointing to the 'next page' button, I cannot seem to make it work for some reason.

The reason I am using 'popup link' is that, if I do not, a page is opened without the next / previous buttons. Note that if I use the 'popup link' type of selector on both the first case and the 'next page', it works beautifully so long as I test the application using 'data preview'. Scraping, however, fails miserably.

Here is the export code:

{"_id":"hudoc","startUrl":["https://hudoc.echr.coe.int/eng#{"languageisocode":["ENG"],"documentcollectionid2":["GRANDCHAMBER","CHAMBER"],"isplaceholder":["False"]}"],"selectors":[{"id":"fistcase","type":"SelectorPopupLink","parentSelectors":["_root"],"selector":"div.results-list-block div:nth-of-type(1) div.result-item div.headlineContaniner a.document-link","multiple":false,"delay":0},{"id":"next","type":"SelectorPopupLink","parentSelectors":["next"],"selector":"div.echr-img-navigator-nextdoc-normal","multiple":false,"delay":0},{"id":"nextcase","type":"SelectorText","parentSelectors":["fistcase","next"],"selector":"div.echr-img-navigator-nextdoc-normal","multiple":true,"regex":"","delay":0},{"id":"title","type":"SelectorText","parentSelectors":["fistcase","next"],"selector":"div:nth-of-type(2) p:nth-of-type(14) > span","multiple":false,"regex":"","delay":0}]}

Thank you very much for any help!

#2

Hi there!

Website seem to be somewhat bugged.

The easiest solution for your case would be pressing first case manually, and let WebScraper go by itself, as having 2 selectors inside click selector fails on second case for some reason.

Try this one:
{"_id":"hudoc_test","startUrl":["https://hudoc.echr.coe.int/eng#{\"documentcollectionid2\":[\"GRANDCHAMBER\",\"CHAMBER\"]}"],"selectors":[{"id":"fistcase","type":"SelectorElementClick","selector":"div.modal","parentSelectors":["_root"],"multiple":true,"delay":"3000","clickElementSelector":"div.echr-img-navigator-nextdoc-normal","clickType":"clickMore","discardInitialElements":false,"clickElementUniquenessType":"uniqueText"},{"id":"title","type":"SelectorText","selector":"[class=bookmark]+","parentSelectors":["fistcase"],"multiple":false,"regex":"","delay":0}]}

Please note that in order for it to work properly, you have to press case 1 manually, as WebScraper Click selector is limited (by me) to a modal window that appears only if you click a particular case.

Once you are done with the scrape (i.e. you've got needed amount of records -- it's shown inside modal window case #/of #), just close the modal window for it to successfully load the results. Don't close the scrape window, just the modal.

P.S. you may decrease delay in Element Click selector to as low as 1000 ms if it loads fast for you.