I have set up webcrawler for the site
Url: https://eur-lex.europa.eu/search.html
The crawler seems to work fine and complete its work but when it is done I see fewer scraped results than in the results count. I get 53 where tthe results say 59. Any suggestions are welcome:
Sitemap:
{"_id":"eur-lex-new","startUrl":["https://eur-lex.europa.eu/search.html?page=1&qid=1541187898483&DB_TYPE_OF_ACT=judgment&CASE_LAW_SUMMARY=false&DTS_DOM=EU_LAW&typeOfActStatus=JUDGMENT&type=advanced&AU_CODED=CJ&lang=en&SUBDOM_INIT=EU_CASE_LAW&date0=ALL:01012016|31012016&DTS_SUBDOM=EU_CASE_LAW"],"selectors":[{"id":"celex","type":"SelectorText","parentSelectors":["_root","pager"],"selector":"div.SearchResult:nth-of-type(n+5) div.col-sm-6:nth-of-type(1) dl","multiple":true,"regex":"","delay":0},{"id":"pager","type":"SelectorLink","parentSelectors":["_root","pager"],"selector":"div.ResultsToolsWrapper:nth-of-type(3) a.btn:nth-of-type(2)","multiple":false,"delay":0}]}