Webscraper is not scraping links in table

How to?
#1

In the site URL,there are 2 drop downs, after selecting values in both drop-downs, a table (main table) is then generated. Inside the table are links in the third column. Inside these links is a static table, I am trying scrape the information inside these static tables. However, My webscraper only loops through the drop downs, but does not go into the links of the main table. What am I doing wrong, I feel like I am so close and the sitemap just needs a bit of tweaking, but I cannot for the life of me figure out what the issue is.

Url: https://services.cds.ca/applications/taxforms/Eligible.nsf/All_en?OpenForm&_ga=2.84192089.1542579991.1620737439-1909997006.1617701331

Sitemap:
{id:"sitemap code"}

{"_id":"plescds","startUrl":["Request Rejected option:not(:contains("select"))","clickType":"clickMore","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueCSSSelector"},{"id":"wrap2","type":"SelectorElementClick","parentSelectors":["wrap1"],"selector":"parent","multiple":true,"delay":2000,"clickElementSelector":"select[name='SelectAlpha'] option:not(:contains("select"))","clickType":"clickOnce","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueCSSSelector"},{"id":"maintable","type":"SelectorElementClick","parentSelectors":["wrap2"],"selector":"div.sort-table select("DivList")","multiple":true,"delay":"500","clickElementSelector":".Label2 a","clickType":"clickOnce","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueCSSSelector"},{"id":"subtable","type":"SelectorTable","parentSelectors":["maintable"],"selector":"table.Label2","multiple":true,"columns":[{"header":"Period","name":"Period","extract":true},{"header":"Dividend Designation","name":"Dividend Designation","extract":true},{"header":"Record Date (m/d/y)","name":"Record Date (m/d/y)","extract":true},{"header":"Payment Date (m/d/y)","name":"Payment Date (m/d/y)","extract":true},{"header":"Amount Per Share","name":"Amount Per Share","extract":true}],"delay":0,"tableDataRowSelector":"tr:nth-of-type(n+2)","tableHeaderRowSelector":"tr:nth-of-type(1)"}]}

#2

@jta123 Hello. I can't open your sitemap. :frowning: Invalid JSON, probably due to some pasting error. I'm not entirely sure if it would work because the page is reloaded and it produces a unique link each time, element click would work if it wasn't reloaded. So I would just go by using the URLs i get from each year and letter and extract the info.

{"_id":"services","startUrl":["https://services.cds.ca/applications/taxforms/Eligible.nsf/All_en?OpenForm&year=2008&alpha=C"],"selectors":[{"id":"card","type":"SelectorElement","parentSelectors":["_root"],"selector":"body:has(table#DivList)","multiple":true,"delay":0},{"id":"table","type":"SelectorElement","parentSelectors":["card"],"selector":".sort-table tbody tr","multiple":true,"delay":0},{"id":"tax-year","type":"SelectorText","parentSelectors":["table"],"selector":"td:nth(0)","multiple":false,"regex":"","delay":0},{"id":"last update","type":"SelectorText","parentSelectors":["table"],"selector":"td:nth(1)","multiple":false,"regex":"","delay":0},{"id":"security-description","type":"SelectorText","parentSelectors":["table"],"selector":"td:nth(2)","multiple":false,"regex":"","delay":0},{"id":"status","type":"SelectorText","parentSelectors":["table"],"selector":"td:nth(3)","multiple":false,"regex":"","delay":0}]}

1 Like
#3

Thanks so much for the assistance. I have edited the sitemap now. Would that be a manual process of copying and pasting each URL in or does webscraper.io have the functionality to automate that part out ?

#4

@jta123 There is this solution Is it possible to have more than one input url in the same sitmap?

And it's not hard to edit all of it knowing that the link changes according to year and letter.

link change
link change901×327 18.1 KB

1 Like
#5

Great, thanks once again for the help!

1 Like