Extract text from a link to another website

#1

Hi

I've been using web scraper for a week or so and have solved scraping every page using this site, so thank you for that.

I have another problem now. I want to extract the postcode from a link to another site. I can see the postcode in the string but cannot work out how to extract it using element attribute. The other option I was looking at was at to achieve my goal was to follow the link to the other site and the postcode is clearly displayed so I was hoping to grab it using text selector but this messes up my site map and nothing is extracted. Any help would be very much appreciated

Cheers

Rick

#2

Can you share the sitemap?

#3

Hi
This is as far as I have got with the site map.

{"_id":"testrightmove","startUrl":["https://www.rightmove.co.uk/property-for-sale/find.html?locationIdentifier=REGION^257&maxPrice=70000&propertyTypes=detached%2Csemi-detached%2Cterraced&primaryDisplayPropertyType=houses&includeSSTC=false&mustHave=&dontShow=&furnishTypes=&keywords="],"selectors":[{"id":"property","type":"SelectorLink","parentSelectors":["property-card"],"selector":"div.propertyCard-details a.propertyCard-link","multiple":false,"delay":0},{"id":"description","type":"SelectorText","parentSelectors":["property"],"selector":"h1.fs-22","multiple":false,"regex":"","delay":0},{"id":"address","type":"SelectorText","parentSelectors":["property"],"selector":"div.left address.pad-0","multiple":false,"regex":"","delay":0},{"id":"property-card","type":"SelectorElementClick","parentSelectors":["_root"],"selector":".propertyCard-wrapper","multiple":true,"delay":"2500","clickElementSelector":".pagination-direction--next","clickType":"clickMore","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueCSSSelector"},{"id":"Search Criteria","type":"SelectorText","parentSelectors":["_root"],"selector":"h1","multiple":false,"regex":"","delay":0},{"id":"Price","type":"SelectorText","parentSelectors":["property"],"selector":".property-header-price strong","multiple":false,"regex":"","delay":0},{"id":"Added on Rightmove","type":"SelectorText","parentSelectors":["property"],"selector":"div#firstListedDateValue","multiple":false,"regex":"","delay":0},{"id":"Agent","type":"SelectorText","parentSelectors":["property"],"selector":"#aboutBranchLink strong","multiple":false,"regex":"","delay":0},{"id":"Phone Number","type":"SelectorText","parentSelectors":["property"],"selector":".bdr-b .fs-19 strong","multiple":false,"regex":"","delay":0},{"id":"postcode","type":"SelectorElementClick","parentSelectors":["property"],"selector":"div[itemtype='http://schema.org/Residence']","multiple":false,"delay":"2500","clickElementSelector":"span.check-broadband-speed","clickType":"clickOnce","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueCSSSelector"},{"id":"postcode link","type":"SelectorElementAttribute","parentSelectors":["postcode"],"selector":"a.see-all-offers","multiple":false,"extractAttribute":"","delay":0}]}

I either want it to extract the postcode which i can see in the link using element attribute.
or go to the page on the link and extract it as text as it appears on the linked website.
If both options are possible I'd like to do the option which is the quickest. The site map was quite quick until I introduced the selectors to extract the postcode so i don't want to slow it down too much.

Hope the above makes sense, let me know if anymore explanation is needed.

Cheers

Rick

#4

If you are just after the postcode, you can get it from the HTML using regex:

{"_id":"testrightmove","startUrl":["https://www.rightmove.co.uk/property-for-sale/find.html?locationIdentifier=REGION^257&maxPrice=70000&propertyTypes=detached%2Csemi-detached%2Cterraced&primaryDisplayPropertyType=houses&includeSSTC=false&mustHave=&dontShow=&furnishTypes=&keywords="],"selectors":[{"id":"property","type":"SelectorLink","parentSelectors":["property-card"],"selector":"div.propertyCard-details a.propertyCard-link","multiple":false,"delay":0},{"id":"description","type":"SelectorText","parentSelectors":["property"],"selector":"h1.fs-22","multiple":false,"regex":"","delay":0},{"id":"address","type":"SelectorText","parentSelectors":["property"],"selector":"div.left address.pad-0","multiple":false,"regex":"","delay":0},{"id":"property-card","type":"SelectorElementClick","parentSelectors":["_root"],"selector":".propertyCard-wrapper","multiple":true,"delay":"2500","clickElementSelector":".pagination-direction--next","clickType":"clickMore","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueCSSSelector"},{"id":"Search Criteria","type":"SelectorText","parentSelectors":["_root"],"selector":"h1","multiple":false,"regex":"","delay":0},{"id":"Price","type":"SelectorText","parentSelectors":["property"],"selector":".property-header-price strong","multiple":false,"regex":"","delay":0},{"id":"Added on Rightmove","type":"SelectorText","parentSelectors":["property"],"selector":"div#firstListedDateValue","multiple":false,"regex":"","delay":0},{"id":"Agent","type":"SelectorText","parentSelectors":["property"],"selector":"#aboutBranchLink strong","multiple":false,"regex":"","delay":0},{"id":"Phone Number","type":"SelectorText","parentSelectors":["property"],"selector":".bdr-b .fs-19 strong","multiple":false,"regex":"","delay":0},{"id":"postcode","type":"SelectorHTML","parentSelectors":["property"],"selector":"script:contains('postcode')","multiple":false,"regex":"(?<=postcode\\\":\\\")[^\"]*","delay":0}]}

#5

Hi Webber

That is perfect, thanks so much

Cheers

Rick

#6