Describe the problem.
I was able to scrape columns of data from a website. But I have one column of data which are url redirects, how do I test this redirects and get the real url out of it.
Url: https://goo.gl/z9CSqx > output should be https://www.youtube.com/user/angdatingdaan
Thanks.
Hello!
This is not so easy as it looks, you will have to create a 'Link' selector to go to the page, because Google Short URL do redirect to the page only when you clicked on them, there's no original URL to the shortened page stored in it.
After that, you can extract an attribute that can help you identify the page URL either partially or fully.
In your case, if you scrape only YouTube users, your sitemap would look like this:
{"_id":"test2","startUrl":["https://forum.webscraper.io/t/how-to-test-redirects-and-get-the-real-url"],"selectors":[{"id":"shortenedURL","type":"SelectorLink","selector":"div.cooked a:nth-of-type(1)","parentSelectors":["_root"],"multiple":false,"delay":0},{"id":"User","type":"SelectorElementAttribute","selector":"ytd-item-section-renderer.style-scope:nth-of-type(4) ytd-grid-video-renderer.style-scope:nth-of-type(1) yt-formatted-string.style-scope a.yt-simple-endpoint","parentSelectors":["shortenedURL"],"multiple":false,"extractAttribute":"href","delay":"0"}]}
It will stay on the page for about 4-5 seconds and then it will go to Youtube to pick the user address from a video that user has published.
P.S. Kristaps please correct me if i'm wrong.