Scraping forum posts from a Google search

How to?
#1

Hello

First-timer here. I'm trying to get the scraper to click through a list of google search results of forum threads and then scrape the individual posts.

This is what I've done so far:

{"_id":"googlerobo","startUrl":["https://www.google.com/search?q=roboadvisor+forum+site:www.wertpapier-forum.de&ei=GBg3XeaaBdH8kwWy9r2ABQ&start=0&sa=N&ved=0ahUKEwjmhpbbnsvjAhVR_qQKHTJ7D1A4ChDy0wMIkgE&biw=1440&bih=651"],"selectors":[{"id":"result","type":"SelectorLink","parentSelectors":["_root","resultnext"],"selector":"h3.LC20lb","multiple":true,"delay":0},{"id":"resultnext","type":"SelectorLink","parentSelectors":["_root","resultnext"],"selector":"a.pn","multiple":true,"delay":0},{"id":"link","type":"SelectorHTML","parentSelectors":["result"],"selector":"div.g:nth-of-type(n+9) h3","multiple":true,"regex":"","delay":0},{"id":"post","type":"SelectorText","parentSelectors":["result"],"selector":"article:nth-of-type(n+3) [itemprop='text'] > p","multiple":true,"regex":"","delay":0}]}

But all my scrape does is click through the google search results pages and not click into the threads.

#2

Try this. You'll need to build new data scrapers under result.

{"_id":"google-search-forum","startUrl":["https://www.google.com/search?q=roboadvisor+forum+site:www.wertpapier-forum.de&ei=GBg3XeaaBdH8kwWy9r2ABQ&start=0&sa=N&ved=0ahUKEwjmhpbbnsvjAhVR_qQKHTJ7D1A4ChDy0wMIkgE&biw=1440&bih=651"],"selectors":[{"id":"result","type":"SelectorLink","parentSelectors":["_root","resultnext"],"selector":"div#center_col div > a[href*='wertpapier-forum']","multiple":true,"delay":0},{"id":"resultnext","type":"SelectorLink","parentSelectors":["_root","resultnext"],"selector":"div#foot td > a#pnnext","multiple":true,"delay":0}]}