Add BULK Urls from Start URL

needtesting · May 12, 2020, 4:05pm

Hello everybody.

Im trying to scrape a website.

Been checking a lot of posts on how to add bulk URLs but i can't seem how to figure it out.

This is my sitemap.

{"_id":"test1","startUrl":["http://streamboosting.com/links.html"],"selectors":[{"id":"titulo","type":"SelectorText","parentSelectors":["_root"],"selector":"span.a-size-extra-large","multiple":false,"regex":"","delay":0},{"id":"valoraciones","type":"SelectorText","parentSelectors":["_root"],"selector":"span#acrCustomerReviewText","multiple":false,"regex":"","delay":0},{"id":"preciomedio","type":"SelectorText","parentSelectors":["_root"],"selector":"span.offer-price","multiple":false,"regex":"","delay":0},{"id":"estrellas","type":"SelectorText","parentSelectors":["_root"],"selector":".reviewCountTextLinkedHistogram i.a-icon-star","multiple":false,"regex":"","delay":0},{"id":"categoria","type":"SelectorText","parentSelectors":["_root"],"selector":"li:nth-of-type(3) a.a-color-tertiary","multiple":false,"regex":"","delay":0},{"id":"ISBN10","type":"SelectorText","parentSelectors":["_root"],"selector":".content li:nth-of-type(5)","multiple":false,"regex":"","delay":0},{"id":"ISBN12","type":"SelectorText","parentSelectors":["_root"],"selector":".content li:nth-of-type(6)","multiple":false,"regex":"","delay":0},{"id":"paginas","type":"SelectorText","parentSelectors":["_root"],"selector":"td .content > ul > li:nth-of-type(1)","multiple":false,"regex":"","delay":0},{"id":"imagen","type":"SelectorImage","parentSelectors":["_root"],"selector":"img.frontImage","multiple":false,"delay":0},{"id":"desc","type":"SelectorElementClick","parentSelectors":["desc"],"selector":"span#bdSeeAllPrompt","multiple":false,"delay":0,"clickElementSelector":"span#bdSeeAllPrompt","clickType":"clickOnce","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueText"},{"id":"descc","type":"SelectorElementClick","parentSelectors":["_root"],"selector":"span#bdSeeAllPrompt","multiple":false,"delay":0,"clickElementSelector":"span#bdSeeAllPrompt","clickType":"clickOnce","discardInitialElements":"do-not-discard","clickElementUniquenessType":"uniqueText"},{"id":"desc2","type":"SelectorText","parentSelectors":["_root"],"selector":"div#bookDescription_feature_div.feature","multiple":false,"regex":"","delay":0}]}

I've read that if you upload an .html with all the URLs the scrapper should scrape everything, however, it doesn't scrape anything.

Any guess on how to add multiple urls at once from a single Start URL?

Thanks you

bretfeig · May 14, 2020, 12:53am

There are several ways to do this but I am not familiar with the ability to upload multiple urls with a html file. Maybe @iconoclast knows?

You can add multiple URL's one by one by clicking the + sign in the metadeta box.

The only other way I know how to do is by formatting each url to be encapsulated by quotes separated by commas and striching that into a sitemap to be imported. I made a tutorial on it here : https://www.youtube.com/watch?v=ToMPE4wyon8&t=42s

Note: I don't sell, promote or get paid to promote any software or service.. Shit I don't even have enough views to sell ads, purely a hobby.

leemeng · January 6, 2021, 11:08am

There's a site called Pastelink which can convert your URL list to clickable links, plus it will publish them on a page. So you can adjust your sitemap to scrape from that page.