trying to scrape rpggeek.com and I have a problem with the [next] pagination.
ex. CCC - D&D Convention-Created Content | Series | RPGGeek
and where it says "linked items" those are the items I want to scrape. I can get one page, even get data from the linked page, but nothing past the first page. I have tried many different ways.
the current code:
{"_id":"rpggeekpagination","startUrl":["CCC - D&D Convention-Created Content | Series | RPGGeek('Periodical Articles'):contains(''):contains('Podcast Episodes'):contains(''):contains(''):contains(''):contains('Sort:'):contains(''):contains('Name'):contains('Rank'):contains('Num Ratings'):contains('Average Rating'):contains('Num Owned'):contains('Prev. Owned'):contains('For Trade'):contains('Want in Trade'):contains('Wishlist'):contains('Comments'):contains('Year Published'):contains(''):contains(''):contains('Category:'):contains(''):contains(''):contains('Accessory (dice, maps, screens, cards)'):contains('Alternate Rules (Variant Core Rules)'):contains('Campaign Setting (game world)'):contains('Core Rules (min needed to play)'):contains('Non-Game Book (but RPG Related)'):contains('Non-Rankable Item (item will not show in the rankings)'):contains('Quick Start / Fast Play Rules'):contains('Scenario / Adventure / Module'):contains('Software (for maps, char sheets, etc)'):contains('Solo Play Resource'):contains('Sourcebook (rules/options to enhance play)'):contains(''):contains('Genre:'):contains(''):contains(''):contains('Action / Adventure'):contains('Action / Adventure (Gangsters / Mafia)'):contains('Action / Adventure (Nautical)'):contains('Action / Adventure (Pulp)'):contains('Action / Adventure (Sports)'):contains('Action / Adventure (Superheroes)'):contains('Action / Adventure (Swashbuckling / Pirates)'):contains('Adult / Mature'):contains('Animal'):contains('Animal (Anthropomorphic)'):contains('Animation / Comics'):contains('Animation / Comics (Anime / Manga)'):contains('Animation / Comics (Western Cartoon / Comic)'):contains('Childhood'):contains('Comedy / Satire'):contains('Corporations'):contains('Crime'):contains('Crime (Conspiracy)'):contains('Crime (Criminal / Law Enforcement)'):contains('Crime (Mystery / Detective / Noir)'):contains('Crime (Spy / Espionage)'):contains('Culture'):contains('Culture (Asian / Far Eastern)'):contains('Fantasy'):contains('Fantasy (Arabian Nights)'):contains('Fantasy (Arthurian)'):contains('Fantasy (High Fantasy)'):contains('Fantasy (Low Fantasy)'):contains('Fantasy (Modern Urban Fantasy)'):contains('Fantasy (Science Fantasy)'):contains('Fantasy (Sword & Sorcery)'):contains('Generic / Universal'):contains('History'):contains('History (Alternative)'):contains('History (American West)'):contains('History (Americas)'):contains('History (Ancient)'):contains('History (Britain / Ireland)'):contains('History (Europe)'):contains('History (Medieval)'):contains('History (Prehistoric)'):contains('History (Victorian Era)'):contains('History (Vikings)'):contains('Holiday-Themed'):contains('Horror'):contains('Horror (Cthulhu Mythos)'):contains('Horror (Supernatural)'):contains('Horror (Zombies)'):contains('Kaiju'):contains('Kid-Friendly'):contains('Magical Girl'):contains('Martial Arts'):contains('Mecha'):contains('Modern'):contains('Modern (Near Future)'):contains('Modern (Pop Culture)'):contains('Modern (Urban)'):contains('Multiverse'):contains('Mythology / Folklore'):contains('Mythology / Folklore (Fairy Tales)'):contains('Occult'):contains('Occult (Magic)'):contains('Occult (Psionics / Mentalism)'):contains('Occult (Symbolism / Tarot)'):contains('Politics / Government'):contains('Religion'):contains('Science Fiction'):contains('Science Fiction (Cyberpunk)'):contains('Science Fiction (Hard Sci Fi)'):contains('Science Fiction (Post Apocalypse)'):contains('Science Fiction (Space Opera)'):contains('Science Fiction (Time / Dimension Travel)'):contains('Social (Relationships / Romance)'):contains('Steampunk'):contains('War'):contains('Weird / Surreal') div.pages","type":"SelectorPagination"},{"id":"wrapper","multiple":true,"parentSelectors":["_root"],"selector":"#module_5 tr[bgcolor]","type":"SelectorElement"},{"id":"link","linkType":"linkFromHref","multiple":true,"parentSelectors":["wrapper"],"selector":".geekitem_linkeditems_title > a","type":"SelectorLink"},{"id":"year","multiple":false,"parentSelectors":["wrapper"],"regex":"","selector":"tr tr:contains('Year Published') td[width]","type":"SelectorText"},{"id":"title","multiple":false,"parentSelectors":["link"],"regex":"","selector":"span[itemprop='name']","type":"SelectorText"},{"id":"Description","multiple":false,"parentSelectors":["link"],"regex":"","selector":"div.panel.panel-expandable","type":"SelectorText"},{"id":"classification","multiple":false,"parentSelectors":["link"],"regex":"","selector":"classifications-module","type":"SelectorText"},{"id":"full credits","linkType":"linkFromHref","multiple":false,"parentSelectors":["link"],"selector":".credits-cta a","type":"SelectorLink"},{"id":"primary name","multiple":false,"parentSelectors":["full credits"],"regex":"","selector":"li:nth-of-type(1) .ng-scope div.ng-scope","type":"SelectorText"},{"id":"designer","multiple":false,"parentSelectors":["full credits"],"regex":"","selector":"li:nth-of-type(5) .outline-item-description > div","type":"SelectorText"},{"id":"artist","multiple":false,"parentSelectors":["full credits"],"regex":"","selector":"li:nth-of-type(6) .outline-item-description > div","type":"SelectorText"},{"id":"publisher","multiple":false,"parentSelectors":["full credits"],"regex":"","selector":"li:nth-of-type(7) .outline-item-description > div","type":"SelectorText"},{"id":"production","multiple":false,"parentSelectors":["full credits"],"regex":"","selector":"li:nth-of-type(8) .outline-item-description > div","type":"SelectorText"}]}