Challenges while Scrapping University Website

Pranay · July 27, 2018, 2:34pm

University of Sydney :-
Starting URL: https://sydney.edu.au/courses/search.html
Course Page URL with data: https://sydney.edu.au/courses/courses/uc/bachelor-of-advanced-computing-and-bachelor-of-science.html

• The problem which I am facing while scraping the courses for this university is that each time the course page open with default information is for a domestic student. I want to get help in understanding how to make drop down selection of each page before I capture the data. Please note that when I make drop down selections, the URL does not change.
• I am attaching the screen shots of default selection and that do I want to select before capturing the data. For example in default view:-
ust

• After selecting each parameter from the above section then I will get the information for every course for international students. Below screenshot

tt-1

• I want to capture Admission Criteria, Tuition Fees, English Language Proficiency for IELTS

jasond · July 29, 2018, 4:55am

In your second link, the data have all been "pre-delivered," in JSON format, along with the Webpage.

Examples of your second link and it's related JSON file URLs (found by studying the F12 Developer Tools > Network):

https://sydney.edu.au/courses/courses/uc/bachelor-of-advanced-computing-and-bachelor-of-science.html

https://sydney.edu.au/courses/courses/uc/bachelor-of-advanced-computing-and-bachelor-of-science.entryrequirement.year.country.json

https://sydney.edu.au/courses/courses/uc/bachelor-of-advanced-computing-and-bachelor-of-science.fee.json

So if you cannot find other ways to scrape it more automatically from start to end, you may have to use Webscraper to scrape one level, then construct the URLs to scrape such JSON files. (You can construct the JSON URLs manually, perhaps with help of Excel). Next, find some way to parse the JSON files, and rearrange the data to your specific needs.

Tools:

Firefox may present the above JSON URLs in more human readable format.

Google search "online JSON viewers or convertors" They can help, but they will not produce customized CSV output for your specific needs. You may need to write your own script to parse the data to get your needed output format.