Repository navigation
Study abroad schema - #211
Open
NathanP9000 wants to merge 5 commits into
Open
NathanP9000 wants to merge 5 commits into
NathanP9000 wants to merge 5 commits into
Conversation
…nd analyze structure to make a good schema.
KBui4
reviewed
Oct 22, 2026
Contributor
There was a problem hiding this comment.
I don't think schema.go is a good name for this scraper. Rename it to studyAbroad.go or something intuitive.
Contributor
|
Also, please remove the binary file for schema. |
KBui4
reviewed
Oct 22, 2026
| chromedpCtx, cancelFnc = chromedp.NewContext(allocCtx) | ||
|
|
||
| // Detect browser path | ||
| chromePath := os.Getenv("CHROME_PATH") |
Contributor
There was a problem hiding this comment.
I don't think we need this feature. chromedp already searches for chrome automatically.
KBui4
reviewed
Oct 22, 2026
| // records that describe them, to {outDir}/studyAbroad for the parser to combine. | ||
| // Set PROGRAM_IDS to a comma-separated list to scrape specific programs instead of discovering them all. | ||
| func ScrapeStudyAbroad(outDir string) { | ||
| programIDs, err := programIDsFromEnv(os.Getenv("PROGRAM_IDS")) |
Contributor
There was a problem hiding this comment.
Instead of being an environment variable, make it an option to specify program IDs when running the scraper.
nebula-api moved api/ to rest/ and the module is now github.com/UTDNebula/nebula-api. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Scraper saves raw discovery records and brochures; parser merges them into schema.Program. Adds -studyAbroad for scrape and parse, and CHROME_PATH support. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mikehquan19
force-pushed
the
study-abroad-schema
branch
from
October 26, 2026 03:49
ac11df4 to
2197099
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds a study abroad (Terradotta) scraper and parser. Depends on UTDNebula/nebula-api#405 for the Program schema.
The scraper (-scrape -studyAbroad) gets every active program from the discovery search and saves each program's raw brochure. The parser (-parse -studyAbroad) turns the brochures into schema.Program, adds the name, summary, terms and locations from the discovery records, and writes studyAbroad.json.
Also moves all imports from nebula-api/api/schema to nebula-api/rest/schema, since nebula-api moved api/ to rest/, and adds an optional CHROME_PATH to .env.template.
Testing
Ran the full pipeline locally:
./api-tools -scrape -studyAbroad -headless, then./api-tools -parse -studyAbroad. All 147 active programs scraped and parsed.Note: 69 of the 147 programs have no
nextAppCycleCostSheet. That's how Terradotta returns them, not a parser bug.AI Disclosure
I used AI to come up with the code, and I am working on review the code. I made this MR, so others can see and give feedback.
Checklist