Tree @master (Download .tar.gz)
Commit History
@master
git clone https://simonvolpert.com/fediscover/
- Add a "verbose" option to print every URL being processed Simon Volpert 4 years ago
- Add skipped URL cound in the post-crawl report Simon Volpert 4 years ago
- Add blacklisting Simon Volpert 4 years ago
- Remove duplicate matches URLs from pages when scraping Simon Volpert 4 years ago
- Automatically crawl the next URL if the profile queue is empty Simon Volpert 4 years ago
- Add following/follower URLs to queue when crawling a profile URL Simon Volpert 4 years ago
- Refactor URL loading code Simon Volpert 4 years ago
- Refactor URL crawling Simon Volpert 4 years ago
- Move page scraping code into its own function Simon Volpert 4 years ago
- Add some code comments Simon Volpert 4 years ago
- Add "dry-run" and "random" options Simon Volpert 4 years ago
- Normalize profile URLs passed on the command line before processing Simon Volpert 4 years ago
- Put print-to-standard-error code into its own function Simon Volpert 4 years ago
- Move repeating URL caching code into a function Simon Volpert 4 years ago
- Move newly processed link count to the appropriate session state containers Simon Volpert 4 years ago
- Print all unimportant messages to standard error Simon Volpert 4 years ago
- Reword follow page regex to satisfy the linter Simon Volpert 4 years ago
- Fix use-before-declaring bug Simon Volpert 4 years ago
- Process profile URLs passed to "crawl" correctly Simon Volpert 4 years ago
- Insert additional profile pages in the front of the crawlable URL list Simon Volpert 4 years ago
- Enforce crawlable URL uniqueness Simon Volpert 4 years ago
- Add a README and a LICENSE Simon Volpert 4 years ago
- Add some code comments Simon Volpert 4 years ago
- Add timeout to page load Simon Volpert 4 years ago
- Locate all the pages of the user's following/followers URL Simon Volpert 4 years ago
- Include followers in crawling URLs Simon Volpert 4 years ago
- Write a trailing newline into cache files Simon Volpert 4 years ago
- Lint the code Simon Volpert 4 years ago
- Extract profile URLs and following links from downloaded page Simon Volpert 4 years ago
- Add web page loader and cacher Simon Volpert 4 years ago