Tree @master (Download .tar.gz)
Commit History
@master
git clone https://simonvolpert.com/fediscover/
- Add skipped URL cound in the post-crawl report Simon Volpert 2 years ago
- Add blacklisting Simon Volpert 2 years ago
- Remove duplicate matches URLs from pages when scraping Simon Volpert 2 years ago
- Automatically crawl the next URL if the profile queue is empty Simon Volpert 2 years ago
- Add following/follower URLs to queue when crawling a profile URL Simon Volpert 2 years ago
- Refactor URL loading code Simon Volpert 2 years ago
- Refactor URL crawling Simon Volpert 2 years ago
- Move page scraping code into its own function Simon Volpert 2 years ago
- Add some code comments Simon Volpert 2 years ago
- Add "dry-run" and "random" options Simon Volpert 2 years ago
- Normalize profile URLs passed on the command line before processing Simon Volpert 2 years ago
- Put print-to-standard-error code into its own function Simon Volpert 2 years ago
- Move repeating URL caching code into a function Simon Volpert 2 years ago
- Move newly processed link count to the appropriate session state containers Simon Volpert 2 years ago
- Print all unimportant messages to standard error Simon Volpert 2 years ago
- Reword follow page regex to satisfy the linter Simon Volpert 2 years ago
- Fix use-before-declaring bug Simon Volpert 2 years ago
- Process profile URLs passed to "crawl" correctly Simon Volpert 2 years ago
- Insert additional profile pages in the front of the crawlable URL list Simon Volpert 2 years ago
- Enforce crawlable URL uniqueness Simon Volpert 2 years ago
- Add a README and a LICENSE Simon Volpert 2 years ago
- Add some code comments Simon Volpert 2 years ago
- Add timeout to page load Simon Volpert 2 years ago
- Locate all the pages of the user's following/followers URL Simon Volpert 2 years ago
- Include followers in crawling URLs Simon Volpert 2 years ago
- Write a trailing newline into cache files Simon Volpert 2 years ago
- Lint the code Simon Volpert 2 years ago
- Extract profile URLs and following links from downloaded page Simon Volpert 2 years ago
- Add web page loader and cacher Simon Volpert 2 years ago
- Select a random entry from the unprocessed user list to show Simon Volpert 2 years ago