Development Guide
Branches, commits and pull requests: Contributing.
Architecture
cmd/lobster/ # run(): flags, exit codes, crawl then optional load
internal/cli/ # Flag registration, config building, auth, help text
internal/config/ # Configuration file loading (strict keys, ${VAR} substitution)
internal/domain/ # Config, results and their validation
internal/crawler/ # Link extraction, same-host filter, deduplication
internal/tester/ # Crawl: worker pool, rate limit, redirect policy, completeness
internal/load/ # -load: open-loop schedule, ownership gate
internal/stats/ # Fixed log-bucket histogram, nearest-rank percentiles
internal/validator/ # Targets and the PASS/FAIL verdict
internal/reporter/ # Console, JSON and HTML reports
internal/robots/ # robots.txt parsing
internal/util/ # Address checks, URL validation, sanitization
Flow: crawler → tester → (load) → validator + reporter.
Development Workflow
Setup
git clone https://github.com/1mb-dev/lobster.git
cd lobster
make build
./lobster -url http://localhost:3000 -allow-private-ips
make help lists all build targets.
Running Tests
make test # every package under the race detector, integration tests included
make test-verbose # the same, verbose
make test-coverage # coverage report
make lint # golangci-lint
go test ./internal/tester -run TestRedirect # matching tests in one package
go test ./cmd/lobster -run TestRun_JSONKeysMatchGolden -update # regenerate the JSON key golden files
- Tests use
httptestservers only, never real endpoints; setIgnoreRobots: trueunless robots.txt is under test. cmd/lobster/main_test.goruns the whole command throughrun()and checks exit codes, output and the JSON keys againstcmd/lobster/testdata/*.golden. A JSON key change fails it until the golden files are regenerated.- The rate, Crawl-delay and load tests run on real clocks and take seconds each.
Key Implementation Details
Crawl
- One goflow token bucket with a burst of 1 spaces requests
1/-rateapart across all workers; a robots.txtCrawl-delaylowers the rate to1/delay. - A pending counter tracks queued and in-flight URLs; the run ends when it reaches zero, or at the
-durationdeadline. The URL queue is never closed. - The crawl is complete when nothing is pending, no URL was dropped from a full queue, and no request was cut off by the deadline, and no page’s body was cut off before its links were read (the links read before the cut are still queued).
- Links come from streaming up to
MaxResponseSize(10MB default) of each HTML page throughhtml.Tokenizer; a largerContent-Lengthskips the page. When the base page is served over https (after redirects), http links on port 80 are fetched over https and the two spellings of a page are one; no link is fetched over a weaker scheme than it names.
Load
- Request
iis due atstart + i/rate; the schedule never waits for responses, and latency is measured from the due time. - Up to 256 requests are in flight; a request that waits for a slot, and every request after it due before that slot came free, is counted late. No request due after the window is sent, and none waits for a slot after it; a request not sent by its end goes out only if it fell due after the latest waited-for slot came free and a slot is free, and is otherwise unsent and fails the run.
- Ten consecutive 429, 5xx, connection errors or refusals abort the load.
Connection Safety
util.DialControlchecks every address at dial time, after DNS resolution; there is no proxy.- The redirect policy follows only the base origin (or its http-to-https upgrade), at most 5 hops, and checks robots.txt again.
- A load that needed no ownership acknowledgement also refuses public addresses at dial time.
Metrics
Response times go into a fixed histogram of 224 log buckets (8 per octave), so memory does not grow with the number of requests. Percentiles use the nearest rank and report the upper bound of the bucket holding it: at most about 9% above the exact value.
Key Dependencies
goflow: token bucket rate limiting for the crawl (internal/tester).
golang.org/x/net: html.Tokenizer for link extraction (internal/crawler).
Common Tasks
Adding a Flag
- Register it in
internal/cli/config.go(RegisterConfigFlagsandflagSetters), or incmd/lobster/main.goif it is not part of the configuration. - Add the field and its JSON key to
domain.Config, and validate it inConfig.Validate. - Add it to the help text in
internal/cli/display.goand todocs/configuration.md.
Release Checklist
- Update
CHANGELOG.md. make cipasses.- Merge to
main; wait for CI to pass on the merge commit. - Tag the merge commit with an annotated tag whose message is the release’s pet name, or empty for an unnamed release:
git tag -a vX.Y.Z -m "<name>". Push the tag. The release workflow creates a draft release titledvX.Y.Z - <name>(orvX.Y.Z) with generated notes. - Replace the generated notes with what upgraders must do, any security lines, and the changelog and compare links, then publish.
-version reads the version from the build: make build passes git describe through ldflags, and go install records the module version.