Quick Start Guide
Installation
Using Go Install
go install github.com/1mb-dev/lobster/v2/cmd/lobster@latest
From Source
git clone https://github.com/1mb-dev/lobster.git
cd lobster
go build -o lobster ./cmd/lobster
Basic Usage
1. Check Your Local Application
Start your application on port 3000, then run:
lobster -url http://localhost:3000 -allow-private-ips
This will:
- Follow same-host links from the base page up to depth 3 and fetch each page once (on an http base, pages reached through an
https://link are fetched over https as well) - Space requests at 2 per second (default
-rate), shared by 5 workers - Stop when every page has been fetched, or after 2 minutes (default
-duration), which at the default rate allows about 240 requests - Print a summary and a PASS or FAIL line per performance target, and exit 0 on PASS, 1 on FAIL
-allow-private-ips is required for localhost and private addresses.
2. Customize the Crawl
lobster \
-url http://localhost:3000 \
-allow-private-ips \
-rate 10 \
-duration 5m \
-output results.json \
-html report.html
Parameters:
-rate 10: up to 10 requests per second in total-duration 5m: give the crawl up to 5 minutes-output results.json: write the results as JSON-html report.html: write an HTML report of the crawl with a response-time histogram
3. Configure Performance Targets
Create a configuration file config.json:
{
"base_url": "http://localhost:3000",
"rate": 10.0,
"output_file": "results.json",
"performance_targets": {
"p95_response_time_ms": 100,
"success_rate": 99.0
}
}
Run with config:
lobster -config config.json -allow-private-ips
A target left out, or set to 0, is not judged. Only success_rate (99%) is on by default.
4. Add a Load Phase
After a complete crawl, -load replays the pages it found at -rate for a set time:
lobster -url http://localhost:3000 -allow-private-ips -rate 50 -load 30s
Against a public host, or one whose robots.txt sets a Crawl-delay, Lobster also needs -i-own-this and LOBSTER_I_OWN_THIS=<host>. See Configuration.
Understanding Results
The console shows the crawl’s totals, success rate and response times (average, minimum, maximum, p95, p99), whether the crawl completed, a count of URLs per status code, the load phase if one ran, and one PASS or FAIL line per enabled target.
Console output from a crawl of a local 21-page site at -rate 20:
============================================================
RESULTS
============================================================
Duration: 1.001655626s
URLs Discovered: 21
Total Requests: 21
Successful Requests: 21
Failed Requests: 0
Average Response Time: 863.559µs
Min Response Time: 257.549µs
Max Response Time: 5.663142ms
p95 / p99: 1.448154ms / 5.663142ms
Requests/Second: 20.97
Success Rate: 100.00%
Crawl Complete: yes
------------------------------------------------------------
URL VALIDATION SUMMARY
------------------------------------------------------------
HTTP 200: 21 URL(s)
============================================================
============================================================
PERFORMANCE TARGET VALIDATION
============================================================
PASS Success Rate: 100.00%
Target: ≥ 99.0%
Overall: PASS (1/1 targets met)
Key Metrics:
- Success Rate: the percentage of requests that got a 200-399 status
- p95 / p99: 95% (99%) of requests were at least this fast, to within about 9%
- Crawl Complete: every discovered page was requested; an incomplete crawl fails the run
- Requests/Second: requests divided by the crawl’s duration; with
-load, the load phase reports its achieved rate
With Authentication
For authenticated endpoints, use environment variables:
# Basic authentication
export LOBSTER_AUTH_PASSWORD="secret"
lobster -url http://localhost:3000 -allow-private-ips -auth-type basic -auth-username admin
# Bearer token
export LOBSTER_AUTH_TOKEN="your-api-token"
lobster -url http://localhost:3000 -allow-private-ips -auth-type bearer
See Configuration for all authentication options.
Next Steps
- Configuration Reference - All CLI flags and options
- Responsible Use - Only test systems you own
- Issues - Report a problem