Configuration Reference
Lobster is configured with CLI flags, a JSON configuration file, and environment variables for credentials. lobster -help lists every flag.
How a configuration is built
- Defaults, listed in the tables below.
- Configuration file (
-config): each key in the file replaces its default; a key left out keeps it. An unknown key is an error. - CLI flags: a flag replaces the file’s value only when it is set on the command line.
- Validation: an invalid value stops the run with exit code 2 before any request is sent.
# The file sets concurrency 5; the flag overrides it with 20
lobster -config config.json -concurrency 20
CLI Flags
Crawl
| Flag | Type | Default | Description |
|---|---|---|---|
-url |
string | http://localhost:3000 |
Base URL to crawl |
-concurrency |
int | 5 | Number of concurrent workers |
-duration |
duration | 2m | Time limit for the crawl; the run ends sooner when every page has been fetched |
-timeout |
duration | 30s | HTTP request timeout |
-rate |
float | 2.0 | Requests per second, shared by all workers (minimum 0.1) |
-max-depth |
int | 3 | Maximum crawl depth; 0 fetches the base page only |
-queue-size |
int | 10000 | URL queue buffer size |
-user-agent |
string | lobster/<version> |
User-Agent header |
-ignore-robots |
bool | false | Ignore robots.txt directives |
Load
| Flag | Type | Default | Description |
|---|---|---|---|
-load |
duration | none | After a complete crawl, replay the crawled pages that responded 200-399 at -rate for this long |
-i-own-this |
bool | false | Confirm you own the -load target; also needs LOBSTER_I_OWN_THIS=<host> |
Safety
| Flag | Type | Default | Description |
|---|---|---|---|
-allow-private-ips |
bool | false | Allow a localhost or private-IP target |
-insecure-skip-verify |
bool | false | Skip TLS certificate verification; also needs LOBSTER_INSECURE_TLS=true |
Authentication
| Flag | Type | Description |
|---|---|---|
-auth-type |
string | basic, bearer, cookie or header |
-auth-username |
string | Username for basic authentication |
-auth-password-stdin |
bool | Read the password from stdin (one line) |
-auth-token-stdin |
bool | Read the bearer token from stdin (one line) |
-auth-header |
string | Custom header name; the value comes from LOBSTER_AUTH_HEADER or -auth-header-stdin |
-auth-header-stdin |
bool | Read the -auth-header value from stdin (one line) |
Only one of the three -*-stdin flags can be used per run.
Output
| Flag | Type | Default | Description |
|---|---|---|---|
-output |
string | none | JSON results file |
-html |
string | none | HTML report file |
-verbose |
bool | false | Verbose structured logging |
-no-progress |
bool | false | Disable the progress line |
Other
| Flag | Description |
|---|---|
-config |
Path to a JSON configuration file |
-version |
Show the version |
-help |
Show the help message |
Environment Variables
Secrets are read from the environment or stdin, never from flag values, which show in process lists and shell history.
| Variable | Description |
|---|---|
LOBSTER_AUTH_PASSWORD |
Password for basic authentication |
LOBSTER_AUTH_TOKEN |
Bearer token |
LOBSTER_AUTH_COOKIE |
Cookie, as name=value |
LOBSTER_AUTH_HEADER |
Value for the -auth-header header |
LOBSTER_INSECURE_TLS |
Must be true for -insecure-skip-verify |
LOBSTER_I_OWN_THIS |
Must equal the target’s host name for a -load that needs acknowledgement |
# Password from the environment
export LOBSTER_AUTH_PASSWORD="secret"
lobster -url http://localhost:3000 -allow-private-ips -auth-type basic -auth-username admin
# API key header, value from stdin
vault kv get -field=key secret/api | \
lobster -url https://example.com -auth-type header -auth-header X-API-Key -auth-header-stdin
Configuration File
{
"base_url": "http://localhost:3000",
"concurrency": 10,
"duration": "5m",
"timeout": "30s",
"rate": 10.0,
"max_depth": 3,
"queue_size": 10000,
"load": "",
"user_agent": "",
"ignore_robots": false,
"insecure_skip_verify": false,
"verbose": false,
"output_file": "results.json",
"html_file": "report.html",
"auth": {
"type": "basic",
"username": "admin",
"password": "${LOBSTER_AUTH_PASSWORD}"
},
"performance_targets": {
"requests_per_second": 0,
"avg_response_time_ms": 0,
"p95_response_time_ms": 100,
"p99_response_time_ms": 0,
"success_rate": 99.0
}
}
An empty load means no load phase; an empty user_agent means lobster/<version>.
Environment Variable Substitution
${VAR_NAME} in the file is replaced with the variable’s value; ${VAR_NAME:-default} falls back to a default. An unset variable without a default is an error.
{
"auth": {
"type": "bearer",
"token": "${API_TOKEN}"
}
}
Auth Configuration Options
| Field | Type | Description |
|---|---|---|
type |
string | basic, bearer, cookie or header |
username |
string | Username for basic auth |
password |
string | Password (use ${ENV_VAR}) |
token |
string | Bearer token (use ${ENV_VAR}) |
cookies |
object | Name-value pairs for cookie auth |
headers |
object | Name-value pairs for header auth |
cookie_file |
string | Path to a Netscape-format cookie file |
Performance Targets
A target of 0 is off. Only success_rate is on by default.
| Field | Default | Passes when |
|---|---|---|
success_rate |
99.0 | the percentage of requests with a 200-399 status is at least this; a -load request also needs its body read within -timeout |
avg_response_time_ms |
0 (off) | the average response time is at most this |
p95_response_time_ms |
0 (off) | the 95th-percentile response time is at most this |
p99_response_time_ms |
0 (off) | the 99th-percentile response time is at most this |
requests_per_second |
0 (off) | the -load phase’s achieved rate is at least this; skipped without -load |
Percentiles use the nearest-rank method over a histogram whose buckets are about 9% wide, so a reported percentile is at most one bucket above the exact value.
With -load, the latency and success-rate targets judge the load phase, and the crawl is judged by its own success rate.
Exit Codes
| Code | Meaning |
|---|---|
| 0 | The crawl completed and every enabled target passed |
| 1 | A target failed, the crawl was incomplete, or the load phase aborted or left requests unsent |
| 2 | Usage or configuration error, a refused base URL, a refused -load, or a runtime error |
The JSON report’s performance_validation.overall_status is PASS or FAIL, matching exit codes 0 and 1.
lobster -url http://localhost:3000 -allow-private-ips -config targets.json -no-progress
echo "exit code: $?"
A crawl is complete when every discovered URL was requested: none dropped from a full queue (raise -queue-size) and none cut off by -duration (raise -duration or -rate), and no page’s body was cut off before all its links were read (links_cut_off: raise -timeout or -duration). The JSON report gives crawl_complete and, when it is false, incomplete_reason.
Links are read from the first 10MB of each HTML page. A page that declares a larger Content-Length is not read for links, and the rest of a longer page is not read; neither makes the crawl incomplete.
Load Phase
-load 30s runs after a complete crawl. It sends one request every 1/-rate seconds to the crawled pages that responded 200-399, in turn:
- Requests go out on schedule however slowly the server responds, and latency is measured from each request’s scheduled send time, so a slow server shows in the latency instead of lowering the rate.
- At most 256 requests are in flight. A request that waits for a free slot is counted as late, and so is every request after it due before that slot came free.
- No request due after the
-loadwindow is sent. A request due inside it but not sent when it ends goes out afterwards only if no wait for a slot delayed it and a slot is free; otherwise it is counted as unsent and fails the run. The run ends once the requests in flight finish (within-timeout). - Late and unsent do not say what filled the slots: a slow server does, and so does the burst of overdue requests lobster sends after it stalls itself (on a busy machine, for example). Such a burst near the end of the window can fail the run against a server that never slowed down.
- 429 responses are not retried. After 10 consecutive 429, 5xx, connection errors (a connection refused, reset or timed out, or a body cut off or not read within
-timeout) or refusals (an address or redirect lobster does not allow), the load stops, and the run exits 1. -rateis capped at 1000 with-load.-concurrencyapplies to the crawl only.
A load against a host that resolves to a public address, or whose robots.txt sets a Crawl-delay, needs both -i-own-this and LOBSTER_I_OWN_THIS set to the host name. -allow-private-ips does not count. A load that needs neither can connect only to private addresses.
LOBSTER_I_OWN_THIS=staging.example.com lobster -url https://staging.example.com \
-rate 50 -load 1m -i-own-this -output results.json
Examples
# Local application with defaults
lobster -url http://localhost:3000 -allow-private-ips
# Base page only
lobster -url http://localhost:3000 -allow-private-ips -max-depth 0
# Faster crawl with a longer time limit, JSON and HTML reports
lobster -url http://localhost:3000 -allow-private-ips -rate 10 -duration 5m \
-output results.json -html report.html
# Internal service with a self-signed certificate
export LOBSTER_INSECURE_TLS=true
lobster -url https://dev.internal:8443 -insecure-skip-verify -allow-private-ips
Tuning
Rate and Concurrency
-rate is one limit shared by all workers, with a burst of 1: requests are spaced 1/-rate seconds apart whatever -concurrency is. A robots.txt Crawl-delay lowers the crawl’s rate to one request per delay. Below 1.0 req/s, Lobster asks for confirmation on a terminal.
More workers help when responses are slow: with a 1s response time and -rate 10, about 10 workers keep the rate.
Queue Size
The default of 10,000 suits most sites. If the run reports incomplete_reason: queue_full, raise -queue-size.
robots.txt
Lobster respects robots.txt by default, and checks it again on every redirect. Use -ignore-robots only when you own the site or have explicit permission.
Troubleshooting
Many 429 or timeout errors
Lower -rate or raise -timeout.
Only the base page is fetched
Check that the page has <a href> links to the same host, and that -max-depth is above 0.
An incomplete crawl is covered under Exit Codes.