hack3rs.ca network-security

/learning/tools/spiderfoot :: tool-guide-20

defender@hack3rs:~/learning/tools$ open spiderfoot

SpiderFoot

OSINT automation / asset discovery

SpiderFoot automates OSINT collection and correlation for asset discovery, exposure reviews, and reconnaissance awareness in authorized defensive workflows.

how-to-learn-this-tool-like-a-defender

Work through the stages in order. Each one builds on the previous. Skipping straight to 'run a command' without knowing what the output means is how analysts end up misreading evidence under pressure.

$Name the specific question this tool answers — and one question it cannot answer alone.
$Run the simplest command in a lab against a host you control; read every field in the output before moving on.
$Identify which output fields are direct evidence and which are inferences the tool made on your behalf.
$Pull a second source — a log, a PCAP, a SIEM event — that either confirms or contradicts what the tool reported.
$Write down the exact command you ran, what you expected, what you got, and what you are doing next.

preflight-checklist-before-using-tool

$Confirm in writing: who authorized this, what hosts are in scope, and what the maximum acceptable impact is.
$State the question you are trying to answer — not 'run the tool' but 'confirm whether port 443 is open on 10.10.20.15'.
$Name the second source you will use if the tool output is ambiguous (log, PCAP, CMDB, another tool).
$Record the start time, the host or interface you ran it on, and the exact command — enough for another analyst to reproduce it.
$Know what normal output looks like for this host before you run anything in anger.

how-experts-read-output

$Field recognition: identify the two or three fields that directly answer your question and ignore the rest for now.
$Scope check: confirm the output covers the host, interface, and time window you intended — not a cached or adjacent result.
$Evidence type: is this a direct observation (packet captured, port open) or an inference the tool made (service guessed from banner)?
$Correlation: name the one other source — a log line, a PCAP stream, a CMDB entry — that would confirm or contradict this.
$Decision: close the question, escalate with evidence, refine the scope, or collect another source — pick one and do it.

official-links

open docs Reference docs for SpiderFoot — commands, flags, config options release notes Changelog and release history for SpiderFoot project site SpiderFoot project home and maintainer information

ethical-use-and-defense-scope

SpiderFoot should only touch systems you are authorized to test. That includes passive tools: if it collects data, hits a service, or reads credentials, you need explicit scope before it runs.

Before you run it: write down the target, the expected output, and the stop condition. After you run it: save the commands and output so another analyst can reproduce your results without guessing.

Defenders use SpiderFoot to find gaps — weak configs, missing detections, credential exposure, protocol abuse paths. Attackers use the same output. Keep that tension in mind. The measure of a good lab session is what you hardened or detected afterward, not how far the tool ran.

tool-history-origin-and-purpose

$When created: SpiderFoot emerged in the 2010s as an OSINT automation platform.
$Why it was created: Defenders needed a way to scale repetitive OSINT tasks and identify public exposure faster than manual lookups alone.

It was created to automate and correlate open-source intelligence checks across many modules for infrastructure and identity exposure review.

why-defenders-still-use-it

Security teams use SpiderFoot for external attack-surface discovery, asset inventory validation, and awareness of data exposed in public sources.

How the tool evolved

+Expanded module ecosystem for many OSINT data sources.
+Used in both security operations and assessment workflows.
+Helpful for exposure review and training on OSINT correlation.

when-this-tool-is-a-good-fit

+External attack surface discovery and exposure inventory.
+Recon awareness training for defenders.
+Asset validation and public footprint review.

when-to-use-another-tool-or-source

!When you need host process/user context, pair with endpoint or OS logs.
!When you need ownership and business impact, pair with CMDB/ticketing/asset context.
!When the tool output is ambiguous, validate using a second evidence source before concluding.
!When production risk is high, test in a lab first and use change coordination.

1. Where SpiderFoot Fits in a Defender's Workflow

SpiderFoot automates OSINT collection and correlation for asset discovery, exposure reviews, and reconnaissance awareness in authorized defensive workflows.

The role here is "OSINT automation / asset discovery." That scoping matters. A triage tool used as an investigation tool produces the wrong level of depth; an investigation tool used as a monitoring tool burns analyst time. Pick the right phase, then pick the tool.

Start with a concrete question — "Is this service reachable from the DMZ?" or "Do we have stale DNS records for this domain?" — rather than opening the tool and seeing what turns up.

2. Running It Safely and Repeatably

Write down target scope, authorized impact, and a stop condition before you run anything. If the lab ends and your notes only say "ran SpiderFoot", the session didn't count as learning.

Baseline first. Collect one clean-state output, label it, save it. Then make your change or run your test. Without a before state, you can't tell what SpiderFoot actually found versus what was already there.

No tool output is self-contained. Pair SpiderFoot findings with packet captures, host logs, asset inventory, and change tickets before drawing conclusions.

3. Reading Output Like an Analyst

SpiderFoot output answers a narrow question. Check scope first: right host, right interface, right time window, right protocol layer. If any of those are off, the output is misleading, not wrong — a subtler problem.

Collect a known-good example before chasing anomalies. An analyst who has only ever seen bad output can't explain why something is suspicious — they can only say it looks different. Baseline removes that ambiguity.

Every output review ends with a decision: close it, escalate it, tune a detection, patch something, or collect more evidence. "Interesting" without a next action isn't a finding.

4. Lab Design and Practice

One goal per lab session. Not "learn SpiderFoot" — something specific: validate a lockout policy, catch a stale record, confirm a port is filtered end-to-end. Narrow goals produce usable results.

Run both the normal case and the failing case in the same session. The contrast is what builds judgment. Analysts who have only seen success don't recognize partial failures under pressure.

Finish with a written summary: what you observed, the evidence behind it, what you still don't know, and one thing that should change — a control, a detection, a runbook entry. That summary is the actual output of the lab.

scenario-teaching-playbooks

Work through each scenario step by step. The goal is to practice making decisions with the tool — not just executing commands — so the workflow becomes automatic before you need it under pressure.

1. External attack surface discovery and exposure inventory.

Suggested starting block: Orientation And Safe Startup

$Write the question you need to answer and the exact hosts or segments you are authorized to inspect.
$Run the first command from the selected command block; note the timestamp and interface used.
$Read the output field by field — identify what the tool confirmed versus what it inferred.
$Check a second source (host log, SIEM alert, PCAP, ticket, or CMDB record) that covers the same time window.
$Write one sentence stating your finding, your confidence level, and the next action.

2. Recon awareness training for defenders.

Suggested starting block: Defender Notes And Evidence Workflow

$Write the question you need to answer and the exact hosts or segments you are authorized to inspect.
$Run the first command from the selected command block; note the timestamp and interface used.
$Read the output field by field — identify what the tool confirmed versus what it inferred.
$Check a second source (host log, SIEM alert, PCAP, ticket, or CMDB record) that covers the same time window.
$Write one sentence stating your finding, your confidence level, and the next action.

3. Asset validation and public footprint review.

Suggested starting block: Correlation And Follow-Up

$Write the question you need to answer and the exact hosts or segments you are authorized to inspect.
$Run the first command from the selected command block; note the timestamp and interface used.
$Read the output field by field — identify what the tool confirmed versus what it inferred.
$Check a second source (host log, SIEM alert, PCAP, ticket, or CMDB record) that covers the same time window.
$Write one sentence stating your finding, your confidence level, and the next action.

cli-workflows

Lab-safe commands for authorized environments. Run each one, read the output, and note what field or value tells you something useful before moving to the next.

cli-walkthroughs-with-expected-output

One command per block, with sample output. Study the output before you run the command yourself — you should recognize what you are looking at when it appears on your screen.

Orientation And Safe Startup

Beginner

Command

python3 sf.py -h

Example Output

SpiderFoot 4.0: Open Source Intelligence Automation.
usage: sf.py [-h] [-d] [-l IP:port] [-m mod1,mod2,...] [-s TARGET]
             [-t type1,type2,...] [-T] [-o tab|csv|json] [--version]

optional arguments:
  -h, --help    show this help message and exit
  -l IP:port    IP and port to listen on (default: 127.0.0.1:5001)
  -s TARGET     target for scanning
  -m MODULES    comma-separated list of modules to use

$ how to read it: Read the key fields — host, port, protocol, state — then ask whether the output answers the question you started with. If it raises a new question instead, collect a second source before drawing a conclusion.

Defender Notes And Evidence Workflow

Intermediate

Command

printf "goal:
scope:
expected_output:
normal_pattern:
failure_pattern:
next_action:
" > tool-labs/spiderfoot/notes/session.txt

Example Output

goal: enumerate digital footprint of authorized test target via OSINT automation
scope: lab-company.example — authorized test target, no real organizations in scope
expected_output: SpiderFoot report with discovered IPs, emails, domains, and data sources
normal_pattern: only lab-created test data returned, no real PII surfaced
failure_pattern: API errors, rate limits, or real organization data appears in output
next_action: document discovered exposures, assess scope accuracy, export HTML report

Correlation And Follow-Up

Advanced

Command

journalctl --since "-15 min" | tail -n 40 || true

Example Output

Mar 17 10:12:01 lab-host sshd[2341]: Accepted publickey for analyst from 10.0.0.5 port 54321
Mar 17 10:12:44 lab-host sudo[2345]: analyst : TTY=pts/0 ; COMMAND=/usr/bin/systemctl status
Mar 17 10:14:03 lab-host systemd[1]: Started Session 4 of user analyst.
Mar 17 10:15:19 lab-host kernel: [UFW ALLOW] IN=eth0 SRC=10.0.0.5 DST=10.0.0.10 PROTO=TCP DPT=443

command-anatomy-and-expert-usage

Each card explains what the command is for, what can go wrong, and what the output means. Syntax is easy to look up; knowing which command to reach for — and what to ignore in the output — is the skill worth building.

Orientation And Safe Startup

Beginner

Command

python3 sf.py -h

Command Anatomy

$Base command: python3
$Primary arguments/options: sf.py -h
$Operator goal: know what answer you expect before you run it; if the output surprises you, investigate before concluding.

Use And Risk

$ intent: Collect, validate, or document evidence in a defensive workflow.

$ risk: Review command impact before running; validate in lab first if uncertain.

$ learning focus: Baseline command: learn what normal output looks like.

Show sample output and interpretation notes

SpiderFoot 4.0: Open Source Intelligence Automation.
usage: sf.py [-h] [-d] [-l IP:port] [-m mod1,mod2,...] [-s TARGET]
             [-t type1,type2,...] [-T] [-o tab|csv|json] [--version]

optional arguments:
  -h, --help    show this help message and exit
  -l IP:port    IP and port to listen on (default: 127.0.0.1:5001)
  -s TARGET     target for scanning
  -m MODULES    comma-separated list of modules to use

$ expert reading pattern: Check that the scope matches what you intended, pick out the two or three fields that answer your question, then find one other source that confirms before you act.

Orientation And Safe Startup

Beginner

Command

python3 sf.py -h || true

Command Anatomy

$Base command: python3
$Primary arguments/options: sf.py -h || true
$Operator goal: know what answer you expect before you run it; if the output surprises you, investigate before concluding.

Use And Risk

$ intent: Collect, validate, or document evidence in a defensive workflow.

$ risk: Review command impact before running; validate in lab first if uncertain.

$ learning focus: Intermediate step: refine scope or extract more useful evidence.

Show sample output and interpretation notes

SpiderFoot 4.0: Open Source Intelligence Automation.
usage: sf.py [-h] [-d] [-l IP:port] [-m mod1,mod2,...] [-s TARGET]
             [-t type1,type2,...] [-T] [-o tab|csv|json] [--version]

optional arguments:
  -h, --help    show this help message and exit
  -l IP:port    IP and port to listen on (default: 127.0.0.1:5001)
  -s TARGET     target for scanning
  -m MODULES    comma-separated list of modules to use

$ expert reading pattern: Check that the scope matches what you intended, pick out the two or three fields that answer your question, then find one other source that confirms before you act.

Orientation And Safe Startup

Beginner

Command

mkdir -p tool-labs/spiderfoot/{notes,artifacts,screenshots}

Command Anatomy

$Base command: mkdir
$Primary arguments/options: -p tool-labs/spiderfoot/{notes,artifacts,screenshots}
$Operator goal: know what answer you expect before you run it; if the output surprises you, investigate before concluding.

Use And Risk

$ intent: Collect, validate, or document evidence in a defensive workflow.

$ risk: Review command impact before running; validate in lab first if uncertain.

$ learning focus: Advanced step: use after baseline and validation are understood.

Show sample output and interpretation notes

# no output — directory created successfully

$ expert reading pattern: Check that the scope matches what you intended, pick out the two or three fields that answer your question, then find one other source that confirms before you act.

Defender Notes And Evidence Workflow

Intermediate

Command

printf "goal:
scope:
expected_output:
normal_pattern:
failure_pattern:
next_action:
" > tool-labs/spiderfoot/notes/session.txt

Command Anatomy

$Base command: printf
$Primary arguments/options: "goal: scope: expected_output: normal_pattern: failure_pattern:
$Operator goal: know what answer you expect before you run it; if the output surprises you, investigate before concluding.

Use And Risk

$ intent: Collect, validate, or document evidence in a defensive workflow.

$ risk: Review command impact before running; validate in lab first if uncertain.

$ learning focus: Baseline command: learn what normal output looks like.

Show sample output and interpretation notes

goal: enumerate digital footprint of authorized test target via OSINT automation
scope: lab-company.example — authorized test target, no real organizations in scope
expected_output: SpiderFoot report with discovered IPs, emails, domains, and data sources
normal_pattern: only lab-created test data returned, no real PII surfaced
failure_pattern: API errors, rate limits, or real organization data appears in output
next_action: document discovered exposures, assess scope accuracy, export HTML report

$ expert reading pattern: Check that the scope matches what you intended, pick out the two or three fields that answer your question, then find one other source that confirms before you act.

Defender Notes And Evidence Workflow

Intermediate

Command

cat tool-labs/spiderfoot/notes/session.txt

Command Anatomy

$Base command: cat
$Primary arguments/options: tool-labs/spiderfoot/notes/session.txt
$Operator goal: know what answer you expect before you run it; if the output surprises you, investigate before concluding.

Use And Risk

$ intent: Collect, validate, or document evidence in a defensive workflow.

$ risk: Review command impact before running; validate in lab first if uncertain.

$ learning focus: Intermediate step: refine scope or extract more useful evidence.

Show sample output and interpretation notes

goal:
scope:
expected_output:
normal_pattern:
failure_pattern:
next_action:

$ expert reading pattern: Check that the scope matches what you intended, pick out the two or three fields that answer your question, then find one other source that confirms before you act.

Defender Notes And Evidence Workflow

Intermediate

Command

printf "timestamp,observation,confidence,validation_source
" > tool-labs/spiderfoot/notes/evidence.csv

Command Anatomy

$Base command: printf
$Primary arguments/options: "timestamp,observation,confidence,validation_source " > tool-labs/spiderfoot/notes/evidence.csv
$Operator goal: know what answer you expect before you run it; if the output surprises you, investigate before concluding.

Use And Risk

$ intent: Collect, validate, or document evidence in a defensive workflow.

$ risk: Review command impact before running; validate in lab first if uncertain.

$ learning focus: Advanced step: use after baseline and validation are understood.

Show sample output and interpretation notes

timestamp  observation  confidence  validation_source
" > tool-labs/spiderfoot/notes/evidence.csv

$ expert reading pattern: Check that the scope matches what you intended, pick out the two or three fields that answer your question, then find one other source that confirms before you act.

Correlation And Follow-Up

Advanced

Command

journalctl --since "-15 min" | tail -n 40 || true

Command Anatomy

$Base command: journalctl
$Primary arguments/options: --since "-15 min" | tail
$Operator goal: know what answer you expect before you run it; if the output surprises you, investigate before concluding.

Use And Risk

$ intent: Collect, validate, or document evidence in a defensive workflow.

$ risk: Review command impact before running; validate in lab first if uncertain.

$ learning focus: Baseline command: learn what normal output looks like.

Show sample output and interpretation notes

Mar 17 10:12:01 lab-host sshd[2341]: Accepted publickey for analyst from 10.0.0.5 port 54321
Mar 17 10:12:44 lab-host sudo[2345]: analyst : TTY=pts/0 ; COMMAND=/usr/bin/systemctl status
Mar 17 10:14:03 lab-host systemd[1]: Started Session 4 of user analyst.
Mar 17 10:15:19 lab-host kernel: [UFW ALLOW] IN=eth0 SRC=10.0.0.5 DST=10.0.0.10 PROTO=TCP DPT=443

$ expert reading pattern: Check that the scope matches what you intended, pick out the two or three fields that answer your question, then find one other source that confirms before you act.

Correlation And Follow-Up

Advanced

Command

tshark -r sample.pcap -q -z io,phs || true

Command Anatomy

$Base command: tshark
$Primary arguments/options: -r sample.pcap -q -z io,phs
$Operator goal: know what answer you expect before you run it; if the output surprises you, investigate before concluding.

Use And Risk

$ intent: Packet capture, packet summary, or PCAP slicing for evidence.

$ risk: Review command impact before running; validate in lab first if uncertain.

$ learning focus: Intermediate step: refine scope or extract more useful evidence.

Show sample output and interpretation notes

Protocol Hierarchy Statistics
eth
 ip
  tcp
   tls
  udp
   dns

$ expert reading pattern: Check that the scope matches what you intended, pick out the two or three fields that answer your question, then find one other source that confirms before you act.

Correlation And Follow-Up

Advanced

Command

printf "finding,owner,action,status
" > tool-labs/spiderfoot/notes/actions.csv

Command Anatomy

$Base command: printf
$Primary arguments/options: "finding,owner,action,status " > tool-labs/spiderfoot/notes/actions.csv
$Operator goal: know what answer you expect before you run it; if the output surprises you, investigate before concluding.

Use And Risk

$ intent: Collect, validate, or document evidence in a defensive workflow.

$ risk: Review command impact before running; validate in lab first if uncertain.

$ learning focus: Advanced step: use after baseline and validation are understood.

Show sample output and interpretation notes

finding  owner  action  status
" > tool-labs/spiderfoot/notes/actions.csv

$ expert reading pattern: Check that the scope matches what you intended, pick out the two or three fields that answer your question, then find one other source that confirms before you act.

Orientation And Safe Startup

python3 sf.py -h
python3 sf.py -h || true
mkdir -p tool-labs/spiderfoot/{notes,artifacts,screenshots}

Defender Notes And Evidence Workflow

printf "goal:
scope:
expected_output:
normal_pattern:
failure_pattern:
next_action:
" > tool-labs/spiderfoot/notes/session.txt
cat tool-labs/spiderfoot/notes/session.txt
printf "timestamp,observation,confidence,validation_source
" > tool-labs/spiderfoot/notes/evidence.csv

Correlation And Follow-Up

journalctl --since "-15 min" | tail -n 40 || true
tshark -r sample.pcap -q -z io,phs || true
printf "finding,owner,action,status
" > tool-labs/spiderfoot/notes/actions.csv

defensive-use-cases

$External attack surface discovery and exposure inventory.
$Recon awareness training for defenders.
$Asset validation and public footprint review.

common-mistakes

$Treating OSINT findings as current truth without verification.
$Failing to document timestamps and source reliability.
$Scanning outside policy or violating source terms/rate limits.

expert-habits-for-free-self-study

Free teaching resource. The loop that makes analysts better: ask a precise question, collect evidence, read it carefully, validate against a second source, document what you found, and repeat with a harder question.

$Pick the least disruptive command that can still answer the question — then run that one first.
$Before you look at output, write one sentence stating what you expect to see.
$Mark each output field as 'observed' or 'inferred by tool' before acting on it.
$Save the exact command with flags and target — not a paraphrase — so another analyst can run the same thing.
$During a quiet period, capture what normal output looks like from key hosts; store those samples where you can find them during an incident.
$When you escalate, include the command output, the timestamp, and one sentence on why it matters — not just 'looks suspicious'.

knowledge-check

?What question is this tool best suited to answer first?
?What permissions or scope approvals are needed before using it?
?Which second evidence source should you pair with it for higher confidence?
?What does normal output look like for your environment?

teaching-answer-guide

Show teaching hints

#Start from the tool’s role and the scenario you are investigating.
#Never rely on one tool alone for high-confidence incident decisions.
#Document normal output patterns during calm periods so anomalies are easier to spot.
#Prefer lab validation for new commands, rules, or scans before production use.

practice-plan

# Pick one specific question SpiderFoot can answer in your lab, write it down, then write the authorized scope before opening the tool.
# Run the normal case first. Save the output, label it, note the exact command. That is your baseline.
# Run the failure or misconfiguration case. Document what changed in the output and how you would recognize it without already knowing the answer.
# Write a three-sentence summary: what you observed, what evidence supports it, and what you would do next in a real incident.

<- previous tool Steghide -> next tool Metasploit Framework