Extractor

Indicator extraction.

:class:IOCExtractor is the configurable entry point. :func:extract runs it with the defaults: normalization, validation, and deduplication on, and the noisy types (file paths, user agents, mutexes) left out.

Classes:

• IOCExtractor –

  Configurable IOC processor.

Functions:

• resolve_types –

  Resolve extract_types into an ordered list of type keys.

• extract –

  Extract indicators with the default :class:IOCExtractor settings.

• merge –

  Merge extractions that were produced with the same settings.

Attributes:

• logger –
• Validator –
• PageSpan –

logger module-attribute

logger = logging.getLogger('qarbaal.extract')

Validator module-attribute

Validator = Callable[[Indicator], bool]

PageSpan module-attribute

PageSpan = tuple[int, int, int]

IOCExtractor dataclass

IOCExtractor(extract_types: Sequence[str] | None = None, normalize: bool = True, validate: bool = True, deduplicate: bool = True, hide_private: bool = False, drop_invalid: bool = False, max_input_chars: int = 2000000, max_matches_per_type: int = 20000, strict_limits: bool = False, allow: Sequence[str] | None = None, deny: Sequence[str] | None = None, defang_patterns: Sequence[tuple[str, str]] | None = None, custom_patterns: Sequence[PatternSpec] | None = None, validators: Sequence[Validator] | None = None, chunk_overlap: int = 8192, chunk_size: int = 1048576)

Configurable IOC processor.

The default instance extracts the high-precision types, refangs defanged text, normalizes values, validates them, and deduplicates by fingerprint. File paths, user agents, and mutexes are opt-in through extract_types.

Parameters:

• extract_types (Sequence[str] | None, default: None ) –

  Type names to extract. None uses the default set.

• normalize (bool, default: True ) –

  Canonicalize values and apply NFKC before the sieve.

• validate (bool, default: True ) –

  Record structural validation notes and a confidence score.

• deduplicate (bool, default: True ) –

  Merge repeats that share a fingerprint.

• hide_private (bool, default: False ) –

  Omit tagged IPv4, IPv6, and CIDR results.

• drop_invalid (bool, default: False ) –

  Omit indicators that fail validation.

• max_input_chars (int, default: 2000000 ) –

  Maximum characters scanned from one source.

• max_matches_per_type (int, default: 20000 ) –

  Cap on raw matches for a single type.

• strict_limits (bool, default: False ) –

  Raise :class:~qarbaal.exceptions.InputTooLargeError instead of truncating.

• allow (Sequence[str] | None, default: None ) –

  Glob patterns. When set, an indicator must match one of them.

• deny (Sequence[str] | None, default: None ) –

  Glob patterns. A match drops the indicator.

• defang_patterns (Sequence[tuple[str, str]] | None, default: None ) –

  Extra (regex, replacement) refang rules, applied before the built-in rules.

• custom_patterns (Sequence[PatternSpec] | None, default: None ) –

  Extra regular expressions. See :class:~qarbaal.patterns.PatternSpec.

• validators (Sequence[Validator] | None, default: None ) –

  Callables that receive an :class:~qarbaal.types.Indicator and return False to mark it invalid.

• chunk_overlap (int, default: 8192 ) –

  Characters kept between streaming windows. Keep this larger than the longest indicator you care about. The default is 8192, twice the 4096-character token cap.

• chunk_size (int, default: 1048576 ) –

  Read size used by :meth:extract_path for plain text.

Methods:

• __post_init__ –
• extract –

  Extract indicators from one string.

• extract_iter –

  Extract from a stream of text chunks.

• extract_many –

  Extract several named sources and correlate them.

• extract_path –

  Extract indicators from a file.

Attributes:

• extract_types (Sequence[str] | None) –
• normalize (bool) –
• validate (bool) –
• deduplicate (bool) –
• hide_private (bool) –
• drop_invalid (bool) –
• max_input_chars (int) –
• max_matches_per_type (int) –
• strict_limits (bool) –
• allow (Sequence[str] | None) –
• deny (Sequence[str] | None) –
• defang_patterns (Sequence[tuple[str, str]] | None) –
• custom_patterns (Sequence[PatternSpec] | None) –
• validators (Sequence[Validator] | None) –
• chunk_overlap (int) –
• chunk_size (int) –
• types (tuple[IOCType, ...]) –

  Indicator types this extractor will emit, in selection order.

extract_types class-attribute instance-attribute

extract_types: Sequence[str] | None = None

normalize class-attribute instance-attribute

normalize: bool = True

validate class-attribute instance-attribute

validate: bool = True

deduplicate class-attribute instance-attribute

deduplicate: bool = True

hide_private class-attribute instance-attribute

hide_private: bool = False

drop_invalid class-attribute instance-attribute

drop_invalid: bool = False

max_input_chars class-attribute instance-attribute

max_input_chars: int = 2000000

max_matches_per_type class-attribute instance-attribute

max_matches_per_type: int = 20000

strict_limits class-attribute instance-attribute

strict_limits: bool = False

allow class-attribute instance-attribute

allow: Sequence[str] | None = None

deny class-attribute instance-attribute

deny: Sequence[str] | None = None

defang_patterns class-attribute instance-attribute

defang_patterns: Sequence[tuple[str, str]] | None = None

custom_patterns class-attribute instance-attribute

custom_patterns: Sequence[PatternSpec] | None = None

validators class-attribute instance-attribute

validators: Sequence[Validator] | None = None

chunk_overlap class-attribute instance-attribute

chunk_overlap: int = 8192

chunk_size class-attribute instance-attribute

chunk_size: int = 1048576

types property

types: tuple[IOCType, ...]

Indicator types this extractor will emit, in selection order.

__post_init__

__post_init__() -> None

extract

extract(text: str, *, source: str | None = None, pages: Sequence[PageSpan] | None = None, truncated: bool = False) -> Extraction

Extract indicators from one string.

Parameters:

• text (str) –

  Source text.

• source (str | None, default: None ) –

  Sighting label. Defaults to <input>.

• pages (Sequence[PageSpan] | None, default: None ) –

  Optional (page, start, end) spans into text.

• truncated (bool, default: False ) –

  Set when a loader already cut the text to the size limit.

Returns:

• Extraction –

  Indicators, provenance, and statistics for this source.

Raises:

• InputTooLargeError –

  strict_limits is set and text is too long.

extract_iter

extract_iter(chunks: Iterable[str], *, source: str | None = None) -> Extraction

Extract from a stream of text chunks.

Only the overlap window is retained between chunks, so a large file does not have to sit in memory as one string. Offsets and line numbers are counted from the start of the stream. A match that straddles a chunk boundary is kept when it fits in chunk_overlap.

Parameters:

• chunks (Iterable[str]) –

  Pieces of one source, in order.

• source (str | None, default: None ) –

  Sighting label. Defaults to <stream>.

Returns:

• Extraction –

  One extraction for the whole stream.

Raises:

• InputTooLargeError –

  strict_limits is set and the stream is too long.

extract_many

extract_many(sources: Mapping[str, str] | Sequence[tuple[str, str]]) -> Extraction

Extract several named sources and correlate them.

Parameters:

• sources (Mapping[str, str] | Sequence[tuple[str, str]]) –

  A mapping or a sequence of (name, text) pairs.

Returns:

• Extraction –

  One extraction. Indicators seen in more than one source are returned

• by ( Extraction ) –

  meth:~qarbaal.types.Extraction.shared.

extract_path

extract_path(path: str | Path, *, parser: str | None = None) -> Extraction

Extract indicators from a file.

Plain text (.txt, .log, .md, and files without a suffix) is read in chunks. Other formats are parsed by :func:qarbaal.ingest.load_document up to max_input_chars.

Parameters:

• path (str | Path) –

  File to read.

• parser (str | None, default: None ) –

  Optional parser name that overrides the suffix.

Returns:

• Extraction –

  An extraction whose source label is the path.

resolve_types

resolve_types(requested: Sequence[str] | None, custom: Sequence[str] = ()) -> tuple[str, ...]

Resolve extract_types into an ordered list of type keys.

None selects :data:~qarbaal.types.DEFAULT_TYPES plus any custom pattern names. hash expands to MD5, SHA-1, SHA-256, and SHA-512. all also enables file paths, user agents, and mutexes.

Parameters:

• requested (Sequence[str] | None) –

  Names, aliases, or :class:~qarbaal.types.IOCType members.

• custom (Sequence[str], default: () ) –

  Custom pattern type names that are legal in an explicit list.

Returns:

• tuple[str, ...] –

  Type keys, de-duplicated, in the requested order.

Raises:

• ValueError –

  A name is not a known type or a custom pattern.

extract

extract(text: str, *, source: str | None = None, hide_private: bool = False) -> Extraction

Extract indicators with the default :class:IOCExtractor settings.

Normalization, validation, and deduplication are on. File paths, user agents, and mutexes are not extracted. Pass an :class:IOCExtractor when you need those types or different limits.

Parameters:

• text (str) –

  Report, log, or any other text. Defanged indicators are refanged.

• source (str | None, default: None ) –

  Label stored on each sighting. Defaults to <input>.

• hide_private (bool, default: False ) –

  Drop IPv4, IPv6, and CIDR values that carry a noise tag.

Returns:

• An ( Extraction ) –

  class:~qarbaal.types.Extraction.

merge

merge(extractions: Sequence[Extraction]) -> Extraction

Merge extractions that were produced with the same settings.

Indicators with the same fingerprint become one indicator. Sightings are concatenated and the first-seen value is kept. Input character counts are summed. Use :meth:IOCExtractor.extract_many when you still have the text.

Parameters:

• extractions (Sequence[Extraction]) –

  Results to combine, in source order.

Returns:

• Extraction –

  One extraction. An empty input returns an empty extraction.