Skip to content

Ingest

Read files into text the sieve can scan.

Plain text, JSON, CSV, XML, EML, and MBOX use the standard library. HTML, PDF, DOCX, YAML, and PCAP are optional extras. A missing extra raises :class:~qarbaal.exceptions.OptionalDependencyError with the install command. The core package does not import those libraries.

Classes:

  • LoadedDocument –

    Text produced from a file, plus page spans when the format has them.

Functions:

Attributes:

logger module-attribute 

logger = logging.getLogger('qarbaal.ingest')

PageSpan module-attribute 

PageSpan = tuple[int, int, int]

LoadedDocument dataclass 

LoadedDocument(source: str, text: str, format: str, pages: tuple[PageSpan, ...] = (), truncated: bool = False)

Text produced from a file, plus page spans when the format has them.

Attributes:

  • source (str) –

    Path that was read.

  • text (str) –

    Text the sieve should scan. For email this is the decoded body, not the raw bytes. For JSON, CSV, XML, and YAML it is the file text, so line numbers match the file.

  • format (str) –

    Parser name, such as json or pdf.

  • pages (tuple[PageSpan, ...]) –

    (page, start, end) spans into text. Empty when the format has no pages.

  • truncated (bool) –

    Whether the reader stopped at max_chars.

source instance-attribute 

source: str

text instance-attribute 

text: str

format instance-attribute 

format: str

pages class-attribute instance-attribute 

pages: tuple[PageSpan, ...] = ()

truncated class-attribute instance-attribute 

truncated: bool = False

supported_formats 

supported_formats() -> tuple[str, ...]

Return the suffixes :func:load_document accepts, in a stable order.

load_document 

load_document(path: str | Path, *, parser: str | None = None, max_chars: int = 2000000, pcap_max_packets: int = 20000) -> LoadedDocument

Load a file into text.

Parameters:

  • path (str | Path) –

    File to read.

  • parser (str | None, default: None ) –

    Parser name (json, eml, pdf, ...) instead of the suffix.

  • max_chars (int, default: 2000000 ) –

    Maximum characters returned. Longer input is truncated.

  • pcap_max_packets (int, default: 20000 ) –

    Packet cap for the PCAP parser.

Returns:

Raises: