Skip to main content
Version: 0.0.5 (stable)

agrag.loaders.types.ReadOptions

ReadOptions(encoding:str | None = None, max_document_bytes:int = 32 * 1024 * 1024, store_text:bool = True, store_raw_record:bool = False, text_column:str | None = None, id_column:str | None = None, title_column:str | None = None, published_at_column:str | None = None, content_date_column:str | None = None, json_mode:JsonMode = JsonMode.AUTO, csv_mode:CsvMode = CsvMode.ROWS, csv_delimiter:str | None = None, normalization:Normalization = Normalization()) -> None

Per-source reader configuration.

Frozen so it is safe to share across worker processes.

Attributes:

  • encoding (str | None) – The text encoding to use. None lets the decoder detect it.
  • max_document_bytes (int) – The largest source, prose or record, the loader will read.
  • store_text (bool) – When false, the document text is an empty string.
  • store_raw_record (bool) – When true, a record document keeps its raw row data.
  • text_column (str | None) – The column that holds document text. None uses the first of text, body, content and description that exists, or the last column.
  • id_column (str | None) – The column whose value becomes the document id.
  • title_column (str | None) – The column whose value becomes the document title.
  • published_at_column (str | None) – The column whose ISO date or date-time becomes Document.published_at. Blank values mean no date.
  • content_date_column (str | None) – The column whose ISO date or date-time becomes Document.content_date. Blank values mean no date.
  • json_mode (JsonMode) – The JSON reading mode.
  • csv_mode (CsvMode) – The CSV reading mode.
  • csv_delimiter (str | None) – The column separator. None infers it from the extension.
  • normalization (Normalization) – How to normalize decoded text: byte-order mark, newline form and Unicode form. The default removes the mark, uses LF and applies NFKC. Chunk offsets index the normalized text.

content_date_column​

content_date_column: str | None = None

csv_delimiter​

csv_delimiter: str | None = None

csv_mode​

csv_mode: CsvMode = CsvMode.ROWS

encoding​

encoding: str | None = None

id_column​

id_column: str | None = None

json_mode​

json_mode: JsonMode = JsonMode.AUTO

max_document_bytes​

max_document_bytes: int = 32 * 1024 * 1024

normalization​

normalization: Normalization = field(default_factory=Normalization)

published_at_column​

published_at_column: str | None = None

store_raw_record​

store_raw_record: bool = False

store_text​

store_text: bool = True

text_column​

text_column: str | None = None

title_column​

title_column: str | None = None