FileSystem

Read from or write to object stores and filesystems. Supports S3, GCS, Azure Blob, and local filesystem.

Quick Example

apiVersion: laminar.io/v1
kind: Table
spec:
  name: log_files
  connector: filesystem
  config:
    tableType:
      path: s3://my-bucket/logs/
      compressionFormat: gzip
      regexPattern: ".*\\.json\\.gz$"
      storageOptions:
        s3.region: us-east-1
  schema:
    format:
      json: {}
    fields:
      - field_name: timestamp
        field_type:
          type:
            primitive: DateTime
        nullable: false
      - field_name: message
        field_type:
          type:
            primitive: Utf8
        nullable: false

Configuration

Required

Property	Type	Description
`tableType.path`	string	URI of folder to read from (s3://, gs://, az://, or local)
`tableType.compressionFormat`	string	`none`, `zstd`, or `gzip`

Optional

Property	Type	Description
`tableType.regexPattern`	string	Regex pattern to filter files
`tableType.storageOptions`	object	Cloud storage credentials

JSON Schema Reference

Connection Table Schema

{
  "type": "object",
  "properties": {
    "tableType": {
      "oneOf": [
        {
          "title": "Source",
          "type": "object",
          "properties": {
            "path": {"type": "string"},
            "compressionFormat": {"enum": ["none", "zstd", "gzip"]},
            "regexPattern": {"type": "string"},
            "storageOptions": {"type": "object"}
          },
          "required": ["path", "compressionFormat"]
        },
        {
          "title": "Sink",
          "type": "object",
          "properties": {
            "path": {"type": "string"},
            "storageOptions": {"type": "object"},
            "rollingPolicy": {
              "type": "object",
              "properties": {
                "fileSizeBytes": {"type": "integer"},
                "intervalSeconds": {"type": "integer"},
                "inactivitySeconds": {"type": "integer"}
              }
            },
            "fileNaming": {
              "type": "object",
              "properties": {
                "prefix": {"type": "string"},
                "suffix": {"type": "string"},
                "strategy": {"enum": ["serial", "uuid", "uuidV7", "ulid"]}
              }
            }
          },
          "required": ["path"]
        }
      ]
    }
  },
  "required": ["tableType"]
}