Apache Iceberg

Apache Iceberg connector for writing streaming data to Iceberg tables. Supports REST and AWS Glue catalogs.

Quick Example

apiVersion: laminar.io/v1
kind: Profile
spec:
  name: iceberg-rest
  connector: iceberg
  config:
    catalog:
      type: rest
      url: http://localhost:8181
      warehouse: my-warehouse
 
---
apiVersion: laminar.io/v1
kind: Table
spec:
  name: events_iceberg
  connector: iceberg
  connection_profile_id: iceberg-rest
  config:
    type: sink
    sink_table_config:
      namespace: analytics
      table_name: events
      rolling_policy:
        file_size_bytes: 134217728
        interval_seconds: 300
      partitioning:
        fields:
          - name: event_date
            transform: identity
        shuffle_by_partition:
          enabled: true
  schema:
    format:
      parquet: {}
    fields:
      - field_name: event_id
        field_type:
          type:
            primitive: Utf8
        nullable: false
      - field_name: event_date
        field_type:
          type:
            primitive: Date32
        nullable: false

Configuration

REST Catalog

Property	Type	Required	Description
`catalog.type`	string	Yes	`rest`
`catalog.url`	string	Yes	Base URL for the REST catalog
`catalog.warehouse`	string	No	Warehouse name
`catalog.token`	string	No	Authentication token

AWS Glue Catalog

Property	Type	Required	Description
`catalog.type`	string	Yes	`glue`
`catalog.glue_catalog_config.region`	string	Yes	AWS region
`catalog.glue_catalog_config.warehouse`	string	No	S3 warehouse path
`catalog.glue_catalog_config.access_key_id`	string	No	AWS access key (uses env if not set)
`catalog.glue_catalog_config.secret_access_key`	string	No	AWS secret key

JSON Schema Reference

Connection Profile Schema

{
  "type": "object",
  "properties": {
    "catalog": {
      "oneOf": [
        {
          "type": "object",
          "properties": {
            "type": {"const": "rest"},
            "url": {"type": "string", "format": "uri"},
            "warehouse": {"type": "string"},
            "token": {"type": "string"}
          },
          "required": ["type", "url"]
        },
        {
          "type": "object",
          "properties": {
            "type": {"const": "glue"},
            "glue_catalog_config": {
              "type": "object",
              "properties": {
                "region": {"type": "string"},
                "warehouse": {"type": "string"},
                "access_key_id": {"type": "string"},
                "secret_access_key": {"type": "string"}
              },
              "required": ["region"]
            }
          },
          "required": ["type", "glue_catalog_config"]
        }
      ]
    }
  },
  "required": ["catalog"]
}

Connection Table Schema

{
  "type": "object",
  "properties": {
    "type": {"const": "sink"},
    "sink_table_config": {
      "type": "object",
      "properties": {
        "table_name": {"type": "string"},
        "namespace": {"type": "string"},
        "location_path": {"type": "string"},
        "rolling_policy": {
          "type": "object",
          "properties": {
            "file_size_bytes": {"type": "integer"},
            "interval_seconds": {"type": "integer"},
            "inactivity_seconds": {"type": "integer"}
          }
        },
        "partitioning": {
          "type": "object",
          "properties": {
            "fields": {
              "type": "array",
              "items": {
                "type": "object",
                "properties": {
                  "name": {"type": "string"},
                  "transform": {"enum": ["identity", "hour", "year", "month"]}
                },
                "required": ["name"]
              }
            },
            "shuffle_by_partition": {
              "type": "object",
              "properties": {
                "enabled": {"type": "boolean"}
              }
            }
          }
        }
      },
      "required": ["table_name"]
    }
  },
  "required": ["type", "sink_table_config"]
}