Skip to main content
Version: Next

Partition Evolution

Partition evolution lets you change a table's partition strategy without rewriting existing data files. Old files keep their original partition spec, and new files use the new spec.

Why It Matters​

Partitioning decisions that work on day one often become wrong later:

  • A table starts small and later needs finer time partitions.
  • A high-cardinality column creates too many small partitions.
  • Query patterns shift from daily reports to customer-level lookups.
  • A partition field turns out to have skewed values.

Without partition evolution, teams often migrate to a new table. Iceberg avoids that by storing partition specs in metadata.

How It Works​

Each data file is written with a partition spec ID. When the table's spec changes, Iceberg keeps old and new specs side by side.

At read time, Iceberg plans files using the spec that applies to each file. This is why old data can be partitioned by month while new data is partitioned by day.

Add a Partition Field​

ALTER TABLE prod.db.events
ADD PARTITION FIELD days(event_time);

Drop a Partition Field​

ALTER TABLE prod.db.events
DROP PARTITION FIELD level;

Dropping a partition field does not rewrite old files. It only changes how new data is written.

Replace a Partition Field​

ALTER TABLE prod.db.events
REPLACE PARTITION FIELD event_month WITH days(event_time) AS event_day;

Use replacement when the old and new fields represent a controlled strategy change.

Operational Notes​

  • Partition evolution affects new writes immediately.
  • Old files remain in the old layout until compaction or rewrite jobs change them.
  • Query planning must consider multiple specs, so keep the number of specs reasonable.
  • Partition changes should be paired with monitoring for file counts, scan size, and query latency.

Example Migration​

  1. Start with months(event_time) for a low-volume table.
  2. Data volume grows and daily reports become common.
  3. Replace the field with days(event_time).
  4. New writes use daily partitions.
  5. Rewrite old high-value partitions only if query performance requires it.

Key Takeaway​

Partition evolution makes partition strategy a metadata decision instead of a table migration. It is powerful because it lets layout follow real query patterns over time.