Enable LargeListArray support in Parquet reader schema validation

[callmepandey]

Summary

Follow-up to #502. The data conversion layer now supports LargeListArray (64-bit offsets) via ProjectRecordBatch, but the Parquet reader's schema validation still rejects LARGE_LIST types. Additionally, the reader needs to expose Arrow's list_type property to allow users to request LargeListArray output.

Problem

1. ValidateParquetSchemaEvolution in parquet_schema_util.cc:177-180 only accepts ::arrow::Type::LIST:

case TypeId::kList:
  if (arrow_type->id() == ::arrow::Type::LIST) {
    return {};
  }
  break;

2. Arrow's Parquet reader defaults to Type::LIST output. Without exposing ArrowReaderProperties::set_list_type(), users cannot request LargeListArray output.

Proposed Solution

1. Update schema validation to accept both list types

case TypeId::kList:
  if (arrow_type->id() == ::arrow::Type::LIST ||
      arrow_type->id() == ::arrow::Type::LARGE_LIST) {
    return {};
  }
  break;

2. Add kListType to ReaderProperties

Expose a property to configure the Arrow list type preference.

3. Pass through to Arrow reader

In ParquetReader::Impl::Open(), call arrow_reader_properties.set_list_type() with the configured value.

Why This Is Safe

1. Iceberg's ListType doesn't distinguish between LIST and LARGE_LIST
2. The projection layer (ProjectRecordBatch) already handles both via templated ProjectListArrayImpl<>
3. Both represent the same logical "list" concept, just with different offset sizes

Files to Change

• src/iceberg/parquet/parquet_schema_util.cc - Update ValidateParquetSchemaEvolution
• src/iceberg/parquet/parquet_reader.cc - Pass list_type to ArrowReaderProperties
• src/iceberg/reader.h - Add kListType to ReaderProperties
• src/iceberg/test/parquet_test.cc - Add integration tests

Related

• Closes the remaining work from Add support for Arrow LargeListArray in Parquet data projection #502

[callmepandey]

@wgtmac Does this approach look good? If so, I can start implementing it.

[wgtmac]

Is this enough to read large list array from the parquet reader? I think we need to add a reader property to pass it through the reader.

[callmepandey]

Good catch! I focused on projection/validation but missed that Arrow defaults to Type::LIST output regardless of data. We need to expose ArrowReaderProperties::set_list_type() through our ReaderProperties.

Updated scope for this issue:

1. Update ValidateParquetSchemaEvolution to accept LARGE_LIST (done in current PR)
2. Add kListType to ReaderProperties
3. Pass through to arrow_reader_properties.set_list_type() in ParquetReader::Impl::Open()

Let me know if this looks right and I'll update the PR.

[wgtmac]

You might want to follow https://github.com/apache/iceberg-cpp/blob/main/src/iceberg/file_reader.h#L71 to add a read.arrow.use-large-list key with false as the default value.

Then, set ArrowReaderProperties::set_list_type(...) for Parquet reader based on the that property: https://github.com/apache/arrow/blob/25b7247f9713ad837636019124466b4e752d3409/cpp/src/parquet/properties.h#L1183.

For completeness, perhaps we should also support Avro reader as well (can be a separate change of course).