Access Iceberg tables managed in AWS Glue Iceberg REST catalog and S3 Tables
Memo to access Iceberg tables managed in AWS Glue Iceberg REST catalog and S3 Tables.
I used DuckDB 1.4.3.
Setup
- Install and load extensions
INSTALL aws; INSTALL httpfs; INSTALL iceberg; LOAD iceberg; - Create a secret that is used to access the Iceberg REST catalog and tables
CREATE SECRET ( TYPE s3, PROVIDER credential_chain, CHAIN sts, ASSUME_ROLE_ARN 'arn:aws:iam::account_id:role/role', REGION 'us-east-1' ); - Connect to the catalog
ATTACH 'account_id' AS glue_catalog ( TYPE iceberg, ENDPOINT 'glue.REGION.amazonaws.com/iceberg', AUTHORIZATION_TYPE 'sigv4' ); -- for federated catalogs of s3 table buckets ATTACH 'account_id:s3tablescatalog/table_bucket_name' AS glue_catalog ( TYPE iceberg, ENDPOINT_TYPE 'glue' );
Note: To create tables in Glue Iceberg REST catalog, you need to specify ENDPOINT_TYPE instead of ENDPOINT or
disable support_stage_create explicitly.
support_stage_create is not supported in Glue Iceberg REST catalog and it is enabled by default when ENDPOINT is specified.
Check
- Confirm that the catalog is attached
D show databases; | database_name | |---------------| | glue_catalog | | memory |
The document suggests to use show all tables to see attached tables, but it didn’t return attached tables.
Query
To access attached Iceberg tables, you need to specify a table by catalog.namespace.table format.
select count(*) from glue_catalog.tpch.nation;
- create schema (database in REST catalog) and table
create schema glue_catalog.test_db; create table glue_catalog.test_db.test_tbl as select n from generate_series(1, 100) s(n);
D select count(*) from glue_catalog.test_db.test_tbl;
+--------------+
| count_star() |
+--------------+
| 100 |
+--------------+