Compute
Access modes
Available access modes vary by whether Unity Catalog is enabled or not
- With Unity Catalog enabled, standard and dedicated modes are recommended.
- standard access mode
- Libraries and init scripts must be added to the allowlist to use them
- allowlist cannot be disabled
- Instance profiles are unavailable, specifically IMDS is unavailable
- Service credentials can be used as an alternative of instance profiles.
- dedicated access mode
-
Standard compute runs user code in full isolation with no access to lower-level resources.
- See Lakeguard for the details
- standard access mode
- The legacy access mode no isolation shared is also available
- Unity Catalog is disabled on the compute
Unity Catalog
- Unity Catalog is account-level resource
- The default catalog varies by whether Unity Catalog is enabled or not (doc)
- Unity Catalog enabled -> the workspace catalog
- Unity Catalog disabled ->
hive_metastore
Credentials
- Use a role registered in service credentials for aws client
boto3_session = boto3.Session( botocore_session=dbutils.credentials.getServiceCredentialsProvider(credential_name) ... )
Links
Databricks Utilities (dbutils)
dbutils provides utility commands for notebooks.