azure-storage-file-datalake-py

Microsoft ｜ 雲端與基礎設施 ｜ 原文 ｜ MIT

用 Python 操作 Data Lake Gen2 的階層式檔案系統、目錄與 ACL

---
name: azure-storage-file-datalake-py
description: |
  Python 版的 Azure Data Lake Storage Gen2 SDK。用於階層式檔案系統、大數據分析，以及檔案／目錄操作。
  Triggers: "data lake", "DataLakeServiceClient", "FileSystemClient", "ADLS Gen2", "hierarchical namespace".
license: MIT
metadata:
  author: Microsoft
  version: "1.0.0"
  package: azure-storage-file-datalake
---

# Python 版 Azure Data Lake Storage Gen2 SDK

給大數據分析工作負載用的階層式檔案系統。

## 安裝

```bash
pip install azure-storage-file-datalake azure-identity
```

## 環境變數

```bash
AZURE_STORAGE_ACCOUNT_URL=https://<account>.dfs.core.windows.net  # 所有驗證方式都必填
AZURE_TOKEN_CREDENTIALS=prod # 只有在 production 使用 DefaultAzureCredential 時才需要
```

## 驗證與生命週期

> **🔑 下面每個程式碼範例都適用兩條規則：**
>
> 1. **優先用 `DefaultAzureCredential`。** 它在本機（Azure CLI / VS Code / Developer CLI）和 Azure 上（managed identity、workload identity）都能用，不用改程式碼。避免用 connection string、account key 或 API key，它們會繞過 Entra 的稽核和輪替。
>    - 本機開發：直接用 `DefaultAzureCredential` 就好。
>    - Production：設定 `AZURE_TOKEN_CREDENTIALS=prod`（或 `AZURE_TOKEN_CREDENTIALS=<specific_credential>`），把 credential chain 限縮在 production 安全的 credential。
> 2. **每個 client 都用 context manager 包起來**，這樣 HTTP transport、socket 和 token cache 才會確實釋放：
>    - 同步：`with <Client>(...) as client:`
>    - 非同步：`async with <Client>(...) as client:` **而且** `async with DefaultAzureCredential() as credential:`（來自 `azure.identity.aio`）
>
> 範例片段可能會簡化這段設定，但 production 的程式碼一定要兩條規則都遵守。

```python
from azure.identity import DefaultAzureCredential, ManagedIdentityCredential
from azure.storage.filedatalake import DataLakeServiceClient

# 本機開發：DefaultAzureCredential。Production：設定 AZURE_TOKEN_CREDENTIALS=prod 或 AZURE_TOKEN_CREDENTIALS=<specific_credential>
credential = DefaultAzureCredential(require_envvar=True)
# 或在 production 直接用特定的 credential：
# See https://learn.microsoft.com/python/api/overview/azure/identity-readme?view=azure-python#credential-classes
# credential = ManagedIdentityCredential()
account_url = "https://<account>.dfs.core.windows.net"

with DataLakeServiceClient(account_url=account_url, credential=credential) as service_client:
    # 在這裡使用 service_client（各種操作請看後面的章節）
    ...
```

## Client 階層

| Client | 用途 |
|--------|---------|
| `DataLakeServiceClient` | Account 層級的操作 |
| `FileSystemClient` | Container（檔案系統）操作 |
| `DataLakeDirectoryClient` | 目錄操作 |
| `DataLakeFileClient` | 檔案操作 |

## 檔案系統操作

```python
# 建立檔案系統（container）
file_system_client = service_client.create_file_system("myfilesystem")

# 取得既有的
file_system_client = service_client.get_file_system_client("myfilesystem")

# 刪除
service_client.delete_file_system("myfilesystem")

# 列出檔案系統
for fs in service_client.list_file_systems():
    print(fs.name)
```

## 目錄操作

```python
file_system_client = service_client.get_file_system_client("myfilesystem")

# 建立目錄
directory_client = file_system_client.create_directory("mydir")

# 建立巢狀目錄
directory_client = file_system_client.create_directory("path/to/nested/dir")

# 取得目錄 client
directory_client = file_system_client.get_directory_client("mydir")

# 刪除目錄
directory_client.delete_directory()

# 重新命名／移動目錄
directory_client.rename_directory(new_name="myfilesystem/newname")
```

## 檔案操作

### 上傳檔案

```python
# 取得檔案 client
file_client = file_system_client.get_file_client("path/to/file.txt")

# 從本機檔案上傳
with open("local-file.txt", "rb") as data:
    file_client.upload_data(data, overwrite=True)

# 上傳 bytes
file_client.upload_data(b"Hello, Data Lake!", overwrite=True)

# 附加資料（用於大檔案）
file_client.append_data(data=b"chunk1", offset=0, length=6)
file_client.append_data(data=b"chunk2", offset=6, length=6)
file_client.flush_data(12)  # 提交資料
```

### 下載檔案

```python
file_client = file_system_client.get_file_client("path/to/file.txt")

# 下載全部內容
download = file_client.download_file()
content = download.readall()

# 下載到檔案
with open("downloaded.txt", "wb") as f:
    download = file_client.download_file()
    download.readinto(f)

# 下載指定範圍
download = file_client.download_file(offset=0, length=100)
```

### 刪除檔案

```python
file_client.delete_file()
```

## 列出內容

```python
# 列出路徑（檔案和目錄）
for path in file_system_client.get_paths():
    print(f"{'DIR' if path.is_directory else 'FILE'}: {path.name}")

# 列出目錄裡的路徑
for path in file_system_client.get_paths(path="mydir"):
    print(path.name)

# 遞迴列出
for path in file_system_client.get_paths(path="mydir", recursive=True):
    print(path.name)
```

## 檔案／目錄屬性

```python
# 取得屬性
properties = file_client.get_file_properties()
print(f"Size: {properties.size}")
print(f"Last modified: {properties.last_modified}")

# 設定 metadata
file_client.set_metadata(metadata={"processed": "true"})
```

## 存取控制（ACL）

```python
# 取得 ACL
acl = directory_client.get_access_control()
print(f"Owner: {acl['owner']}")
print(f"Permissions: {acl['permissions']}")

# 設定 ACL
directory_client.set_access_control(
    owner="user-id",
    permissions="rwxr-x---"
)

# 更新 ACL 項目
from azure.storage.filedatalake import AccessControlChangeResult
directory_client.update_access_control_recursive(
    acl="user:user-id:rwx"
)
```

## 非同步 Client

```python
from azure.storage.filedatalake.aio import DataLakeServiceClient
from azure.identity.aio import DefaultAzureCredential

async def datalake_operations():
    async with DefaultAzureCredential() as credential:
        async with DataLakeServiceClient(
            account_url="https://<account>.dfs.core.windows.net",
            credential=credential
        ) as service_client:
            file_system_client = service_client.get_file_system_client("myfilesystem")
            file_client = file_system_client.get_file_client("test.txt")
            
            await file_client.upload_data(b"async content", overwrite=True)
            
            download = await file_client.download_file()
            content = await download.readall()

import asyncio
asyncio.run(datalake_operations())
```

## 最佳實務

1. **同步或非同步選一個，然後保持一致。** 同一條呼叫路徑裡，不要混用 `azure.storage.filedatalake` 同步 client 和 `azure.storage.filedatalake.aio` 非同步 client。每個 module 只選一種模式。
2. **client 和非同步 credential 一律用 context manager。** 每個 client 都用 `with DataLakeServiceClient(...) as client:`（同步）或 `async with DataLakeServiceClient(...) as client:`（非同步）包起來。非同步的 `DefaultAzureCredential`（來自 `azure.identity.aio`）也要用 `async with credential:`，token 和 transport 才會清乾淨。
3. **用 `DefaultAzureCredential`**，在本機開發和 Azure 之間都能用同一套驗證（盡量避免 connection string／API key）。
4. 要有檔案系統語意，就**用 hierarchical namespace**
5. 大檔案上傳**用 `append_data` + `flush_data`**
6. **在目錄層級設定 ACL**，讓子項目繼承
7. 高吞吐量的情境**用非同步 client**
8. 要完整列出目錄，**用 `get_paths` 搭配 `recursive=True`**
9. 自訂檔案屬性要**設定 metadata**
10. 簡單的物件儲存情境，**考慮改用 Blob API**

## 參考檔案

| 檔案 | 內容 |
|------|----------|
| [references/capabilities.md](references/capabilities.md) | 其他非主要的能力、operation group 涵蓋範圍，以及 production 檢查清單。 |
| [references/non-hero-scenarios.md](references/non-hero-scenarios.md) | 次要／進階情境的專屬非主要流程範例。 |

---
name: azure-storage-file-datalake-py
description: |
  Azure Data Lake Storage Gen2 SDK for Python. Use for hierarchical file systems, big data analytics, and file/directory operations.
  Triggers: "data lake", "DataLakeServiceClient", "FileSystemClient", "ADLS Gen2", "hierarchical namespace".
license: MIT
metadata:
  author: Microsoft
  version: "1.0.0"
  package: azure-storage-file-datalake
---

# Azure Data Lake Storage Gen2 SDK for Python

Hierarchical file system for big data analytics workloads.

## Installation

```bash
pip install azure-storage-file-datalake azure-identity
```

## Environment Variables

```bash
AZURE_STORAGE_ACCOUNT_URL=https://<account>.dfs.core.windows.net  # Required for all auth methods
AZURE_TOKEN_CREDENTIALS=prod # Required only if DefaultAzureCredential is used in production
```

## Authentication & Lifecycle

> **🔑 Two rules apply to every code sample below:**
>
> 1. **Prefer `DefaultAzureCredential`.** It works locally (Azure CLI / VS Code / Developer CLI) and in Azure (managed identity, workload identity) with no code change. Avoid connection strings, account/API keys — they bypass Entra audit and rotation.
>    - Local dev: `DefaultAzureCredential` works as-is.
>    - Production: set `AZURE_TOKEN_CREDENTIALS=prod` (or `AZURE_TOKEN_CREDENTIALS=<specific_credential>`) to constrain the credential chain to production-safe credentials.
> 2. **Wrap every client in a context manager** so HTTP transports, sockets, and token caches are released deterministically:
>    - Sync: `with <Client>(...) as client:`
>    - Async: `async with <Client>(...) as client:` **and** `async with DefaultAzureCredential() as credential:` (from `azure.identity.aio`)
>
> Snippets may abbreviate this setup, but production code should always follow both rules.

```python
from azure.identity import DefaultAzureCredential, ManagedIdentityCredential
from azure.storage.filedatalake import DataLakeServiceClient

# Local dev: DefaultAzureCredential. Production: set AZURE_TOKEN_CREDENTIALS=prod or AZURE_TOKEN_CREDENTIALS=<specific_credential>
credential = DefaultAzureCredential(require_envvar=True)
# Or use a specific credential directly in production:
# See https://learn.microsoft.com/python/api/overview/azure/identity-readme?view=azure-python#credential-classes
# credential = ManagedIdentityCredential()
account_url = "https://<account>.dfs.core.windows.net"

with DataLakeServiceClient(account_url=account_url, credential=credential) as service_client:
    # Use service_client here (see following sections for operations)
    ...
```

## Client Hierarchy

| Client | Purpose |
|--------|---------|
| `DataLakeServiceClient` | Account-level operations |
| `FileSystemClient` | Container (file system) operations |
| `DataLakeDirectoryClient` | Directory operations |
| `DataLakeFileClient` | File operations |

## File System Operations

```python
# Create file system (container)
file_system_client = service_client.create_file_system("myfilesystem")

# Get existing
file_system_client = service_client.get_file_system_client("myfilesystem")

# Delete
service_client.delete_file_system("myfilesystem")

# List file systems
for fs in service_client.list_file_systems():
    print(fs.name)
```

## Directory Operations

```python
file_system_client = service_client.get_file_system_client("myfilesystem")

# Create directory
directory_client = file_system_client.create_directory("mydir")

# Create nested directories
directory_client = file_system_client.create_directory("path/to/nested/dir")

# Get directory client
directory_client = file_system_client.get_directory_client("mydir")

# Delete directory
directory_client.delete_directory()

# Rename/move directory
directory_client.rename_directory(new_name="myfilesystem/newname")
```

## File Operations

### Upload File

```python
# Get file client
file_client = file_system_client.get_file_client("path/to/file.txt")

# Upload from local file
with open("local-file.txt", "rb") as data:
    file_client.upload_data(data, overwrite=True)

# Upload bytes
file_client.upload_data(b"Hello, Data Lake!", overwrite=True)

# Append data (for large files)
file_client.append_data(data=b"chunk1", offset=0, length=6)
file_client.append_data(data=b"chunk2", offset=6, length=6)
file_client.flush_data(12)  # Commit the data
```

### Download File

```python
file_client = file_system_client.get_file_client("path/to/file.txt")

# Download all content
download = file_client.download_file()
content = download.readall()

# Download to file
with open("downloaded.txt", "wb") as f:
    download = file_client.download_file()
    download.readinto(f)

# Download range
download = file_client.download_file(offset=0, length=100)
```

### Delete File

```python
file_client.delete_file()
```

## List Contents

```python
# List paths (files and directories)
for path in file_system_client.get_paths():
    print(f"{'DIR' if path.is_directory else 'FILE'}: {path.name}")

# List paths in directory
for path in file_system_client.get_paths(path="mydir"):
    print(path.name)

# Recursive listing
for path in file_system_client.get_paths(path="mydir", recursive=True):
    print(path.name)
```

## File/Directory Properties

```python
# Get properties
properties = file_client.get_file_properties()
print(f"Size: {properties.size}")
print(f"Last modified: {properties.last_modified}")

# Set metadata
file_client.set_metadata(metadata={"processed": "true"})
```

## Access Control (ACL)

```python
# Get ACL
acl = directory_client.get_access_control()
print(f"Owner: {acl['owner']}")
print(f"Permissions: {acl['permissions']}")

# Set ACL
directory_client.set_access_control(
    owner="user-id",
    permissions="rwxr-x---"
)

# Update ACL entries
from azure.storage.filedatalake import AccessControlChangeResult
directory_client.update_access_control_recursive(
    acl="user:user-id:rwx"
)
```

## Async Client

```python
from azure.storage.filedatalake.aio import DataLakeServiceClient
from azure.identity.aio import DefaultAzureCredential

async def datalake_operations():
    async with DefaultAzureCredential() as credential:
        async with DataLakeServiceClient(
            account_url="https://<account>.dfs.core.windows.net",
            credential=credential
        ) as service_client:
            file_system_client = service_client.get_file_system_client("myfilesystem")
            file_client = file_system_client.get_file_client("test.txt")
            
            await file_client.upload_data(b"async content", overwrite=True)
            
            download = await file_client.download_file()
            content = await download.readall()

import asyncio
asyncio.run(datalake_operations())
```

## Best Practices

1. **Pick sync OR async and stay consistent.** Do not mix `azure.storage.filedatalake` sync clients with `azure.storage.filedatalake.aio` async clients in the same call path. Choose one mode per module.
2. **Always use context managers for clients and async credentials.** Wrap every client in `with DataLakeServiceClient(...) as client:` (sync) or `async with DataLakeServiceClient(...) as client:` (async). For async `DefaultAzureCredential` from `azure.identity.aio`, also use `async with credential:` so tokens and transports are cleaned up.
3. **Use `DefaultAzureCredential`** for portable auth across local dev and Azure (avoid connection strings / API keys when possible).
4. **Use hierarchical namespace** for file system semantics
5. **Use `append_data` + `flush_data`** for large file uploads
6. **Set ACLs at directory level** and inherit to children
7. **Use async client** for high-throughput scenarios
8. **Use `get_paths` with `recursive=True`** for full directory listing
9. **Set metadata** for custom file attributes
10. **Consider Blob API** for simple object storage use cases

## Reference Files

| File | Contents |
|------|----------|
| [references/capabilities.md](references/capabilities.md) | Additional non-hero capabilities, operation-group coverage, and production checklists. |
| [references/non-hero-scenarios.md](references/non-hero-scenarios.md) | Dedicated non-hero examples for secondary/advanced scenarios. |