通过托管标识通过 Synapse 访问 Azure 存储帐户
Posted
技术标签:
【中文标题】通过托管标识通过 Synapse 访问 Azure 存储帐户【英文标题】:Access Azure Storage account via Synapse through Managed Identity 【发布时间】:2021-04-07 17:50:05 【问题描述】:我正在尝试通过托管标识通过 Azure 突触连接到 Azure Blob 存储,具体步骤如下:
为服务器分配一个身份
作为贡献者授予对 Blob 存储上服务器的访问权限
执行了以下查询
创建主密钥
CREATE DATABASE SCOPED CREDENTIAL MSI WITH IDENTITY = 'Managed Service Identity';
CREATE EXTERNAL DATA SOURCE [BlobStorage] WITH
( TYPE = hadoop,
位置 = 'abfss://@.dfs.core.windows.net', 凭证 = MSI )
创建的外部文件格式
当我尝试创建外部表时,我收到以下错误:
External file access failed due to internal error: 'Error occurred while accessing HDFS: Java exception raised on call to HdfsBridge_IsDirExist. Java exception message: HdfsBridge::isDirExist - Unexpected error encountered checking whether directory exists or not: AbfsRestOperationException: Operation failed: "This endpoint does not support BlobStorageEvents or SoftDelete. Please disable these account features if you would like to use this endpoint.", 409, HEAD, https://<<>>.dfs.core.windows.net/<<>>//?upn=false&action=getAccessControl&timeout=90'
那么我错过了什么?
【问题讨论】:
【参考方案1】:早安
这是一个示例脚本,用于显示使用身份传递(托管或用户身份)连接到 Blob 存储,此示例将适用于 SQL Serverless 池,对于托管池,您需要添加其他参数来指定 hadoop .如果有问题,请告诉我。它们也是您最后可能需要的附加 Powershell 脚本。
--Create a master key, once per database
--CREATE MASTER KEY ENCRYPTION BY PASSWORD = 'RandomPassword!!££1132';
/*
DROP EXTERNAL TABLE dbo.Test_useridentity
DROP EXTERNAL DATA SOURCE blobstorage_via_useridentity
DROP EXTERNAL TABLE dbo.Test_managedidentity
DROP EXTERNAL DATA SOURCE blobstorage_via_managedidentity
DROP DATABASE SCOPED CREDENTIAL cred_via_managedidentity
DROP EXTERNAL FILE FORMAT textfile_csv_withheader
*/
--Create external file format for CSV
CREATE EXTERNAL FILE FORMAT textfile_csv_withheader WITH (
FORMAT_TYPE = DELIMITEDTEXT,
FORMAT_OPTIONS (
FIELD_TERMINATOR = ',',
STRING_DELIMITER = '\"',
FIRST_ROW = 2
)
);
--Create Credentials for accessing external data source using various methods
--This has been done at storage account level for this example
CREATE DATABASE SCOPED CREDENTIAL cred_via_managedidentity WITH IDENTITY =
'Managed Identity' GO
CREATE EXTERNAL DATA SOURCE blobstorage_via_managedidentity WITH (
CREDENTIAL = cred_via_managedidentity,
LOCATION = 'abfss://container@account.dfs.core.windows.net' )
--Dont specify the credential for user identity passthrough
CREATE EXTERNAL DATA SOURCE blobstorage_via_useridentity WITH (
LOCATION = 'abfss://container@account.dfs.core.windows.net' )
CREATE EXTERNAL TABLE dbo.Test_ManagedIdentity ( [col1] varchar(100),
[col2] varchar(100), [col3] varchar(100) ) WITH ( LOCATION =
'/test.csv',
DATA_SOURCE = blobstorage_via_managedidentity,
FILE_FORMAT = [textfile_csv_withheader] );
CREATE EXTERNAL TABLE dbo.Test_UserIdentity ( [col1] varchar(100),
[col2] varchar(100), [col3] varchar(100) ) WITH ( LOCATION =
'/test.csv',
DATA_SOURCE = blobstorage_via_useridentity,
FILE_FORMAT = [textfile_csv_withheader] );
--Added use 'synapseaccountname' to allow access via the managed identity as Storage Blob Data Reader
select * from Test_ManagedIdentity --This will work for Private Links, others will not
--Added my user to the storage 'xxxxxxx@microsoft.com' account as Storage Blob Data Reader, can take 5-10 mins to replicate
select * from Test_UserIdentity
根据您的安全设置来允许通过防火墙进行连接,您可能需要的 powershell 脚本是 --
$resourceGroupName = "xxxx-rg-name"
$accountName = "xxxxSynapseAccountNamexxxx"
$tenantId = "Guid for your Azure Tenant"
$resourceId1 = "/subscriptions/xxxxx-aaaa-sssss-guid/resourcegroups/xxxx-rg-name/providers/Microsoft.Synapse/workspaces/xxxxSynapseAccountNamexxxx"
Add-AzStorageAccountNetworkRule -ResourceGroupName $resourceGroupName -Name $accountName -TenantId $tenantId -ResourceId $resourceId1
$rule = Get-AzStorageAccountNetworkRuleSet -ResourceGroupName $resourceGroupName -Name $accountName
$rule.ResourceAccessRules
希望对你有帮助,如果有问题请告诉我
【讨论】:
以上是关于通过托管标识通过 Synapse 访问 Azure 存储帐户的主要内容,如果未能解决你的问题,请参考以下文章
通过 Synapse Pipelines 执行 Azure Synapse Notebook 时访问 Key Vault 时出错
无法通过 SSMS 访问 Azure Synapse Spark 表
使用托管标识与 Azure SQL 的 EF Core 连接
从 Node.js 函数访问 Azure Synapse Analytics