在Azure Stack上部署Kubernetes

Posted 2021-04-16 华来四Azure混合云

tags:

篇首语：本文由小常识网(cha138.com)小编为大家整理，主要介绍了在Azure Stack上部署Kubernetes相关的知识，希望对你有一定的参考价值。

提示

微软Azure Stack交流微信群
微软Azure Docker交流微信群

Azure Stack是我们实现DevOps的重要利器，而要实现DevOps，离不开容器编排平台Kubernetes的支持。本文将介绍如何在Azure Stack ASDK环境里部署Kubernetes，在AKS for Azure Stack Mooncake正式发布之前，可以让我们更方便地进行测试。

这次实验环境位于Azure Global，请参考《Azure上一键部署Azure Stack》，该环境拥有以下特点：

单节点ASDK环境，默认情况下，其region为Local，外部域名为azurestack.external
位于Azure Global，也就是说可以无限制地访问Internet，不受伟大的防火墙影响
Azure Stack用Azure中国的Azure AD账户进行部署，这和我们国内的生产环境完全一致。目前不支持ADFS模式部署。

不能用原版的acs-engine

大家知道，如果要在Azure中国里创建Kubernetes，可以借助acs-engine来实现。但是Azure Stack则没那么简单，原因是因为Azure Stack的ARM API端点和Azure公有云完全不同，对于生产环境的多节点Azure Stack来说，则每个企业部署，其ARM API端点都有变化，所以必须对acs-engine进行修改，把Azure Stack相关的Cloud Profile参数开放出来。

以下是一个Azure Stack部署Kubernets的JSON文档实例，可以看到这里可以指定与Azure Stack相关的Cloud Profile参数：

{

"apiVersion": "vlabs",

"properties": {

"orchestratorProfile": {

"orchestratorType": "Kubernetes",

"orchestratorRelease": "1.8",

   "kubernetesConfig": {

"CustomHyperkubeImage": "msazurestackdocker/k8s1.8:latest",

"networkPolicy": "none"

}

},

"cloudProfile": {

   "name": "AzureStackCloud",

"managementPortalURL": "",

"publishSettingsURL": "",

"serviceManagementEndpoint": "https://management.azurestackci14.partner.onmschina.cn/9290752b-dc4c-4d0b-baf3-3a053daf101b",

"resourceManagerEndpoint": "https://management.local.azurestack.external",

"activeDirectoryEndpoint": "https://login.chinacloudapi.cn/",

"galleryEndpoint": "https://portal.local.azurestack.external:30015/",

"keyVaultEndpoint": "",

"graphEndpoint": "https://graph.chinacloudapi.cn/",

"storageEndpointSuffix": "local.azurestack.external",

"sqlDatabaseDNSSuffix": "",

"trafficManagerDNSSuffix": "",

"keyVaultDNSSuffix": "vault.local.azurestack.external",

"serviceBusEndpointSuffix": "",

"serviceManagementVMDNSSuffix": "chinacloudapp.cn",

"resourceManagerVMDNSSuffix": "cloudapp.azurestack.external",

"containerRegistryDNSSuffix": "",

"resourceManagerRootCertificate": "*******ED26E1A51F5BB96E9356D6D610B74",

"location": "local"

   },

"masterProfile": {

"count": 1,

"dnsPrefix": "k8s-11145",

"vmSize": "Standard_D2_v2",

   "StorageProfile" : "StorageAccount"

},

"agentPoolProfiles": [

{

"name": "agentpool1",

"count": 1,

"vmSize": "Standard_D2_v2",

"availabilityProfile": "AvailabilitySet",

     "StorageProfile" : "StorageAccount"

}

],

"linuxProfile": {

"adminUsername": "azureuser",

"ssh": {

"publicKeys": [

{

"keyData": ""

}

]

}

},

"servicePrincipalProfile": {

"clientId": "",

"secret": ""

}

}

}

我们可以自己修改该json文档，使其符合我们的Azure Stack部署情况，然后用修改版的acs-engine生成ARM部署模板和参数模板。

https://github.com/radhikagupta5/AzureStack-QuickStart-Templates/tree/radhikgu-acs/101-acsengine-kubernetes-1803

准备工作

建议将Azure Stack注册到Azure公有云并激活，这样我们就可以使用应用市场项目，需要下载Ubuntu 16.04.201802220，以及Custom Script for Linux, 2.0.3。

在Azure Stack上部署Kubernetes

自动生成自定义的json文档

https://github.com/ahpeng/K8SOnAzureStack

然后执行以下命令，粗斜体部分的参数请用实际的值代替：

Import-module .ConnectAzureStack.Connect.psm1

Import-Module .AzureStack.AcsEngine.psm1 -Force

$namingSuffix = 10000..99999 | Get-Random

$masterDnsPrefix = "k8s-" + $namingSuffix

$tenantSubscriptionId = "*****-9923-4c0f-ad04-cd0045ff9d7f"

$tenantSubscriptionId的参数值参考以下的截图。

在Azure Stack上部署Kubernetes

$CloudAdminPass = ConvertTo-SecureString "Password" -AsPlainText -Force

$cloudAdminCredential = New-Object System.Management.Automation.PSCredential ("azurestackcloudadmin", $CloudAdminPass)

$serviceAdmin = "AdminName@TenantName.partner.onmschina.cn"

$AdminPass = ConvertTo-SecureString "Password" -AsPlainText -Force

$serviceAdminCredential = New-Object System.Management.Automation.PSCredential ($serviceAdmin, $AdminPass)

$TenantAdmin = "TenantUser@TenantName.partner.onmschina.cn"

$TenantAdminPass = ConvertTo-SecureString "Password" -AsPlainText -Force

$tenantAdminCredential = New-Object System.Management.Automation.PSCredential ($TenantAdmin, $TenantAdminPass)

$acsSshKey = "****"

$apiModelParameters = @{'ErcsComputerName' = "192.168.200.225"; 'CloudAdminCredential' = $cloudAdminCredential; 'ServiceAdminCredential' = $serviceAdminCredential; 'TenantAdminCredential' = $tenantAdminCredential; 'TenantSubscriptionId' = $tenantSubscriptionId; 'MasterDnsPrefix' = $masterDnsPrefix; 'LinuxVmSshKey' = $acsSshKey; 'NamingSuffix' = $namingSuffix;}

然后执行以下命令。

$apiModel = Prepare-AcseApiModel @apiModelParameters

命令执行结果如图所示。

VERBOSE: Retrieving stampinformation from ERCS: 192.168.200.225.

VERBOSE: Retrieving Root CAcertificated from: https://management.local.azurestack.external/metadata/endpoints?api-version=1.0

VERBOSE: Total elementes in thecert chain are 2

VERBOSE: Last element in certchain is issued by: CN=contoso-DC-CA, DC=contoso, DC=com

VERBOSE: Retrieved certificatethumbprint is: ******3E92C84C04200B18806D279A55606E4

VERBOSE: TenantId:c7917735-5d61-4832-8b54-b11d5f1e7810, TenantArmEndpoint: https://management.local.azurestack.external

VERBOSE: Adding TenantAzureStack Environment.

VERBOSE: Performing theoperation "adding environment" on target "AzureStackUser".

VERBOSE: Logging to AzureCloudwith tenantId: ****-***-4832-8b54-b11d5f1e7810

WARNING: Parameter 'Password' isobsolete. New-AzureRmADApplication: The parameter "Password" is beingchanged from a string to a SecureString in an upcoming breaking change release.

VERBOSE: Created new SPNClientID: ****-***-482e-b487-d2990c4bba65, Secret:****-****-4f49-aeb3-abc9f7b0d916

VERBOSE: Preparing the API model

VERBOSE: Placing the API modelto local location.

VERBOSE: Upload the locallycreated API model to a Storage Account.

VERBOSE: Creating or retrievingresource group: k8ssa-90322.

VERBOSE: Creating or retrievingstorage account: k8ssa90322.

VERBOSE: Creating or retrievingcontainer account: k8ssaci90322.

VERBOSE: Uploaded the API modelto: https://k8ssa90322.blob.local.azurestack.external/k8ssaci90322/azurestack.json.

VERBOSE: Blob root path: https://k8ssa90322.blob.local.azurestack.external/k8ssaci90322

可以看到，该PowerShell命令会从ERCS虚拟机的特权端点获取Azure Stack的证书指纹，并且会自动在Azure中国创建服务主体，其名称和密码如命令行所示。并且自动将这些参数写入到新生成的azurestack.json文件里。

确保Azure服务主体拥有Azure Stack的权限

由于Kubernetes需要能够在Azure Stack上创建负载均衡器、NSG规则等资源，所以需要对Azure服务主体(Azure SPN)进行赋权，以便其拥有合适的权限，其做法和Azure公有云并无二致。可以在上述的命令结果中找到SPN AppID，然后在Azure Stack租户订阅里给该SPN账户赋予参与者的权限。如图所示。

在Azure Stack上部署Kubernetes