Enable Kerberos secured Hadoop cluster with Cloudera Manager

Posted 2020-09-13

I created an secured Hadoop cluster for P&G with cloudera manager, and this document is to record how to enable kerberos secured cluster with cloudera manager. Firstly we should have a cluster that contains kerberos KDC and kerberos clients

1. Install KDC server
Only one server run this, note, kdc is only install on a single server

sudo yum -y install krb5-server krb5-libs krb5-workstation krb5-auth-dialog openldap-clients

This command will install Kerberos Server and some useful commands with krb5-workstation

2.Modify /var/kerberos/krb5kdc/kdc.conf

kdcdefaults] kdc_ports = 88
 kdc_tcp_ports = 88
 
[realms]
 PG.COM = {
  #master_key_type = aes256-cts
  acl_file = /var/kerberos/krb5kdc/kadm5.acl
  dict_file = /usr/share/dict/words
  admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab
  supported_enctypes = aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal
  max_renewable_life = 7d
 }

PG.COM used to be EXAMPLE.COM
and add max_renewable_life = 7d or more longer than this, 4w means 4 weeks

3. Modify/etc/krb5.conf

libdefaults]default_realm = PG.COM
dns_lookup_kdc = false
dns_lookup_realm = false
ticket_lifetime = 259200
renew_lifetime = 604800
forwardable = true
default_tgs_enctypes = rc4-hmac
default_tkt_enctypes = rc4-hmac
permitted_enctypes = rc4-hmac
udp_preference_limit = 1
kdc_timeout = 3000
[realms]
 PG.COM = {
  kdc = pg-dmp-master2.hadoop
  admin_server = pg-dmp-master2.hadoop
 }

EXAMPLE.COM to PG.COM and in [realms] area, kdc point to node where I install KDC and admin_server point to server where installed kadmin

4. Create the realm

kdb5_util create -s -r PG.COM

This will create the working realm named PG.COM

5. Create the admin user principle in PG.COM on KDC server

kadmin.local -q "addprinc root/admin"

I use root/admin as an admin user, you can type in a different password for this admin, note, root/admin is not same as root, they can keep in kerberos’s database either, and they are two account.

6. Edit /var/kerberos/kadm.acl

*/[email protected]

This will define who is admin in PG.COM realm, this means everyone with /[email protected] could be kerberos admin.

7. Check all the kerberos’s configure files, ensure there are no errors.

/etc/krb5.conf
/var/kerberos/krb5kdc/krb.conf
/var/kerberos/krb5kdc/kadm.acl

8. And now start KDC and Kadmin service on KDC server

service krb5kdc start
service kadmin start

This will start KDC and Kadmin service on KDC server

9. Login to all other nodes of this cluster, and run this

yum install krb5-workstation krb5-libs krb5-auth-dialog openldap-clients cyrus-sasl-plain cyrus-sasl-gssapi

Installing kerberos clients on all nodes

And then, back to Cloudera Manager , click Cluster->Operation, and find enable kerberos in dropdown menu. Then enable all the question, and Next->Next->Next… till it ends.

When the all procedure done, you can use kerberos secured hadoop cluster.

 

So there were some more questions when installed kerberos secured cluster.

How could I add a new user principle?

kadmin.local -q "addprinc [email protected]"

And the note is , when you add a common user to the cluster, you should run this command above on Kadmin(KDC) server, and [email protected] is a normal user in kerberos, if you want to add an admin user, you should use username/[email protected], the /admin is which you defined in kadm.acl

 

How should I administrating HDFS or YARN, which I mean, how to use hdfs user or yarn user?

Well, since cloudera manager will automatically create several users in cluster such as hdfs, yarn, hbase… and these users were all defined as nologin without any password. And these users are all admin users of Hadoop but not in kerberos database. So you can’t ask ticket like this.

# kinit hdfs
kinit: Client not found in Kerberos database while getting initial credentials
# kinit [email protected]
kinit: Client not found in Kerberos database while getting initial credentials

But you can ask ticket by using keytabs of these cluster users like this

 kinit -kt hdfs.keytab hdfs/[email protected]

current_server is the hostname of  your current logged in server and wants to access with HDFS or YARN, in cluster, hdfs.keytab are all different on each node. so you must write command like this above.

 

When I  run a smoke test like Pi, it gone wrong? I’ve already add a new user principle in KDC database.

17/04/24 11:18:35 INFO mapreduce.Job: Job job_1493003216756_0004 running in uber mode : false
17/04/24 11:18:35 INFO mapreduce.Job:  map 0% reduce 0%
17/04/24 11:18:35 INFO mapreduce.Job: Job job_1493003216756_0004 failed with state FAILED due to: Application application_1493003216756_0004 failed 2 times due to AM Container for appattempt_1493003216756_0004_000002 exited with  exitCode: -1000
For more detailed output, check application tracking page:http://pg-dmp-master1.hadoop:8088/proxy/application_1493003216756_0004/Then, click on links to logs of each attempt.
Diagnostics: Application application_1493003216756_0004 initialization failed (exitCode=255) with output: main : command provided 0
main : run as user is dmp
main : requested yarn user is dmp
User dmp not found

Failing this attempt. Failing the application.
17/04/24 11:18:35 INFO mapreduce.Job: Counters: 0
Job Finished in 1.094 seconds
java.io.FileNotFoundException: File does not exist: hdfs://PG-dmp-HA/user/dmp/QuasiMonteCarlo_1493003913376_864529226/out/reduce-out
        at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1257)
        at org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:1249)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1249)
        at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1817)
        at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1841)
        at org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:314)
        at org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:354)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:363)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
        at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
        at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

In an unsecured cluster, hadoop will distibuted all the containers to nodes and start the containers as an exists username such as yarn. But in a secured cluster, job’s containers will be distributed and run as the username who you submitted it.  So, this error is you add a principle only, but there were no such a user on every node. So the container executor can not start up the application master or mapper or reducer. And you must add this user to each node to solve this problem. Simply use linux command

useradd dmp

On every node in cluster and client, another express way to add user to all nodes you can use openldap to instead add user manually.

 

Finally we should talk about some basic theory of kerberos

There are three phases required for a client to access a service
– Authentication
– Authorization
– Service request

Client sends a Ticket-Granting Ticket (TGT) request to AS

AS checks database to authen2cate client
  – Authentication typically done by checking LDAP/Active Directory
  – If valid, AS sends Ticket/Granting Ticket (TGT) to client

Client uses this TGT to request a service ticket from TGS
  – A service Bcket is validation that a client can access a service

TGS verifies whether client is permitted to use requested service
  – If access granted, TGS sends service Bcket to client

Client can use then use the service
  – Service can validate client with info from the service ticket

kinit program is used to obtain a ticket from Kerberos
klist to see users in kerberos database
kdestroy to explicitly delete your ticket

( Pics and their comments are reffered from Cloudera Administrator Training Course, copyright to cloudera.com )

本文出自 “实践检验真理” 博客，谢绝转载！