Step by Step Guide: How to Configure SSL/TLS on ORACLE RAC (with SCAN) (Doc ID 1448841.1)

Posted 2021-03-23 yaoyangding

---

In this Document

Goal

Solution

References

---

 

Applies to:

Advanced Networking Option - Version 11.2.0.2 to 12.2.0.1 [Release 11.2 to 12.2]
Oracle Net Services - Version 12.2.1.2.0 to 12.2.1.2.0 [Release 12.2]
Information in this document applies to any platform.

Goal

This document will demonstrate the steps required to configure SSL/TLS on RAC or SIHA. Instruction is by example and also shows various methods to check the state of the configuration.

 "SSL" in this document refers to either SSL or TLS protocols.

Solution

1) Configure TCPS protocol endpoints.

In Oracle RAC, clients access one of three scan listeners and are then routed to database listeners. To support SSL all of these listeners must have TCPS protocol endpoints.
Follow steps 1.1 & 1.2 below to add TCPS endpoints to the database (node) listeners and then the scan listeners.

Before starting, a check of the listener resources shows support for TCP endpoints:

[oracle@net-rac1 ~]$ crsctl stat res -p |grep ENDPOINTS

ENDPOINTS=TCP:1521            <= database listener
ENDPOINTS=TCP:1521            <= listener_scan1
ENDPOINTS=TCP:1521            <= listener_scan2
ENDPOINTS=TCP:1521            <= listener_scan3

1.1) Adding TCPS endpoints to the Database listener(s)

[oracle@net-rac1 ~]$ srvctl modify listener -p "TCP:1521/TCPS:1523";

[oracle@net-rac1 ~]$ srvctl stop listener
[oracle@net-rac1 ~]$ srvctl start listener

Database listener configuration checks:

[oracle@net-rac1 ~]$ srvctl config listener
Name: LISTENER
Network: 1, Owner: oracle
Home: <CRS home>
End points: TCP:1521/TCPS:1523

[oracle@net-rac1 ~]$ lsnrctl status

Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcps)(HOST=10.141.155.188)(PORT=1523)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.141.155.183)(PORT=1521)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.141.155.188)(PORT=1521)))

[oracle@net-rac1 ~]$ crsctl stat res -p |grep ENDPOINTS

ENDPOINTS=TCP:1521 TCPS:1523  <= database listener
ENDPOINTS=TCP:1521            <= listener_scan1
ENDPOINTS=TCP:1521            <= listener_scan2
ENDPOINTS=TCP:1521            <= listener_scan3

1.2) Adding TCPS endpoints to scan listeners

[oracle@net-rac1 ~]$ srvctl stop scan_listener
[oracle@net-rac1 ~]$ srvctl stop scan

-------------------
[oracle@net-rac1 ~]$ srvctl modify scan_listener -p TCP:1521/TCPS:1523
------------------
- or -
-------------------
[oracle@net-rac1 ~]$ srvctl remove scan_listener -f
[oracle@net-rac1 ~]$ srvctl add scan_listener -l LISTENER -p TCP:1521/TCPS:1523
-------------------

[oracle@net-rac1 ~]$ srvctl start scan
[oracle@net-rac1 ~]$ srvctl start scan_listener

Scan listener configuration checks:

[oracle@net-rac1 ~]$ srvctl config scan_listener
SCAN Listener LISTENER_SCAN1 exists. Port: TCP:1521/TCPS:1523
SCAN Listener LISTENER_SCAN2 exists. Port: TCP:1521/TCPS:1523
SCAN Listener LISTENER_SCAN3 exists. Port: TCP:1521/TCPS:1523


[oracle@net-rac1 ~]$ lsnrctl status listener_scan3

Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER_SCAN3)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.141.155.186)(PORT=1521)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcps)(HOST=10.141.155.186)(PORT=1523)))

[oracle@net-rac1 ~]$ crsctl stat res -p |grep ENDPOINTS

ENDPOINTS=TCP:1521 TCPS:1523  <= database listener
ENDPOINTS=TCP:1521 TCPS:1523  <= listener_scan1
ENDPOINTS=TCP:1521 TCPS:1523  <= listener_scan2
ENDPOINTS=TCP:1521 TCPS:1523  <= listener_scan3

 

2) Update the "local_listener" startup parameter on each node.

PMON sends the endpoint values stored in local_listener to the SCAN listeners so that they can create appropriate service handlers. Add the TCPS endpoints
for the database/node listener that were created in step 1 to the local_listener startup parameter on each node.

The local listener ip address is unique to each node. State the local instance sid value when issuing alter system, e.g., sid=‘instance‘.

2.1) Select a node and identify the local listener endpoints:

[oracle@net-rac1 admin]$ lsnrctl status |grep PORT

Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcps)(HOST=10.141.155.188)(PORT=1523)))  <= new TCPS endpoint
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.141.155.183)(PORT=1521)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.141.155.188)(PORT=1521)))

The TCPS protocol endpoint is easily identified by the PROTOCOL value.

2.2) Before modifying, review the current local_listener value and make note of it.

[oracle@net-rac1 admin]$ sqlplus "/ as sysdba"

SQL> show parameter local_listener

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
local_listener                       string       (DESCRIPTION=(ADDRESS_LIST=(A
                                                 DDRESS=(PROTOCOL=TCP)(HOST=10.
                                                 141.155.188)(PORT=1521))))

2.3) Add the TCPS endpoint identified in step 2.1 to the local_listener value. Be sure to also set the sid to the local nodes instance name. Set the scope to memory so that changes can be verified before updating the spfile.

SQL> alter system set local_listener=‘(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=10.141.155.188)(PORT=1521))(ADDRESS=(PROTOCOL=TCPS)(HOST=10.141.155.188)(PORT=1523))))‘ scope=memory sid=‘NETRAC1‘;

After modification:

SQL> show parameter local_listener;

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
local_listener                       string       (DESCRIPTION=(ADDRESS_LIST=(A
                                                 DDRESS=(PROTOCOL=TCP)(HOST=10.
                                                 141.155.188)(PORT=1521))(ADDRE
                                                 SS=(PROTOCOL=TCPS)(HOST=10.141
                                                 .155.188)(PORT=1523))))

 

If the RAC cluster is using COST to restrict instance registration all local/node listener COST value lists must include TCPS. Without a TCPS rule the scan listeners TCPS handlers will go into a blocked state.  For more information please see DocID: 1537743.1 "Scan Listener TCPS Service Handlers are Blocked after Implementing COST on an SSL Cluster"

2.4) Once updated in memory PMON notifies the scan listeners (via registration) of the new local_listener value. Using lsnrctl confirm that the scan listeners have created service handlers for the new TCPS endpoints. If the handlers were created properly then commit the change to the spfile by re-issueing the command with "scope=both" (memory and spfile). If the scan listeners do not show the new TCPS handlers then retrace steps and correct any errors. The scan listeners are mirrors of each other so only one scan listener needs to be checked.

Writing final changes to the spfile:

SQL> alter system set local_listener=‘(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=10.141.155.188)(PORT=1521))(ADDRESS=(PROTOCOL=TCPS)(HOST=10.141.155.188)(PORT=1523))))‘ scope=both sid=‘NETRAC1‘;

2.5) Update the remaining nodes until all nodes are properly registering their TCPS endpoints with the scan listeners.

 

3) Create SSL certificates and wallets for the cluster and also for clients that will be connecting to the cluster over SSL.

The choice and usage of a CA (Certificate Cuthority) for certificate signing is up to the reader. DocID 1489301.1 provides the steps to emulate a CA environment using orapki if a CA is not already available in your organization.

To make a successful SSL connection the server and connecting clients must have unique SSL certificates that are signed by the same trusted Certificate Authority. Using the method of choice for your organization create certificate requests for the cluster and then for a test client that will connect to the database over SSL. Have those requests signed by the CA and then build wallets using the signed user certificates and trusted root certificate. (If this is not a familiar process use DocID 1489301.1).

In this SSL setup example there are two wallets, one for the cluster and one for a test client. The user DN‘s for those entities and for the CA are as follows:

Certificate Authority
  DN: CN=Networking/netfl-lablinux1,OU=OSS,O=Oracle,ST=Florida,C=US

RAC Cluster
  DN: C=US,CN=netrac
 
Test Client
  DN: CN=test client

The finished rac cluster (server side) wallet:

[oracle@net-rac1]$ orapki wallet display -wallet /tmp/rcwallet/ewallet.p12 -summary

Requested Certificates:
User Certificates:
Subject:        C=US,CN=netrac
Trusted Certificates:
Subject:        CN=Networking/netfl-lablinux1,OU=OSS,O=Oracle,ST=Florida,C=US

 

The finished client wallet:

[oracle@net-rac1 client_wallet]$ orapki wallet display -wallet . -summary

Requested Certificates:
User Certificates:
Subject:        CN=test client
Trusted Certificates:
Subject:        CN=Networking/netfl-lablinux1,OU=OSS,O=Oracle,ST=Florida,C=US

 

Note that the cluster and client wallets have unique identities but share the same trusted certificate. This is the proper wallet setup for an SSL connection.

 
4) Wallet placement and creation of the obfuscated wallet.

The finished cluster wallet that was created in step 3 should now to be copied to each node of the cluster. There is no specific rule to wallet placement except that the wallet location should be accessable by both the Database (PMON) and by the scan and local listeners which are normally running out of the GI home.

In this example we have chosen a DB home wallet location on each node:

/u01/app/oracle/product/12.1.0.2/db_1/network/admin/wallet

In addition to the PKCS12 wallet or ewallet.p12, an additional wallet file named cwallet.sso must also be created. The cwallet.sso is an obfuscated mirror copy of the ewallet.p12 and is the file that is accessed by PMON and listeners. If the cwallet.sso is created on the cluster it can be copied along with the ewallet.p12 to the wallet directory on each node. The cwallet.sso can also be created on each node separately if ewallet.p12 is already in place. Use the following command run from the same location as ewallet.p12 to create cwallet.sso (you will be prompted for the wallet password).

Syntax: orapki wallet create -wallet [ewallet.p12 location] -auto_login

[oracle@net-rac1 wallet]$ orapki wallet create -wallet /u01/app/oracle/product/12.1.0.2/db_1/network/admin/wallet -auto_login
Oracle PKI Tool : Version 12.1.0.2.0 - Production
Copyright (c) 2004, 2014, Oracle and/or its affiliates. All rights reserved.

Enter wallet password:


[oracle@net-rac1 wallet]$ ls -al

drwxr-xr-x. 2 oracle oracle 4096 Feb  7 11:12 .
drwxr-xr-x. 5 oracle oracle 4096 Feb 15 11:00 ..
-rw-------. 1 oracle oracle 2549 Feb 15 16:13 cwallet.sso
-rw-------. 1 oracle oracle 2472 Feb  7 11:11 ewallet.p12

 

5) Define wallet locations in listener.ora and sqlnet.ora.

As mentioned earlier both PMON and the listener processes of each node must be able to access the wallets.  Each node‘s sqlnet.ora and listener.ora will need to have wallet locations defined. This step should be performed on all nodes.

5.1) Listeners on an 11.2 RAC cluster will normally run out of the Grid Infrastructure (GI) home. Edit the $GRID_HOME/network/admin/listener.ora and add the following items:

SSL_CLIENT_AUTHENTICATION = FALSE

WALLET_LOCATION =
  (SOURCE =
    (METHOD = FILE)
    (METHOD_DATA =
      (DIRECTORY = /u01/app/oracle/product/12.1.0.2/db_1/network/admin/wallet)
    )
  )

 
5.2) Instances on an 11.2 RAC cluster run out of the Database home. Edit the database $ORACLE_HOME/network/admin/sqlnet.ora and add the following items:

SQLNET.AUTHENTICATION_SERVICES= (BEQ, TCPS)

SSL_VERSION = 0

SSL_CLIENT_AUTHENTICATION = FALSE

WALLET_LOCATION =
  (SOURCE =
    (METHOD = FILE)
    (METHOD_DATA =
      (DIRECTORY = /u01/app/oracle/product/12.1.0.2/db_1/network/admin/wallet)
    )
  )

 

6) Restart instances and listeners.

With wallets in place and ora files edited the PMON and listener processes must be restarted so that they pick up the new wallet settings. With the restart the instances will also use the local_listener values that were added in step 2. Re-confirm that scan listeners have the proper tcps handlers, correct any discrepancies.

Command examples:

srvctl stop listener
srvctl start listener

srvctl stop scan_listener
srvctl start scan_listener

srvctl stop database -d netrac
srvctl start database -d netrac

 
7) Testing from a cluster node.

With the cluster environment configured for SSL the simplest way to quickly test is to make an SSL connection on one of the cluster nodes.

7.1) Create a connect descriptor that uses the scan listener TCPS endpoint.

NETRACSSL =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCPS)(HOST = net-scan)(PORT = 1523))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = NETRAC.us.oracle.com)
    )
  )

 

7.2) Make a connection with sqlplus and the TCPS connect descriptor.

[oracle@net-rac1 admin]$ sqlplus mike/mike@netracssl

SQL*Plus: Release 12.1.0.2.0 Production on Fri Feb 22 17:38:17 2015
Copyright (c) 2004, 2014, Oracle.  All rights reserved.

Connected to:
Oracle Database 12 Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options

SQL>

 

8) Testing from a remote client.

8.1) Create a wallet directory and  add the location to the clients sqlnet.ora.

WALLET_LOCATION =
  (SOURCE =
    (METHOD = FILE)
    (METHOD_DATA =
      (DIRECTORY = C:apporacleproduct11.2.0dbhome_1NETWORKADMINwallet)
    )
  )

8.2) Move the client wallet created in step 3 to the client wallet directory and create the cwallet.sso.

C:apporacleproduct12.1.0.2dbhome_1NETWORKADMINwallet>orapki wallet create -wallet . -auto_login

Enter wallet password:

C:apporacleproduct12.1.0.2dbhome_1NETWORKADMINwallet>dir
 Volume in drive C has no label.
 Volume Serial Number is 0865-9427

 Directory of C:apporacleproduct12.1.0.2dbhome_1NETWORKADMINwallet

03/06/2012  04:01 PM    <DIR>          .
03/06/2012  04:01 PM    <DIR>          ..
02/22/2013  05:53 PM             1,101 cwallet.sso
02/28/2012  02:26 PM             1,024 ewallet.p12

8.3) Create a connect descriptor that uses the scan listener TCPS endpoint.

NETRACSSL =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCPS)(HOST = net-scan)(PORT = 1523))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = NETRAC.us.oracle.com)
    )
  )

8.4) Make a connection with sqlplus using the TCPS connect descriptor.

C:apporacleproduct12.1.0.2dbhome_1NETWORKADMIN> sqlplus mike/mike@netracssl

SQL*Plus: Release 12.1.0.2.0 Production on Fri Feb 22 17:56:22 2015
Copyright (c) 2004, 2014, Oracle.  All rights reserved.

Connected to:
Oracle Database 12 Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options

SQL>

References

NOTE:1504785.1 - Listener Does Not Create Handlers for Started Services With Multiple Endpoint Registration (ORA-12520, ORA-12516)
NOTE:1489301.1 - Using Orapki as Certificate Authority
NOTE:1537743.1 - Scan Listener TCPS Service Handlers are Blocked after Implementing COST on an SSL Cluster

Step by Step Guide: How to Configure SSL/TLS on ORACLE RAC (with SCAN) (Doc ID 1448841.1)

To Bottom