Community Articles

Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.

Showing results for Show only | Search instead for Did you mean: Announcements Celebrating as our community reaches 100,000 members! Thank you!

Enabling Https for AmbariServer and troubleshooting secure communication

Apache Ambari

Master Mentor

Created on ‎06-15-2016 01:55 PM - edited ‎08-17-2019 12:00 PM

In this example we will see how to use a self-signed certificate for initial trials to setup ambari console access over HTTPS. Remember that the self signed certificates are not suitable for production environments. So if your organization does not provide you with an SSL server certificate, you must request a new certificate that is signed by a CA.

Users can use several methods to obtain a new signed certificate. For example, users can use Microsoft Internet Information Services (IIS) Manager to request an SSL server certificate from a CA. For testing purposes, you can obtain a free temporary certificate based on an untrusted root from many CAs. To know more about it refer to : https://en.wikipedia.org/wiki/Certificate_authority.

This article is divided into 4 sections.

Section-1). Generating keys self signed certificates.

Section-2). Configuring Ambari Server to use the certificates.

Section-3). Common troubleshooting steps.

Section-4). Setting up the Ambari Server truststore.

Generating keys self signed certificates.

Make sure that "openssl" is installed.

Step-1). Set the ambari host name as an environment variable "AMBARI_SERVER_HOSTNAME" which we will later use to create the certificate name like "ambari1.example.com.key". Users can choose any name for their certificates/keys not necessarily the hostname.

mkdir /etc/ambari-server/certs cd /etc/ambari-server/certs/ export AMBARI_SERVER_HOSTNAME=ambari1.example.com

Step-2). Use the "genrsa" command which will generate an RSA private key. 2048 is the size of the private key to generate in bits. The default is 512. This must be the last option specified.

cd /etc/ambari-server/certs/ openssl genrsa -passout pass:hadoop -out $AMBARI_SERVER_HOSTNAME.key 2048

Output should be something like following:

[root@ambari1 ~]# openssl genrsa -passout pass:hadoop -out $AMBARI_SERVER_HOSTNAME.key 2048 Generating RSA private key, 2048 bit long modulus ..+++ . +++

Step-3). Use the "req" command which primarily creates and processes certificate requests in PKCS#10 format. This option can also be used to create self signed certificates for use as root CAs for example.

openssl req -new -key $AMBARI_SERVER_HOSTNAME.key -out $AMBARI_SERVER_HOSTNAME.csr -subj "/C=IN/ST=Karnataka/L=Bangalore/O=hwx/CN=$AMBARI_SERVER_HOSTNAME"

Step-4). Generating private key "$AMBARI_SERVER_HOSTNAME.crt" using the option "-out" as following:

openssl x509 -req -days 365 -in $AMBARI_SERVER_HOSTNAME.csr -signkey $AMBARI_SERVER_HOSTNAME.key -out $AMBARI_SERVER_HOSTNAME.crt

Now in the "/etc/ambari-server/certs" directory users should see the following files:

[root@ambari1 certs]# cd /etc/ambari-server/certs [root@ambari1 certs]# ls -lart -rw-r--r-- 1 root root 1675 Jun 15 00:01 ambari1.example.com.key -rw-r--r-- 1 root root 997 Jun 15 00:02 ambari1.example.com.csr -rw-r--r-- 1 root root 1184 Jun 15 00:02 ambari1.example.com.crt

Configuring Ambari Server to use the certificates.

Step-5). Use the [ ambari-server setup-security ] option to configure the certificates.

[root@ambari1 certs]# ambari-server setup-security Using python /usr/bin/python Security setup options. =========================================================================== Choose one of the following options: [1] Enable HTTPS for Ambari server. [2] Encrypt passwords stored in ambari.properties file. [3] Setup Ambari kerberos JAAS configuration. [4] Setup truststore. [5] Import certificate to truststore. =========================================================================== Enter choice, (1-5): 1 Do you want to configure HTTPS [y/n] (y)? y SSL port [8443] ? 8443 Enter path to Certificate: /etc/ambari-server/certs/ambari1.example.com.crt Enter path to Private Key: /etc/ambari-server/certs/ambari1.example.com.key Please enter password for Private Key: hadoop Importing and saving Certificate. done. Ambari server URL changed. To make use of the Tez View in Ambari please update the property tez.tez-ui.history-url.base in tez-site Adjusting ambari-server permissions and ownership. NOTE: Restart Ambari Server to apply changes ("ambari-server restart|stop+start")

Step-6). Restart the ambari server and then try accessing the following URL:

ambari-server restart

Step-7). As we are using untrusted Self signed certificates hence we will see the following message in the web browser. In this case it is normal hence click on the " Proceed to ambari1.example.com (unsafe) "

After proceeding further we will see the following console:

Common troubleshooting steps?

Step-9). In order to troubleshoot any SSL related issue users can enable the " javax.net.debug " in the " /var/lib/ambari-server/ambari-env.sh " file by adding the following line in this file.

export AMBARI_JVM_ARGS=$AMBARI_JVM_ARGS' -Djavax.net.debug=ssl'

Now restart the Ambari Server. For more detailed debugging users can also use "- Djavax.net.debug=all "

Step-10). Due to the above debug flag now we can see the following kind of logging in the " ambari-server.out " file.

NOTE: This is " /var/log/ambari-server/ambari-server.out " STDOUT file and not the log file " /var/log/ambari-server/ambari-server.log ".

INFO: Initiating Jersey application, version 'Jersey: 1.18 11/22/2013 01:21 AM' *** found key for : 1 chain [0] = [ [ Version: V1 Subject: CN=ambari1.example.com, O=hwx, L=Bangalore, ST=Karnataka, C=IN Signature Algorithm: SHA1withRSA, OID = 1.2.840.113549.1.1.5 Key: Sun RSA public key, 2048 bits modulus: 22175988901159630311396514667235749078146521163847992788959263233423755273156821620769244786460484287586602112399844584928245594827889197162513174308816812595461796535334661659155198218017182049796110412714443009008738097645038580643845113158858593395477455683493690692652792939276214070135191952868572717664244815115851111728266217307471648754541658144069497483918718341865043212851552819598807666038781460770942617500564581986900865648063480231718632571176573148407333265113899006488650860419740994935408185496217967665765639595473969860865574492100353172445226695292868455012371476694881518611913455083475879522469 public exponent: 65537 Validity: [From: Wed Jun 15 00:02:13 PDT 2016, To: Thu Jun 15 00:02:13 PDT 2017] Issuer: CN=ambari1.example.com, O=hwx, L=Bangalore, ST=Karnataka, C=IN SerialNumber: [ ffeef037 fde4559f] ] Algorithm: [SHA1withRSA] Signature: 0000: 5F CF 38 F6 33 5D DE B9 2F 49 61 DD 08 AA EA 85 _.8.3]../Ia. 0010: 91 D9 89 07 80 B6 91 61 17 76 42 F6 3F A8 D3 27 . a.vB. ' 0020: C0 A1 A2 1D B3 2C DD 90 AA 4D B2 E7 B7 55 89 0C . M. U.. 0030: C8 48 F9 6B 05 5E 9C C9 25 B8 72 49 DB 4A E0 C0 .H.k.^..%.rI.J.. 0040: 32 F1 AB EF 4C C2 03 28 B7 C1 CB 3F 55 86 11 DB 2. L..(. U. 0050: 72 0C AF AA D0 75 0F 85 4D 5B 90 C1 E3 2F A1 70 r. u..M[. /.p 0060: 58 57 57 5F C3 C5 64 25 A9 A9 C5 B1 11 FA 08 68 XWW_..d%. h 0070: 4D 27 C7 69 CB E9 DE 0F 47 60 2E BE 3D 76 E6 51 M'.i. G`..=v.Q 0080: A5 04 4B 7F 81 83 F7 01 A8 A5 EE 1D E1 FE 20 8F ..K. . 0090: 06 B0 3B 07 4A ED A7 BC F4 8F 2A 41 85 B6 C5 78 ..;.J. *A. x 00A0: 91 FA 0F F8 A3 BE 75 77 12 9B 10 8E C2 B4 20 B3 . uw. . 00B0: 8D FB E3 71 80 BD B9 17 65 40 0C 01 11 3C 3D BB . q. e@. 

With the debug we can see the complete SSL communication detail in the out file and can find any failure in the communication. Also we can get other details like " unsupported cipher suite ", etc.


Step-11). Users can also disable some weak ciphers by editing the " /etc/ambari-server/conf/ambari.properties " file by editing the following properties. For multiple ciphers users can separate each cipher using a vertical bar |.


security.server.disabled.ciphers=TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA


Step-12). To disable specific protocols like (SSL, SSLv2, SSLv3) , users can optionally add a list of the following format to ambari.properties .


security.server.disabled.protocols=SSL|SSLv2|SSLv3


Setting up Ambari Server truststore


What is truststore ? In a SSL handshake the purpose of trustStore is to verify credentials and the purpose of keyStore is to provide credential. keyStore in Java stores private key and certificates corresponding to their public keys and require if you are SSL Server or SSL requires client authentication. Users can use the " $JAVA_HOME/bin/keytool " to create the keystores/truststores.


If we want the SSL communication between Ambari and 3rd party services like "Hive View" then may need to configure the truststore in Ambari Server. For example if we configure the Ambari to run on HTTPs then if we navigate to the Ambari "Hive View" tab then we may find the following kind of error :


Also on the ambari server logs we may see the same error as following "Truststore path or password is not set"


15 Jun 2016 05:30:14,861 ERROR [qtp-ambari-client-4139] ServiceFormattedException:99 - Can't get secure connection to https://ambari1.example.com:8443/api/v1/clusters/ClusterDemo/hosts?fields=Hosts%2Fpublic_host_name%2. . Truststore path or password is not set. 15 Jun 2016 05:30:14,862 ERROR [qtp-ambari-client-4139] ServiceFormattedException:100 - java.lang.IllegalStateException: Can't get secure connection to https://ambari1.example.com:8443/api/v1/clusters/ClusterDemo/hosts?fields=Hosts%2Fpublic_host_name%2. . Truststore path or password is not set. java.lang.IllegalStateException: Can't get secure connection to https://ambari1.example.com:8443/api/v1/clusters/ClusterDemo/hosts?fields=Hosts%2Fpublic_host_name%2. . Truststore path or password is not set. at org.apache.ambari.server.controller.internal.URLStreamProvider.getSSLConnection(URLStreamProvider.java:286) at org.apache.ambari.server.controller.internal.URLStreamProvider.processURL(URLStreamProvider.java:172) at org.apache.ambari.server.view.ViewAmbariStreamProvider.getInputStream(ViewAmbariStreamProvider.java:118) at org.apache.ambari.server.view.ViewAmbariStreamProvider.readFrom(ViewAmbariStreamProvider.java:78) at org.apache.ambari.view.utils.ambari.URLStreamProviderBasicAuth.readFrom(URLStreamProviderBasicAuth.java:65) at org.apache.ambari.view.utils.ambari.AmbariApi.requestClusterAPI(AmbariApi.java:173) at org.apache.ambari.view.utils.ambari.AmbariApi.requestClusterAPI(AmbariApi.java:142) at org.apache.ambari.view.utils.ambari.AmbariApi.getHostsWithComponent(AmbariApi.java:99) at org.apache.ambari.view.hive.client.ConnectionFactory.getHiveHost(ConnectionFactory.java:79) at org.apache.ambari.view.hive.client.ConnectionFactory.create(ConnectionFactory.java:68) at org.apache.ambari.view.hive.client.UserLocalConnection.initialValue(UserLocalConnection.java:42) at org.apache.ambari.view.hive.client.UserLocalConnection.initialValue(UserLocalConnection.java:26) at org.apache.ambari.view.utils.UserLocal.get(UserLocal.java:66) at org.apache.ambari.view.hive.resources.browser.HiveBrowserService.databases(HiveBrowserService.java:87)


In order to resolve this issue lets configure the "truststore" on Ambari Server. Following are the steps to create & configure the truststore for Ambari Server:


Step-1). Use the "$JAVA_HOME/bin/keytool" to generate the truststore based on the previously created "ambari1.example.com.crt" certificate.


[root@ambari1 ambari-server]# cd /etc/ambari-server/certs/ [root@ambari1 certs]# keytool -import -file /etc/ambari-server/certs/ambari1.example.com.crt -alias ambari-server -keystore ambari-server-truststore Enter keystore password: hadoop Re-enter new password: hadoop Owner: CN=ambari1.example.com, O=hwx, L=Bangalore, ST=Karnataka, C=IN Issuer: CN=ambari1.example.com, O=hwx, L=Bangalore, ST=Karnataka, C=IN Serial number: ffeef037fde4559f Valid from: Wed Jun 15 00:02:13 PDT 2016 until: Thu Jun 15 00:02:13 PDT 2017 Certificate fingerprints: MD5: 46:44:9D:52:90:C1:D6:9D:9B:2F:11:53:7E:50:8F:16 SHA1: 61:5E:85:B4:4D:3D:C8:7A:BB:0B:C3:48:07:D6:A4:BD:79:BD:A8:A1 Signature algorithm name: SHA1withRSA Version: 1 Trust this certificate? [no]: yes Certificate was added to keystore


Step-2). Configuring Ambari Server to use the "/etc/ambari-server/certs/ambari-server-truststore" using the command [ ambari-server setup-security ] by choosing option (4) as following.


NOTE: Users should stop the ambari-server before making any truststore related configuration changes.


[root@ambari1 ~]# ambari-server stop [root@ambari1 certs]# ambari-server setup-security Using python /usr/bin/python Security setup options. =========================================================================== Choose one of the following options: [1] Enable HTTPS for Ambari server. [2] Encrypt passwords stored in ambari.properties file. [3] Setup Ambari kerberos JAAS configuration. [4] Setup truststore. [5] Import certificate to truststore. =========================================================================== Enter choice, (1-5): 4 Do you want to configure a truststore [y/n] (y)? y TrustStore type [jks/jceks/pkcs12] (jks):jks Path to TrustStore file :/etc/ambari-server/certs/ambari-server-truststore Password for TrustStore: hadoop Re-enter password: hadoop Ambari Server 'setup-security' completed successfully.


Step-3). Start the Ambari Server.


[root@ambari1 certs]# ambari-server start Using python /usr/bin/python Starting ambari-server Ambari Server running with administrator privileges. Organizing resource files at /var/lib/ambari-server/resources. Server PID at: /var/run/ambari-server/ambari-server.pid Server out at: /var/log/ambari-server/ambari-server.out Server log at: /var/log/ambari-server/ambari-server.log Waiting for server start. Ambari Server 'start' completed successfully.


Step-4). The hive view should not be throwing the previously mentioned error now "Truststore path or password is not set" The "ambari.properties" should have the following entry in it now:


ssl.trustStore.type=jks ssl.trustStore.path=/etc/ambari-server/certs/ambari-server-truststore


Similarly if you notice the following kind of error:


org.apache.ambari.view.utils.ambari.AmbariApiException: RA040 I/O error while requesting Ambari . .Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target


The please go through the steps described in the following article amd make sure that we imported AmbariServer certificate inside the Ambari Server's truststore.