Hardening the network communications for your ClickHouse environment is about reducing exposure of someone listening in on traffic and using that against you. Network hardening falls under the following major steps:
IMPORTANT NOTE: Configuration settings can be stored in the default
/etc/clickhouse-server/config.xmlfile. However, this file can be overwritten during vendor upgrades. To preserve configuration settings it is recommended to store them in
/etc/clickhouse-server/config.das separate XML files with the same root element, typically
<yandex>. For this guide, we will only refer to the configuration files in
/etc/clickhouse-server/config.dfor configuration settings.
It’s easier to prevent entry into your system when there’s less points of access, so unused ports should be disabled.
ClickHouse has native support for MySQL client, PostgreSQL clients, and others. The enabled ports are set in the
To reduce exposure to your ClickHouse environment:
Review which ports are required for communication. A complete list of the ports and configurations can be found on the ClickHouse documentation site for Server Settings.
Comment out any ports not required in the configuration files. For example, if there’s no need for the MySQL client port, then it can be commented out:
<!-- <mysql_port>9004</mysql_port> -->
ClickHouse allows for both encrypted and unencrypted network communications. To harden network communications, unencrypted ports should be disabled and TLS enabled.
TLS encryption required a Certificate, and whether to use a public or private Certificate Authority (CA) is based on your needs.
- Public CA: Recommended for external services or connections where you can not control where they will be connecting from.
- Private CA: Best used when the ClickHouse services are internal only and you can control where hosts are connecting from.
- Self-signed certificate: Only recommended for testing environments.
Whichever method is used, the following files will be required to enable TLS with CLickHouse:
- Server X509 Certificate: Default name
- Private Key: Default name
- Diffie-Hellman parameters: Default name
No matter which approach is used, the Private Key and the Diffie-Hellman parameters file will be required. These instructions may need to be modified based on the Certificate Authority used to match its requirements. The instructions below require the use of
openssl, and was tested against version
Generate the private key, and enter the pass phrase when required:
openssl genrsa -aes256 -out server.key 2048
dhparam.pemto create a 4096 encrypted file. This will take some time but only has to be done once:
openssl dhparam -out dhparam.pem 4096
Create the Certificate Signing Request (CSR) from the generated private key. Complete the requested information such as Country, etc.
openssl req -new -key server.key -out server.csr
Store the files
dhparam.pemin a secure location, typically
Retrieving the certificates from a Public CA or Internal CA performed by registering with a Public CA such as Let’s Encrypt or Verisign or with an internal organizational CA service. This process involves:
- Submit the CSR to the CA. The CA will sign the certificate and return it, typically as the file
- Store the file
server.crtin a secure location, typically
Create a Private CA
If you do not have an internal CA or do not need a Public CA, a private CA can be generated through the following process:
Create the Certificate Private Key:
openssl genrsa -aes256 -out internalCA.key 2048
Create the self-signed root certificate from the certificate key:
openssl req -new -x509 -days 3650 -key internalCA.key \ -sha256 -extensions v3_ca -out internalCA.crt
Store the Certificate Private Key and the self-signed root certificate in a secure location.
server.csrfile with the self-signed root certificate:
openssl x509 -sha256 -req -in server.csr -CA internalCA.crt \ -CAkey internalCA.key -CAcreateserial -out server.crt -days 365
Store the file
Self Signed Certificate
To skip right to making a self-signed certificate, follow these instructions.
- IMPORTANT NOTE: This is not recommended for production systems, only for testing environments.
server.keyfile from previous steps, create the self-signed certificate. Replace
my.host.namewith the actual host name used:
openssl req -subj "/CN=my.host.name" -new -key server.key -out server.crt
Store the file
clickhouse-clientuser that connects to the server with the self-signed certificate will have to allow
invalidCertificateHandlerby updating their
clickhouse-clientconfiguration files at
<config> <openSSL> <client> ... <invalidCertificateHandler> <name>AcceptCertificateHandler</name> </invalidCertificateHandler> </client> </openSSL>
Enable TLS in ClickHouse
Once the files
dhparam.dem have been generated and stored appropriately, update the ClickHouse Server configuration files located at /etc/clickhouse-server/config.d.
To enable TLS and disable unencrypted ports:
/etc/clickhouse-server/config.dfiles. Comment out unencrypted ports, including
<!-- <http_port>8123</http_port> --> <!-- <tcp_port>9000</tcp_port> -->
Enable encrypted ports. A complete list of ports and settings is available on the ClickHouse documentation site for Server Settings. For example:
Specify the certificate files to use:
<openSSL> <server> <!-- Used for https server AND secure tcp port --> <certificateFile>/etc/clickhouse-server/server.crt</certificateFile> <privateKeyFile>/etc/clickhouse-server/server.key</privateKeyFile> <dhParamsFile>/etc/clickhouse-server/dhparams.pem</dhParamsFile> ... </server> ... </openSSL>
Encrypt Cluster Communications
If your organization runs ClickHouse as a cluster, then cluster-to-cluster communications should be encrypted. This includes distributed queries and interservice replication. To harden cluster communications:
Create a user for distributed queries. This user should only be able to connect within the cluster, so restrict it’s IP access to only the subnet or host names used for the network. For example, if the cluster is entirely contained in a subdomain named
logos2, etc. This internal user be set with or without a password:
CREATE USER IF NOT EXISTS internal ON CLUSTER 'my_cluster' IDENTIFIED WITH NO_PASSWORD HOST REGEXP '^logos$'
Enable TLS for interservice replication and comment out the unencrypted interserver port by updating the
<!-- <interserver_http_port>9009</interserver_http_port> --> <interserver_https_port>9010</interserver_https_port> -->
Set an the
/etc/clickhouse-server/config.dfiles, and include the internal username and password:
<interserver_http_credentials> <user>internal</user> <password></password> </interserver_http_credentials>
Enable TLS for distributed queries by editing the file
- For ClickHouse 20.10 and later versions, set a shared secret text and setting the port to secure for each shard:
<remote_servers> <my_cluster> <shard> <secret>shared secret text</secret> <!-- Update here --> <internal_replication>true</internal_replication> <replica> <host>logos1</host> <!-- Update here --> <port>9440</port> <!-- Secure Port --> <secure>1</secure> <!-- Update here, sets port to secure --> </replica> </shard> ...
- For previous versions of ClickHouse, set the internal user and enable secure communication:
<remote_servers> <my_cluster> <shard> <internal_replication>true</internal_replication> <replica> <host>logos1</host> <!-- Update here --> <port>9440</port> <!-- Secure Port --> <secure>1</secure> <!-- Update here --> <user>internal</port> <!-- Update here --> </replica> ... </shard> ...