Network Security in Application Development

A friend recently asked me to help diagnose a problem in his system. He handed me a username and password and told me to SSH into one of his servers—a Redis instance that cached database queries and handled some other business logic. The issue took considerable effort, off and on, to pin down, and along the way, at my request, he gave me the credentials for two more servers. In the process, I noticed his weak awareness of network security—something likely shared by many web/mobile app developers. Last year I wrote an article on security focused mainly on social engineering; today I want to talk about network security.

In the example above, my friend made several common mistakes:

1. The server was exposed directly to the internet via a public IP.
2. The management port (SSH port) placed no restrictions on source IPs.
3. He used username + password authentication instead of username + key authentication.

Below, let’s discuss how to address these mistakes.

Don’t expose unnecessary services

Network design has a famous principle: the principle of least privilege. The principle of least privilege requires that every module at a given layer of abstraction in a computing environment—whether a process, a user, or a computer program—can only access the information or resources necessary at that moment. When we grant the smallest possible privilege to every legitimate action, we protect data and functionality from being damaged by error or malicious behavior.

In theory, an internet service should only expose to any source IP those services that must be public, such as a web server, an API server (which is essentially a web server), a load balancer, and so on. In terms of ports, this mainly means 80 (HTTP) and 443 (HTTPS). Other than the public-facing servers that need a public IP, every other server should have only a private IP. If the application is deployed on AWS, then only a public ELB is needed; if the application is deployed in one’s own (or a third party’s) data center, then only the servers running nginx/haproxy and similar services need public IPs.

Internal services such as database servers and cache servers, because they only serve the web servers and do not serve end users, must have only private IPs.

Every server, apart from the essential public-facing service ports, should have all other ports sealed off. If a database server runs MySQL, then open only its default port 3306. When opening a port, pay attention to which source IPs are permitted to access it. Even for a server with a public IP (such as a web server), the source IPs allowed to access SSH should preferably be restricted to internal IPs within the same subnet.

Building a secure SSH service

SSH is mainly used in three scenarios:

1. Server diagnostics
2. Server configuration and software installation
3. Other command-line-based server management tasks

Nowadays, with log service tools like logstash/reimann, logs and events from every server can be centralized on a hub node for convenient querying and diagnostics. So the SSH logins that exist for the purpose of server diagnostics can be replaced. If you mainly use SSH for this purpose, consider choosing a suitable log service tool.

Server configuration and software installation are the main stage for most SSH logins. But in this highly automated era, server provisioning should be done through various ops tools (such as ansible/chef), not by hand. If a large amount of server configuration and software installation is still done manually over SSH, then the team should consider learning an ops tool.

Although in 90% of cases server diagnostics and server configuration/installation can be done through tools, there remain a small number of one-off tasks that must rely on SSH. For example, when a service is unreachable and leaves no logs, you may need to run tcpdump on the server to capture packets for network diagnostics. So temporary SSH access is still necessary.

For this, there are two approaches: 1) set up an SSH server at the network boundary (as shown in the figure above), or 2) configure an SSL VPN.

Set up an SSH server at the boundary between the internal and external networks. Administrators can SSH in via its public IP, then use it as a jump host to SSH further into other servers within the internal network for management.

A dedicated SSH server, because it can be accessed from any location on the internet, needs some special handling: logged-in users should have only minimal privileges (no sudo), should not be able to install software, and the software runnable on the system should ideally be minimized to just SSH and the necessary tools.

Beyond that, you can apply these settings:

• Set SSH login to allow only username + private key authentication, eliminating the possibility of passwords being cracked in transit over the network.
• Change the SSH port to a non-well-known port. While this won’t help against a targeted attack (someone specifically targeting your server can still sniff out the open port one by one against your IP), it prevents you from being singled out by broad attackers (some attacks sniff port 22 across the entire internet, then further attack servers that have that port open).
• Restrict the source IPs that can access it in iptables—unless necessary, do not allow arbitrary source IPs.

Compared with a dedicated SSH server, a more “professional” approach is to configure an SSL VPN. The principle of an SSL VPN is to establish a tunnel between your computer and the target network, so that your computer obtains an internal IP and routes. When you initiate access to a server on the internal network, the IP packets generated by the protocol stack (inner packets) enter the SSL VPN tunnel, are encapsulated with a public IP header, and encrypted using TLS; when the data emerges from the tunnel, it is decrypted, the inner packets extracted, and dispatched to their destination. From the server’s perspective, your access is equivalent to an access coming from within the internal network.

Configuring an SSL VPN is fairly complex, so I won’t elaborate here—you can follow the openvpn documentation step by step. Besides installing it on the server side, each computer that needs to access the internal network must also install VPN software (such as tunnelblick; if you need dynamic tokens, you also need to install Google Authenticator), and generate a client certificate for it. The whole process is troublesome, but the benefit of the trouble is that most attackers can be kept out.

Dynamic security

Planning the network into internal and external zones, granting the outside world minimal access privileges, and allowing users to access the internal network via SSL VPN when necessary for management tasks—from a network security standpoint, this already achieves a relatively high level of security. But in some scenarios with extremely high security requirements, you should also apply dynamic security.

For example, suppose you have a separate database cluster holding user credit card data. Only an administrator named tyr can log into the servers in that cluster. tyr needs to connect to the internal network via SSL VPN, obtain a route to the servers, and then SSH into those servers using his own private key. Although this is already secure enough, the SSH port is open all the time. If a server in the internal network that has an external IP (such as a web server) is compromised, an attacker can use that server as a jump host for further attacks—so an SSH port that stays open all the time carries potential risk. In fact, the SSH port doesn’t need to stay open all the time—is there a way to open the port in iptables only when access is desired, and close it afterward?

When you think of such a question, you’re not far from the solution: you can write your own service—call it, say, knock-knock. Upon receiving a specific command, e.g. the client says, “King of Heaven covers the tiger,” the knock-knock service opens access from that client to the SSH port in iptables; when the client finishes, it says, “The pagoda subdues the river demon,” and the knock-knock service closes the port it just opened in iptables.

There’s a further minor problem: how does the client connect to the knock-knock service? If knock-knock listens on a particular port, wouldn’t that be out of the frying pan and into the fire—closing the SSH port but opening another port with potential risk? Good question. Remember the fun technology from ten years ago called Wakeup-On-LAN? As long as you send a specific Ethernet frame to the target machine within the LAN, the target machine’s network card, upon receiving it, powers on the machine. We can use a similar technique: send specific packets to the target machine, and the service running on that machine—using libpcap to listen to all network traffic—executes the corresponding action when it encounters a signature it recognizes.

Such a service actually has its own name: port-knocking, and there are open source implementations, one of which is knock.

As mentioned, knock can help you dynamically open and close the SSH port, further enhancing server security. It can be used like this:

$ knock yourserver.yourdomain.com 1111:tcp 2222:udp 3333:tcp

When the network receives these three pre-defined connection-establishment packets to specific ports in sequence (for TCP, these are SYN packets), knock executes some action (opening the machine’s SSH port).

Firewalls

Most cloud providers offer basic network firewall functionality, which can restrict inbound and outbound traffic at the network layer. With the security measures above in place and reasonable firewall rules (or ACLs), your network is already fairly secure. If you need more (and more advanced) security measures, consider deploying a software or hardware firewall at the network edge. I won’t go into that here.

The security content covered above is only network security. Security is a comprehensive solution—different security must be done well at different layers. No matter how secure the network, if application-layer security is not done well, you’ll still be attacked until you’re black and blue.