Views, News and Cheers!
On Tuesday, 06th March, I delivered a session on Linux Kernel Architecture at CSE department of SRKR Engineering College, Bhimavaram. The session’s goal is to introduce various subsystems of Linux Kernel and how they work together to deliver a flexible and robust operating system. Here are a few pictures from the event.
Slides used in the linux kernel session are here in the classes page.
Here is a short description of the scenario I was working on. I am using a standard AWS AMI to run tomcat (tomcat7, to be specific.) The default configuration of AWS AMIs (and many other off-the-shelf unix based servers) is such that tomcat (or any other program that runs with a non-superuser credentials) can’t bind to privileged ports. However, tomcat needs to use these privileged ports (443 for TLS and 80 for standard HTTP) to serve public facing pages.
Making tomcat run as superuser is really a bad idea (the why question is beyond this article.) So there are a few tricks to make tomcat work on privileged ports.
There is lot of mindshare around authbind when it comes to hosted environments. The manpage of authbind describes how authbind can be used to make a program bind to sockets on privileged ports. However, if you are using a standard AWS AMI, you may have some challenges using authbind. Also, for automated environments (read Chef) in AWS, I felt authbind is more complicated to work with.
Port redirection using NAT features of iptables is very simple and straight forward. However, it requires an additional configuration on tomcat to use proxy mode on privileged ports.
Here is the NAT configuration using iptables.
sudo /sbin/iptables -t nat -I PREROUTING -p tcp --dport 80 -j REDIRECT --to-port 8080 sudo /sbin/iptables -t nat -I PREROUTING -p tcp --dport 443 -j REDIRECT --to-port 8443 sudo service iptables save
Once this is done, all inbound traffic on 80 will be redirected to 8080. The same is the case with port pair 8443 and 443. This way, tomcat can still bind to port 8080 for HTTP and 8443 for TLS while serving incoming connections on 80 and 443 respectively.
When a client program queries the port information from tomcat, it should respond with port 80 and 443 instead of 8080 and 8443. To ensure that, one can use the proxy support feature of tomcat. Here is the additional configuration in tomcat connector settings in server.xml
<Connector port="8443" proxyPort="443" .../> <Connector port="8080" proxyPort="80" .../>
There are better ways to handle this port redirection when you have front-ending loadbalancers and/or proxy servers in place. Having proxy/loadbalancers solves helps mitigate more issues than just solving the redirection problems. However, the iptables approach is better than authbind approach when you are using a single server on AWS without lot of additional infrastructure and configurations in place.
Bash, the quarter century old shell utility on almost all popular unix based systems, is found to be vulnerable. The exploit works by injecting specially crafted values into an environment variable and using it to invoke a shell command. Once the exploit gets to that level, there is hardly any limit on what can be executed as part of the shell command.
The problem gets worse for the fact that many of the day to day usages of the network facing services have potential to use bash internally. For example, CGI scripts on web servers, convenience utilities offered by network routers and any other limited command execution tools might be the key vulnerability public and guest access private networks. Mitre warns that sshd with ForceCommand is a potential attack vector.
The bug is being termed as Shellshock bug or bash bug. RedHat’s security blog article is one of the earliest articles that discussed the Shellshock bug in detail. Robert Graham of Errata Security is the best known tracker of the issue and has ongoing observations and comments on his blog/twitter account.
Here is how you can check if the current bash is vulnerable on your system. If it prints vulnerable on the first line, then patch your bash package.
$ env x='() { :;}; echo vulnerable' bash -c "echo test completed"
vulnerable
test completed
For web servers, here is the test suggested:
$ curl -i -X HEAD "http://sometestdomainhere.com/" -A '() { :;}; echo "Warning: Server Vulnerable"'
The output looks somewhat like the following listing. If it contains “Warning” text, then it is highly likely that the web server’s bash is (and cgi’s based on bash are) vulnerable. This test doesn’t assure that the system is not vulnerable. You may still have other CGIs run with bash that are vulnerable.
HTTP/1.1 200 OK Date: Fri, 26 Sep 2014 02:51:52 GMT Server: Apache X-Powered-By: PHP/5.4.32 X-Pingback: http://sometestdomainhere.com/xmlrpc.php Link: <http://sometestdomainhere.com/?p=14>; rel=shortlink Content-Type: text/html; charset=UTF-8
Since the Shellshock bug existed for quite a while, all versions of bash that are currently out there in active usage are likely to be vulnerable. Patching some of these devices might be trivial, but there still might be several other devices that are hard to patch.
Errata Security also has notes on wormable nature of the Shellshock bug. So patch your bash package as early as you can.
If you are using AWS and EC2 instances, a reboot of most those instances is on the horizon. Amazon’s AWS informed of this reboot that is scheduled between 02:00 GMT on September 26th and 23:59 GMT on September 30th.
Read more about this reboot on Gigaom and Rightscale. Technical Forums on AWS and other sites are already buzzing with lot of traffic, discussing the potential impact and how to ensure that the services are not impacted.
Given the urgency and magnitude of the instances that are impacted, it looks like the patch is going potentially going to address a security vulnerability. The actual details of the patch and the issues that are fixed by it will be known around October 01st.
Summarizing various discussions on related forums, here is a quick summary of what to watch out for during this AWS / EC2 instance reboot
Here are a few quick checks for those who are getting impacted.
For those who can afford to be heroic enough – why wait till AWS reboots your instances? Reboot these on your own in each availability zone and test the resilience.