Hardening Solaris Systems in Production

When you’re running enterprise workloads on Solaris — databases, application servers, web fronts — the gap between “it works” and “it’s secure” is a lot wider than most people expect. I spent a good chunk of time working through hardening procedures for a fleet of Solaris boxes, and what follows is the procedure we landed on.

The scope covered four server types: database, web, application, and utility. Every one of them got the same four security layers, regardless of role.

The four layers

JASS — the Solaris Security Toolkit, formerly JumpStart Architecture and Security Scripts. It automates the heavy lifting: disabling unnecessary services, tightening file permissions, enforcing password policies. It ships with a library of scripts and a driver file that controls which ones run.

BSM — the Basic Security Module. Solaris’s built-in auditing framework. It logs system calls, tracks who did what, and gives you a trail you can actually follow when something goes wrong.

RSA — two-factor authentication. Passwords alone aren’t enough when you’re dealing with production systems that touch sensitive data.

Tripwire — file integrity monitoring. It takes a baseline snapshot of the filesystem, then continuously checks for changes against that baseline.

Solaris Containers (zones) got the same treatment. The security mechanisms run inside the zones, not just on the global zone.

JASS: the automation layer

JASS is the workhorse. It ships with a secure.driver template that controls which hardening scripts execute. The default template disables a lot of services — sendmail, SAMBA, SNMP, routing — which is sensible for most servers. But every environment is different, so we tweaked the driver file.

Some scripts we left enabled that the default disables. disable-keyboard-abort.fin, for instance — we kept that one on because someone accidentally hitting Ctrl-Alt-Delete on a console shouldn’t be able to reboot a production box. print-rhosts.fin too — we wanted visibility into any rhosts files that snuck in, not just silence.

Scripts we disabled: enable-ftpaccess.fin (FTP was blocked at the IPFilter level anyway), enable-ipfilter.fin (IPFilter was configured separately), enable-process-accounting.fin (not useful for our workload profile), and enable-bart.fin (Berkely Audit Reporting Tool — we had Tripwire covering that ground).

The scripts we kept on did the real work:

  • enable-account-lockout.fin — lock accounts after repeated failed logins
  • enable-coreadm.fin — enable core dump auditing
  • enable-ftp-syslog.fin — log FTP activity to syslog
  • enable-inetd-syslog.fin — log all incoming inetd connections
  • enable-password-history.fin — prevent password reuse
  • enable-priv-nfs-ports.fin — NFS only accepts connections from privileged ports (below 1024)
  • enable-rfc1948.fin — unique-per-connection ID sequence numbers to prevent TCP sequence prediction attacks
  • enable-stack-protection.fin — stack and logging protection
  • enable-tcpwrappers.fin — hosts.allow/hosts.deny for all TCP connections
  • enable-bsm.fin — enable the BSM auditing subsystem
  • install-strong-permissions.fin — tighten file permissions across the system

The JASS scripts directory contains over a hundred .fin files. Most of them are standard hardening routines — setting umask values, restricting root login, configuring banners, locking down system accounts. We reviewed each one and decided whether it made sense for our environment.

BSM: auditing what matters

BSM is Solaris’s answer to “who did what, when, and from where.” We configured it to log locally to /var/audit and also forward to a remote syslog server for centralised collection.

The audit control file (/etc/security/audit_control) looked like this:

dir:/var/audit
flags:lo,ex
minfree:20
naflags:lo,ex
plugin: name=audit_syslog.so;p_flags=lo,ex

The startup sequence in /etc/rc2.d/ ran:

/usr/sbin/auditconfig -setpolicy +cnt
/usr/sbin/auditconfig -setpolicy +zonename
/usr/sbin/auditconfig -setpolicy +argv
/usr/sbin/auditconfig -conf
/usr/sbin/auditconfig -aconf

The +cnt policy adds the connection number, +zonename tags events with the zone they came from, and +argv captures the full command line. That last one is expensive in terms of disk space but invaluable for forensics.

Syslog forwarding was configured in /etc/syslog.conf:

audit.notice;auth.debug @dc5sfsecsyslog01

RSA: two-factor authentication

RSA authentication was the only method accepted on the Solaris boxes. The deployment steps were straightforward:

  1. Pull the latest sdconf.rec file from the RSA Security Console
  2. Download the RSA Agent software for Solaris
  3. Copy sdconf.rec into /var/ace
  4. Unzip and run the installer script
  5. Accept the license and hit enter

That’s it. The RSA agent handles the challenge-response handshake, and the PAM configuration on Solaris routes authentication through it. Passwords alone never reached the system.

Tripwire: file integrity monitoring

Tripwire Enterprise Agent (version 7.1.0 for Intel builds) was our file integrity tool. It works by taking a baseline snapshot of the filesystem using a ruleset that defines which directories and files to monitor. Every subsequent scan compares the current state against that baseline and flags any changes.

Installation involved copying the package from the central repository (tomcat:/backup, with SPARC and x86 variants), extracting it on the target server, running the package installer, and starting the agent. The agent then runs on a schedule, reporting deviations back to the Tripwire Enterprise console.

Solaris Containers (Zones)

Zones are Solaris’s lightweight virtualisation. Each zone is an isolated instance of the OS — processes can’t see into other zones, network traffic is segregated, and the filesystem is partitioned.

There are two zone types: global (the host OS) and non-global (the containers). Non-global zones come in two flavours: whole root (full copy of the OS packages) and sparse root (shares the global zone’s root filesystem, only gets what’s explicitly added).

We ran most applications in non-global zones. Databases stayed in the global zone — the overhead of zone isolation wasn’t worth the marginal security gain for those workloads.

Hardening zones followed the same pattern regardless of network configuration:

  • JASS ran on the global zone and every non-global zone
  • RSA ran on the global zone and every non-global zone
  • BSM ran only on the global zone (audit logs from zones still flow through the global zone’s audit subsystem)

Zone configuration files are XML. Here’s what a sparse zone with a shared network interface looks like:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE zone PUBLIC "-//Sun Microsystems Inc//DTD Zones//EN"
  "file:///usr/share/lib/xml/dtd/zonecfg.dtd.1">
<!--
  DO NOT EDIT THIS FILE. Use zonecfg(1M) instead.
-->
<zone name="dc5sfapp01a" zonepath="/zones/dc5sfapp01a" autoboot="true">
  <inherited-pkg-dir directory="/lib"/>
  <inherited-pkg-dir directory="/platform"/>
  <inherited-pkg-dir directory="/sbin"/>
  <inherited-pkg-dir directory="/usr"/>
  <inherited-pkg-dir directory="/usr/sfw"/>
  <inherited-pkg-dir directory="/export/home/apache-ant-1.6.1"/>
  <inherited-pkg-dir directory="/export/home/jdk1.6.0_13"/>
  <network address="10.5.30.14" physical="nge0"/>
</zone>

A whole root zone with an exclusive network interface is simpler — no inherited package directories, just a network binding:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE zone PUBLIC "-//Sun Microsystems Inc//DTD Zones//EN"
  "file:///usr/share/lib/xml/dtd/zonecfg.dtd.1">
<zone name="dc5sfsmnsageweb01" zonepath="/zones/dc5sfsmnsageweb01"
      autoboot="true" ip-type="exclusive">
  <network address="" physical="nge2"/>
</zone>

The ip-type="exclusive" is the key difference. With exclusive networking, the zone gets its own IP stack — it can run its own IPFilter rules, configure its own routing, and the global zone can’t interfere. That’s a meaningful security boundary.

Password policy

JASS’s set-user-password-reqs.fin script enforces password complexity and rotation. It modifies /etc/default/passwd to set minimum and maximum password age, warning period, and minimum password length. The values come from JASS environment variables (JASS_AGING_MINWEEKS, JASS_AGING_MAXWEEKS, JASS_AGING_WARNWEEKS, JASS_PASS_LENGTH), so you can tune them per environment without editing the script itself.

The script backs up the original file before making changes, which is the kind of detail that matters when you’re running this across dozens of servers.

References

  • ISO/IEC 17799 — Information Technology — Security Techniques — Code of Practice for Information Security Management