Having problem during AIX Server Assessment/Hardening recently. After a few googles,found this great blog
http://www.cymru.comhelps a lot in my assessment.
1. TCP send and receive spaces
The TCP send and receive spaces directly effect the TCP window size parameter. An increased window size will allow for more efficient transfers, particularly bulk transfers such as FTP and HTTP. The default for each is not optimal, and should be increased to 32768 bytes. This value should not be increased above 64K bytes unless the implications of RFC1323 and RFC2018 are fully understood and support for both is enabled.
Do not enable RFC1323 without also enabling support for RFC2018. Remember, pipe drain is a Bad Thing[tm].
A. AIX
/usr/sbin/no -o tcp_sendspace=32768
/usr/sbin/no -o tcp_recvspace=32768
2. Socket queue defense against SYN attacks
While great effort is undertaken to defend any network from those with malicious intent, several ports (largely TCP) must remain open to conduct business. Internet vandals may attempt to exploit these ports to launch a denial of service attack. One of the most popular attacks remains the SYN flood, wherein the socket queue of the attacked host is overwhelmed with bogus connection requests. To defend against such attacks, certain UNIX variants maintain separate queues for inbound socket connection requests. One queue is for half-open sockets (SYN received, SYN|ACK sent), the other queue for fully-open sockets awaiting an accept() call from the application. These two queues should be increased so that an attack of low to moderate intensity will have little to no effect on the stability or availability of the server.
A. AIX
/usr/sbin/no -o clean_partial_conns=1
This setting will instruct the kernel to randomly remove half-open sockets from the q0 queue to make room for new sockets.
3. Redirects
A miscreant can use IP redirects to modify the routing table on a remote host. In a well-designed network, redirects to the end stations should not be required. Both the sending and accepting of redirects should be disabled.
A. AIX
/usr/sbin/no -o ipignoreredirects=1
/usr/sbin/no -o ipsendredirects=0
4. ARP cleanup
It is possible for a miscreant to create a resource exhaustion or performance degredation by filling the IP route cache with bogus ARP entries. In Solaris, there are two parameters that govern the cleanup interval for the IP route cache. For unsolicited ARP responses, the parameter to be tuned is arp_cleanup_interval. In AIX, the cleanup interval is governed by the value of arpt_killc. However, this parameter governs both solicited and unsolicited ARP entries. For this reason, it is likely best to leave the parameter at the default setting of 20 minutes.
A. AIX
/usr/sbin/no -o arpt_killc=20
5. Source routing
With source routing, an attacker can attempt to reach internal IP addresses - including RFC1918 addresses. It is important to disable the acceptance of source routed packets to prevent subtle probes of your internal networks.
A. AIX
/usr/sbin/no -o ipsrcroutesend=0
Disable the sending of source routed packets.
/usr/sbin/no -o ipsrcrouteforward=0
This is important if the box is routing, e.g. a firewall. Disable this feature to prevent the host from forwarding source routed packets.
6. TIME_WAIT setting
On a busy web server, many sockets may linger in the TIME_WAIT state. This is caused by improperly coded client applications that do not properly shut down a socket. This can also be used as a type of DDoS attack.
A. AIX
No tuning recommendations.
7. Broadcast ECHO response
Smurf attacks work by sending ICMP 8 0 (ECHO REQUEST) messages to a broadcast address from a spoofed address. Some IP stacks will respond, by default, to such messages. This should be disabled. Further, if the host is a firewall (router), it should not propogate directed broadcasts.
A. AIX
/usr/sbin/no -o directed_broadcast=0
Do not respond to directed broadcasts.
8. Other broadcast probes
There are two other broadcast probes that a miscreant could utilize against a network. The address mask query can be used to map out the size of the netblock, and set a range for further probes. The timestamp broadcast is another means of mapping and fingerprinting hosts.
A. AIX
/usr/sbin/no -o icmpaddressmask=0
Prevent address mask queries.
other references that might helps during AIX Assessment/Hardening
http://abstractinitiative.com/PerformanceJungle/2009/09/aix-security-hardening-script/
http://publib.boulder.ibm.com
http://www.boran.com/security/sp/aix_hardening.html
http://www.ncftp.com/ncftpd/doc/misc/ephemeral_ports.html
http://slashzeroconf.wordpress.com/2008/01/20/unix-password-selection-expiration/