If you are running more than one server on the same machine, Apache offers the choice of either having a separate set of log files for each server, or using a central set of log files for all servers. If you are running servers on more than one machine, having them share a single log file is harder to achieve, but it is possible, provided that a filesharing system is used (logging into a database, or a special purpose application like syslog).

There are a few file-sharing systems that are widely used:

Network File System (NFS)
NFS is a network file-sharing system. It's a very useful system, when it works. Unfortunately, it breaks too often, which makes it unreliable to use on production systems. NFS is available on most Unix flavors.

Andrew File System (AFS)
AFS is a distributed filesystem that enables cooperating hosts (clients and servers) to efficiently share filesystem resources across both local area and wide area networks. This filesystem is reliable, but it costs money and is available only on the HP, Next, DEC, IBM, SUN, and SGI operating systems. For more information, see http://www.transarc.com/ and http://www.angelfire.com/hi/plutonic/afs-faq.html.

Coda
Coda is a distributed filesystem with its origin in AFS2. It has many features that are very desirable for network filesystems. Coda is platform-independent: you can mix and match servers and clients on any supported platform. As of this writing, it's not clear how stable the system is; some people have reported success using it, but others have had some problems with it. For more information, see http://www.coda.cs.cmu.edu/.

Apache permits the location of the file used for logging purposes to be specified, but it also allows you to specify a program to which all logs should be piped. To log to a program, modify the log handler directive (for example, CustomLog) to use the logging program instead of specifying an explicit filename:

LogFormat "%h %l %u %t \"%r\" %>s %b" common
CustomLog "| /home/httpd/httpd_perl/bin/sqllogger.pl" common

Logging into a database is a common solution, because you can do insertions from different machines into a single database. Unless the logger is programmed to send logs to a few databases at once, this solution is not reliable, since a single database constitutes a single failure point. If the database goes down, the logs will be lost. Sending information to one target is called unicast (see Figure 5-6), and sending to more than one target is called multicast (see Figure 5-7). In the latter case, if one database goes down, the others will still collect the data.

Figure 5-6

Figure 5-6. Unicast solution

 Figure 5-7

Figure 5-7. Multicast solution

Another solution is to use a centralized logger program based on syslog(3) to send all logs to a central location on a master host. syslog(3) is not a very scalable solution, because it's slow. It's also unreliable—it uses UDP to send the data, which doesn't ensure that the data will reach its destination. This solution is also unicast: if the master host goes down, the logs will be lost.

Using syslog

The syslog solution can be implemented using the following configuration:

LogFormat "%h %l %u %t \"%r\" %>s %b" common
CustomLog "| /home/httpd/httpd_perl/bin/syslogger.pl hostnameX" common

where a simple syslogger.pl can look like this:

#!/usr/bin/perl
use Sys::Syslog qw(:DEFAULT setlogsock);

my $hostname = shift || 'localhost';
my $options  = 'ndelay'; # open the connection immediately
my $facility = 'local0'; # one of local0..local7
my $priority = 'info';   # debug|info|notice|warning|err...

setlogsock 'unix';
openlog $hostname, $options, $facility;
while (<>) {
    chomp;
    syslog $priority, $_;
}
closelog;

The syslog utility needs to know the facility to work with and the logging level. We will use local0, one of the special logging facilities reserved for local usage (eight local facilities are available: local0 through local7). We will use the info priority level (again, one of eight possible levels: debug, info, notice, warning, err, crit, alert, and emerg).

Now make the syslog utility on the master machine (where all logs are to be stored) log all messages coming from facility local0 with logging level info to a file of your choice. This is achieved by editing the /etc/syslog.conf file. For example:

local0.info /var/log/web/access_log

All other machines forward their logs from facility local0 to the central machine. Therefore, on all but the master machine, we add the forwarding directive to the /etc/syslog.conf file (assuming that the master machine's hostname is masterhost):

local0.info @masterhost

We must restart the syslogd daemon or send it the HUP kill signal for the changes to take effect before the logger can be used.

One advanced system that provides consolidated logging is mod_log_spread. Based on the group communications toolkit Spread, using IP multicast, mod_log_spread provides reliable, scalable centralized logging whith minimal performance impact on the web servers. For more information, see http://www.backhand.org/mod_log_spread/.