SubversionWiki - User contributions [en]

Server performance tuning for Linux and Unix

2008-07-15T20:17:14Z

Dchristian: /* Watch your entropy */

== General notes ==

There are several good web sites and books about how to setup subversion, but I couldn't find anything about how to optimize performance. This is a guide to understanding and improving the performance of subversion when it is served using svnserve or HTTP (via apache).

Much of the operating system tuning is similar to what would be done for a database or e-mail server. It can be useful to search those areas for additional advice http://dev.mysql.com/doc/refman/5.0/en/innodb-configuration.html

These notes have a lot of Linux specific details, but the concepts should apply to most Unix based systems. Feel free to add details for other flavors of Unix here. Non-posix tuning notes should probably go on another page.

The repository can be stored as either a Berkeley DB database or in the FSFS repository formats. This is hidden from users. FSFS is generally considered to be faster.

== Making reads cheaper ==

The Unix concept of "file access time" (commonly call atime) is a performance problem. When filesystem semantics were being defined, it seemed like a good idea to know when a file was last accessed. The down side is that every file open call now causes a disk write. A few
utilities use this information (e.g. tmpwatch and mail), but subversion never uses atime. Subversion performance is improved by avoiding the access time writes.

For a local filesystem, you can disable this behavior with mount options. On Linux, it's the 'noatime' and 'nodiratime' options. On a NFS filesystem, the atime recording happens on the server and must be disabled in the server's configuration.

A lazy atime approach called "relatime" was introduced in Linux-2.6.20 and mount-2.13. This eliminates most atime writes without breaking the few utilities that need it. This is most useful if the repository must be on the same partition as the mail spool and/or temporary
files. See: http://kerneltrap.org/node/14148 and http://kernelnewbies.org/Linux_2_6_20

== Making writes cheaper ==

Subversion uses uses the fsync() call (or the equivalent on non-Unix operating systems) to tell the operating system to write data to disk. Up until that point, the data is usually only memory and the operating system will write it to disk "when it gets around to it".

By calling fsync() before finishing a commit, subversion is trying to guarantee that everything it said had been done would be there when the machine re-boots. Waiting for data to write out to disk is often the slowest part of a commit.

However, the operating system doesn't always hold up its end of the bargain. On Linux, fsync() only ensures that the data is on its way to the disk "as soon as possible". If write cache is enabled on the drive, then it doesn't actually wait for the data to hit the disk platter before returning. This means there is a window of time that a power loss can cause the disk state to not match what subversion returned.

One way to significantly increase fsync() performance is to use a RAID controller with a battery backed write cache. The cache is treated as part of the disk system. As soon as the data is in the cache, the fsync() can safely return. This means you don't have to wait for the
disk head seek or the data transfer. If power is interrupted, the RAID controller will finish writing out the cache when power is restored.

A newer way to avoid this problem is a flash based disk. There is no latency from head movement or waiting for the disk to rotate. This becomes more significant when writing many small files (like many FSFS writes). The current downsides of flash disks are high cost, limited
capacity, and low write bandwidth (but these problems are improving).

== Reducing the number of writes ==

As of subversion-1.5, transactions can be built up on a different filesystem than the one holding the repository. This is valuable when the repository lives on a slower filesystem like NFS.

To implement this, do the following:
stop all servers that can write to the repository
cd REPO_PATH/db
mv transactions /LOCAL/DISK/PATH/
ln -s /LOCAL/DISK/PATH/transactions .
start the servers

== Reduce directory index size ==

The subversion-1.5 repository format allows the revisions to be stored in subdirectories that don't grow past a specified size. This allows repositories to store many more revisions than can (efficiently) be stored in one directory.

Modern filesystems can handle hundreds of thousands of files in a single directory. However, performance can suffer as the directory index starts to use multiple levels of indirection. Some administration tools may also have trouble with very large directories. Splitting the revision store into sub-directories avoids all these problems.

The shard size can by adjusted by editing the "layout sharded" line in "db/format" after 'svnadmin create' but before populating the repository. The default is 1000 revisions per subdirectory. Non-sharded repositories can be loaded into a new, sharded, repository using "svnadmin load" or "svnsync".

== Optimize write-once files on NFS ==

If the repository is on a NFS filesystem, then a cache consistency check is made every time a file is opened. Since the revision files in a FSFS repository never change, it is worthwhile to skip the cache checks on these files. The subversion-1.5 repository format store
immutable files in specific subdirectories so that this can be done.

The NFS cache check can be disabled on Linux by passing the 'nocto' option to the mount command (note: the man page claims this is ignored, but it isn't on linux-2.6). You need coherency for some files, so the NFS volume is also mounted without the option on a
different mount point. Symbolic links are made from the cache coherent mount point to the 'nocto' mount for these directories: revs and txn-protorevs.

Implementation example (not complete, just an outline of the key steps):
stop all servers that can write to the repository
sudo mount -t nfs nfs_server:/mount_point /mnt/svn -o \
rw,nosuid,tcp,rsize=32768,wsize=32768
sudo mount -t nfs nfs_server:/mount_point /mnt/svn-nocto -o \
rw,nosuid,tcp,rsize=32768,wsize=32768,nocto,actimeo=3600
cd /mnt/svn/repo_path
mv revs revs-nocto
mv txn-protorevs txn-protorevs-nocto
ln -s /mnt/svn-nocto/repo_path/db/revs-nocto revs
ln -s /mnt/svn-nocto/repo_path/db/txn-protorevs-nocto txn-protorevs
start the servers

== Increase NFS caching timeout ==

On Linux, metadata on files from NFS is only kept for a finite period of time. This can be changed by passing the actimeo option to the mount command. The man page claims the default is 60 (seconds), but some experimentation suggests it may be higher than that. For a
'nocto' mount point, this value can be raised to something much larger (e.g. 3600). See the above example.

== Distributing CPU load ==

The subversion communicates with the clients by transmitting differences in state, so the CPU load to calculate the difference can be significant. By storing the repository on NFS, you can have
multiple "front end" (FE) systems that share the computational load and provide redundancy. A network load balancer makes all front ends (FEs) appear as one server to users.

The FEs can either run svnserve or http-DAV. If DAV is used, you need to ensure that the load balancer keeps an entire transaction on the same FE (to allow transactions to be built up on local disk). The load balancer must be configured with "machine affinity" set, so that
all HTTP connections from a client will be routed to the same server. You should also configure apache to keep a single TCP connection for the entire transaction (see example below).

Apache configuration to maintain a TCP connection:
# 1. Enable HTTP persistent connections so a single transaction can
# be built up over a single connection.
KeepAlive on
# 2. Allow as many KeepAlives as required (0 => infinite) to keep
# the same connection alive.
MaxKeepAliveRequests 0
# 3. Limit a child to serving only this 1 connection.
MaxRequestsPerChild 1

The last one is counter-intuitive, but see the "Note" at http://httpd.apache.org/docs/2.2/mod/mpm_common.html#maxrequestsperchild.

== High storage system reliability ==

The purpose of a version control system is to store a sequence of file/directory versions so you can retreive them in the future. None of this matters if the storage system fails.

The simplest step is to do periodic backups of the repository. This limits the loss to the changes that happened since the last backup. If the repository is large and the commit rate is high, it may be impossible to backup frequently enough to prevent significant data
loss. For example, if your repository gets one commit per second and you do a backup every hour, you may lose 3600 revisions if the disk fails. This is a large scale example, but the point is to gather your own numbers and figure out how much you might lose.

The next step is to make the disk system redundant using RAID technology. This allows one (and sometimes more) disks to fail without losing data. This still won't help if additional disks fail
during recovery or the entire array is lost due to fire, theft, etc.

Advanced NFS servers can be configured to do synchronous mirroring and/or asynchronous mirroring (also known as snapshot replication).
These capabilities are available in some commercial servers, or you can find various free alternatives by searching for "NFS server high availability" or "NFS server snapshot replication".

Synchronous mirroring sends every write to two independent storage systems and requires a high bandwidth network (e.g. gigabit ethernet). It reduces performance, but the caching optimizations listed above can help. The primary and slave systems are usually located in different
rooms (or buildings) and on different electrical circuits.

Asynchronous mirroring periodically updates a second storage system with changes from the master. It periodically makes a "snapshot" of the system every few minutes and then transmits the difference between the previous snapshot and the current one to the slave. This uses
less bandwidth, but lags the main filesystem by a several minutes. It can allow a backup filesystem to be located in another geographic region.

Another approach is to use subversion tools to maintain a mirror. Setup svnsync to periodically sync a back up server off the main one. This can lag behind by the polling interval, but it is simple to setup.

You can eliminate the lag by setting up a post-commit script that runs "svnadmin dump --incremental -r N" of that commit onto a separate partition/server. This creates a transaction log of commits that can be replayed on a recent backup to restore full state.

== Watch your entropy ==

When servers handled lots of queries, certain protocols can deplete the [http://en.wikipedia.org/wiki/Entropy_(computing) entropy] [http://en.wikipedia.org/wiki/Urandom pool]. The svn:// protocol (served by svnserve) and [http://en.wikipedia.org/wiki/Simple_Authentication_and_Security_Layer SASL] cyphers read from /dev/random for every new connection. If the entropy pool becomes depleted, then the service will become very slow.

The pool should have 100+ bits in it for good operation. You can check the entropy pool size on linux like this:
sysctl kernel.random.entropy_avail

This should not be a problem if APR was configured with "--with-devrandom=/dev/urandom". Sasl has a similar configuration option (called???). You may need to check how the packages supplied with you OS are configured.

(TODO: reference the SVN benchmarking capabilities in mstone http://mstone.sourceforge.net/)

[[User:Dchristian|Dchristian]] 13:42, 30 May 2008 (PDT)

Server performance tuning for Linux and Unix

2008-07-15T18:22:51Z

Dchristian:

== General notes ==

There are several good web sites and books about how to setup subversion, but I couldn't find anything about how to optimize performance. This is a guide to understanding and improving the performance of subversion when it is served using svnserve or HTTP (via apache).

Much of the operating system tuning is similar to what would be done for a database or e-mail server. It can be useful to search those areas for additional advice http://dev.mysql.com/doc/refman/5.0/en/innodb-configuration.html

These notes have a lot of Linux specific details, but the concepts should apply to most Unix based systems. Feel free to add details for other flavors of Unix here. Non-posix tuning notes should probably go on another page.

The repository can be stored as either a Berkeley DB database or in the FSFS repository formats. This is hidden from users. FSFS is generally considered to be faster.

== Making reads cheaper ==

The Unix concept of "file access time" (commonly call atime) is a performance problem. When filesystem semantics were being defined, it seemed like a good idea to know when a file was last accessed. The down side is that every file open call now causes a disk write. A few
utilities use this information (e.g. tmpwatch and mail), but subversion never uses atime. Subversion performance is improved by avoiding the access time writes.

For a local filesystem, you can disable this behavior with mount options. On Linux, it's the 'noatime' and 'nodiratime' options. On a NFS filesystem, the atime recording happens on the server and must be disabled in the server's configuration.

A lazy atime approach called "relatime" was introduced in Linux-2.6.20 and mount-2.13. This eliminates most atime writes without breaking the few utilities that need it. This is most useful if the repository must be on the same partition as the mail spool and/or temporary
files. See: http://kerneltrap.org/node/14148 and http://kernelnewbies.org/Linux_2_6_20

== Making writes cheaper ==

Subversion uses uses the fsync() call (or the equivalent on non-Unix operating systems) to tell the operating system to write data to disk. Up until that point, the data is usually only memory and the operating system will write it to disk "when it gets around to it".

By calling fsync() before finishing a commit, subversion is trying to guarantee that everything it said had been done would be there when the machine re-boots. Waiting for data to write out to disk is often the slowest part of a commit.

However, the operating system doesn't always hold up its end of the bargain. On Linux, fsync() only ensures that the data is on its way to the disk "as soon as possible". If write cache is enabled on the drive, then it doesn't actually wait for the data to hit the disk platter before returning. This means there is a window of time that a power loss can cause the disk state to not match what subversion returned.

One way to significantly increase fsync() performance is to use a RAID controller with a battery backed write cache. The cache is treated as part of the disk system. As soon as the data is in the cache, the fsync() can safely return. This means you don't have to wait for the
disk head seek or the data transfer. If power is interrupted, the RAID controller will finish writing out the cache when power is restored.

A newer way to avoid this problem is a flash based disk. There is no latency from head movement or waiting for the disk to rotate. This becomes more significant when writing many small files (like many FSFS writes). The current downsides of flash disks are high cost, limited
capacity, and low write bandwidth (but these problems are improving).

== Reducing the number of writes ==

As of subversion-1.5, transactions can be built up on a different filesystem than the one holding the repository. This is valuable when the repository lives on a slower filesystem like NFS.

To implement this, do the following:
stop all servers that can write to the repository
cd REPO_PATH/db
mv transactions /LOCAL/DISK/PATH/
ln -s /LOCAL/DISK/PATH/transactions .
start the servers

== Reduce directory index size ==

The subversion-1.5 repository format allows the revisions to be stored in subdirectories that don't grow past a specified size. This allows repositories to store many more revisions than can (efficiently) be stored in one directory.

Modern filesystems can handle hundreds of thousands of files in a single directory. However, performance can suffer as the directory index starts to use multiple levels of indirection. Some administration tools may also have trouble with very large directories. Splitting the revision store into sub-directories avoids all these problems.

The shard size can by adjusted by editing the "layout sharded" line in "db/format" after 'svnadmin create' but before populating the repository. The default is 1000 revisions per subdirectory. Non-sharded repositories can be loaded into a new, sharded, repository using "svnadmin load" or "svnsync".

== Optimize write-once files on NFS ==

If the repository is on a NFS filesystem, then a cache consistency check is made every time a file is opened. Since the revision files in a FSFS repository never change, it is worthwhile to skip the cache checks on these files. The subversion-1.5 repository format store
immutable files in specific subdirectories so that this can be done.

The NFS cache check can be disabled on Linux by passing the 'nocto' option to the mount command (note: the man page claims this is ignored, but it isn't on linux-2.6). You need coherency for some files, so the NFS volume is also mounted without the option on a
different mount point. Symbolic links are made from the cache coherent mount point to the 'nocto' mount for these directories: revs and txn-protorevs.

Implementation example (not complete, just an outline of the key steps):
stop all servers that can write to the repository
sudo mount -t nfs nfs_server:/mount_point /mnt/svn -o \
rw,nosuid,tcp,rsize=32768,wsize=32768
sudo mount -t nfs nfs_server:/mount_point /mnt/svn-nocto -o \
rw,nosuid,tcp,rsize=32768,wsize=32768,nocto,actimeo=3600
cd /mnt/svn/repo_path
mv revs revs-nocto
mv txn-protorevs txn-protorevs-nocto
ln -s /mnt/svn-nocto/repo_path/db/revs-nocto revs
ln -s /mnt/svn-nocto/repo_path/db/txn-protorevs-nocto txn-protorevs
start the servers

== Increase NFS caching timeout ==

On Linux, metadata on files from NFS is only kept for a finite period of time. This can be changed by passing the actimeo option to the mount command. The man page claims the default is 60 (seconds), but some experimentation suggests it may be higher than that. For a
'nocto' mount point, this value can be raised to something much larger (e.g. 3600). See the above example.

== Distributing CPU load ==

The subversion communicates with the clients by transmitting differences in state, so the CPU load to calculate the difference can be significant. By storing the repository on NFS, you can have
multiple "front end" (FE) systems that share the computational load and provide redundancy. A network load balancer makes all front ends (FEs) appear as one server to users.

The FEs can either run svnserve or http-DAV. If DAV is used, you need to ensure that the load balancer keeps an entire transaction on the same FE (to allow transactions to be built up on local disk). The load balancer must be configured with "machine affinity" set, so that
all HTTP connections from a client will be routed to the same server. You should also configure apache to keep a single TCP connection for the entire transaction (see example below).

Apache configuration to maintain a TCP connection:
# 1. Enable HTTP persistent connections so a single transaction can
# be built up over a single connection.
KeepAlive on
# 2. Allow as many KeepAlives as required (0 => infinite) to keep
# the same connection alive.
MaxKeepAliveRequests 0
# 3. Limit a child to serving only this 1 connection.
MaxRequestsPerChild 1

The last one is counter-intuitive, but see the "Note" at http://httpd.apache.org/docs/2.2/mod/mpm_common.html#maxrequestsperchild.

== High storage system reliability ==

The purpose of a version control system is to store a sequence of file/directory versions so you can retreive them in the future. None of this matters if the storage system fails.

The simplest step is to do periodic backups of the repository. This limits the loss to the changes that happened since the last backup. If the repository is large and the commit rate is high, it may be impossible to backup frequently enough to prevent significant data
loss. For example, if your repository gets one commit per second and you do a backup every hour, you may lose 3600 revisions if the disk fails. This is a large scale example, but the point is to gather your own numbers and figure out how much you might lose.

The next step is to make the disk system redundant using RAID technology. This allows one (and sometimes more) disks to fail without losing data. This still won't help if additional disks fail
during recovery or the entire array is lost due to fire, theft, etc.

Advanced NFS servers can be configured to do synchronous mirroring and/or asynchronous mirroring (also known as snapshot replication).
These capabilities are available in some commercial servers, or you can find various free alternatives by searching for "NFS server high availability" or "NFS server snapshot replication".

Synchronous mirroring sends every write to two independent storage systems and requires a high bandwidth network (e.g. gigabit ethernet). It reduces performance, but the caching optimizations listed above can help. The primary and slave systems are usually located in different
rooms (or buildings) and on different electrical circuits.

Asynchronous mirroring periodically updates a second storage system with changes from the master. It periodically makes a "snapshot" of the system every few minutes and then transmits the difference between the previous snapshot and the current one to the slave. This uses
less bandwidth, but lags the main filesystem by a several minutes. It can allow a backup filesystem to be located in another geographic region.

Another approach is to use subversion tools to maintain a mirror. Setup svnsync to periodically sync a back up server off the main one. This can lag behind by the polling interval, but it is simple to setup.

You can eliminate the lag by setting up a post-commit script that runs "svnadmin dump --incremental -r N" of that commit onto a separate partition/server. This creates a transaction log of commits that can be replayed on a recent backup to restore full state.

== Watch your entropy ==

When servers handled lots of queries, certain protocols can deplete the [http://en.wikipedia.org/wiki/Entropy_(computing) entropy] [http://en.wikipedia.org/wiki/Urandom pool]. The svn:// protocol (served by svnserve) and [http://en.wikipedia.org/wiki/Simple_Authentication_and_Security_Layer SASL] cyphers read from /dev/random for every new connection. If the entropy pool becomes depleted, then the service will become very slow.

The pool should have 100+ bits in it for good operation. You can check the entropy pool size on linux like this:
sysctl kernel.random.entropy_avail

There are patches pending (as of subversion 1.5.0 and apr 1.2.2) to improve this (read from urandom instead of random). TODO: include bug numbers and track when these ship.

(TODO: reference the SVN benchmarking capabilities in mstone http://mstone.sourceforge.net/)

[[User:Dchristian|Dchristian]] 13:42, 30 May 2008 (PDT)

Main Page

2008-06-04T18:10:32Z

Dchristian: /* HowTos */

<big>'''Welcome to the Subversion Wiki.'''</big>

== Got a question about Subversion? ==

Since this wiki seems to lack content at the moment, try the [http://subversion.tigris.org/faq.html Subversion FAQ].

Also, there is [http://subversionary.org another wiki] related to Subversion. At this time, it contains more content than this wiki. However unlike this wiki, it doesn't use the more popular MediaWiki engine, and apprently, due to spam problems, was forced to shut off public editing.

== HowTos ==

* Manage [[System Configurations]] like <tt>/etc</tt> files using subversion.
* Setting up [[Subversion configured for Windows Active Directory HTTPS]]
* Setting up [[Automatic lock-modify-unlock]] for binary files
* How to do [[Server performance tuning for Linux and Unix]]

== Contrib tools ==

* [[svnmerge.py]] - Automatic branch management with merge tracking support.
* [[Repository Upgrade]] - Upgrade from an older repository to latest (1.4) (Windows)
* [[Subclipse]] - The subversion plugin for Eclipse
* [[SVNKit]] - Pure Java Subversion library
* [http://svnnotifier.tigris.org SVN Notifier] - Notifies you about other people's commits to subversion

== MediaWiki help ==

This wiki uses the MediaWiki software. Consult the [http://meta.wikipedia.org/wiki/MediaWiki_User's_Guide User's Guide] for information on using the wiki software.

=== Getting started ===

* [http://www.mediawiki.org/wiki/Help:Configuration_settings Configuration settings list]
* [http://www.mediawiki.org/wiki/Help:FAQ MediaWiki FAQ]
* [http://mail.wikipedia.org/mailman/listinfo/mediawiki-announce MediaWiki release mailing list]
* [http://alfredfazio.ws/notes%3Asubversion Fast-paced overview of Subversion for experienced users]

Server performance tuning for Linux and Unix

2008-06-04T18:06:55Z

Dchristian:

== General notes ==

There are several good web sites and books about how to setup subversion, but I couldn't find anything about how to optimize performance. This is a guide to understanding and improving the performance of subversion when it is served using svnserve or HTTP (via apache).

Much of the operating system tuning is similar to what would be done for a database or e-mail server. It can be useful to search those areas for additional advice http://dev.mysql.com/doc/refman/5.0/en/innodb-configuration.html

These notes have a lot of Linux specific details, but the concepts should apply to most Unix based systems. Feel free to add details for other flavors of Unix here. Non-posix tuning notes should probably go on another page.

The repository can be stored as either a Berkeley DB database or in the FSFS repository formats. This is hidden from users. FSFS is generally considered to be faster.

== Making reads cheaper ==

The Unix concept of "file access time" (commonly call atime) is a performance problem. When filesystem semantics were being defined, it seemed like a good idea to know when a file was last accessed. The down side is that every file open call now causes a disk write. A few
utilities use this information (e.g. tmpwatch and mail), but subversion never uses atime. Subversion performance is improved by avoiding the access time writes.

For a local filesystem, you can disable this behavior with mount options. On Linux, it's the 'noatime' and 'nodiratime' options. On a NFS filesystem, the atime recording happens on the server and must be disabled in the server's configuration.

A lazy atime approach called "relatime" was introduced in Linux-2.6.20 and mount-2.13. This eliminates most atime writes without breaking the few utilities that need it. This is most useful if the repository must be on the same partition as the mail spool and/or temporary
files. See: http://kerneltrap.org/node/14148 and http://kernelnewbies.org/Linux_2_6_20

== Making writes cheaper ==

Subversion uses uses the fsync() call (or the equivalent on non-Unix operating systems) to tell the operating system to write data to disk. Up until that point, the data is usually only memory and the operating system will write it to disk "when it gets around to it".

By calling fsync() before finishing a commit, subversion is trying to guarantee that everything it said had been done would be there when the machine re-boots. Waiting for data to write out to disk is often the slowest part of a commit.

However, the operating system doesn't always hold up its end of the bargain. On Linux, fsync() only ensures that the data is on its way to the disk "as soon as possible". If write cache is enabled on the drive, then it doesn't actually wait for the data to hit the disk platter before returning. This means there is a window of time that a power loss can cause the disk state to not match what subversion returned.

One way to significantly increase fsync() performance is to use a RAID controller with a battery backed write cache. The cache is treated as part of the disk system. As soon as the data is in the cache, the fsync() can safely return. This means you don't have to wait for the
disk head seek or the data transfer. If power is interrupted, the RAID controller will finish writing out the cache when power is restored.

A newer way to avoid this problem is a flash based disk. There is no latency from head movement or waiting for the disk to rotate. This becomes more significant when writing many small files (like many FSFS writes). The current downsides of flash disks are high cost, limited
capacity, and low write bandwidth (but these problems are improving).

== Reducing the number of writes ==

As of subversion-1.5, transactions can be built up on a different filesystem than the one holding the repository. This is valuable when the repository lives on a slower filesystem like NFS.

To implement this, do the following:
stop all servers that can write to the repository
cd REPO_PATH/db
mv transactions /LOCAL/DISK/PATH/
ln -s /LOCAL/DISK/PATH/transactions .
start the servers

== Reduce directory index size ==

The subversion-1.5 repository format allows the revisions to be stored in subdirectories that don't grow past a specified size. This allows repositories to store many more revisions than can (efficiently) be stored in one directory.

Modern filesystems can handle hundreds of thousands of files in a single directory. However, performance can suffer as the directory index starts to use multiple levels of indirection. Some administration tools may also have trouble with very large directories. Splitting the revision store into sub-directories avoids all these problems.

The shard size can by adjusted by editing the "layout sharded" line in "db/format" after 'svnadmin create' but before populating the repository. The default is 1000 revisions per subdirectory. Non-sharded repositories can be loaded into a new, sharded, repository using "svnadmin load" or "svnsync".

== Optimize write-once files on NFS ==

If the repository is on a NFS filesystem, then a cache consistency check is made every time a file is opened. Since the revision files in a FSFS repository never change, it is worthwhile to skip the cache checks on these files. The subversion-1.5 repository format store
immutable files in specific subdirectories so that this can be done.

The NFS cache check can be disabled on Linux by passing the 'nocto' option to the mount command (note: the man page claims this is ignored, but it isn't on linux-2.6). You need coherency for some files, so the NFS volume is also mounted without the option on a
different mount point. Symbolic links are made from the cache coherent mount point to the 'nocto' mount for these directories: revs and txn-protorevs.

Implementation example (not complete, just an outline of the key steps):
stop all servers that can write to the repository
sudo mount -t nfs nfs_server:/mount_point /mnt/svn -o \
rw,nosuid,tcp,rsize=32768,wsize=32768
sudo mount -t nfs nfs_server:/mount_point /mnt/svn-nocto -o \
rw,nosuid,tcp,rsize=32768,wsize=32768,nocto,actimeo=3600
cd /mnt/svn/repo_path
mv revs revs-nocto
mv txn-protorevs txn-protorevs-nocto
ln -s /mnt/svn-nocto/repo_path/db/revs-nocto revs
ln -s /mnt/svn-nocto/repo_path/db/txn-protorevs-nocto txn-protorevs
start the servers

== Increase NFS caching timeout ==

On Linux, metadata on files from NFS is only kept for a finite period of time. This can be changed by passing the actimeo option to the mount command. The man page claims the default is 60 (seconds), but some experimentation suggests it may be higher than that. For a
'nocto' mount point, this value can be raised to something much larger (e.g. 3600). See the above example.

== Distributing CPU load ==

The subversion communicates with the clients by transmitting differences in state, so the CPU load to calculate the difference can be significant. By storing the repository on NFS, you can have
multiple "front end" (FE) systems that share the computational load and provide redundancy. A network load balancer makes all front ends (FEs) appear as one server to users.

The FEs can either run svnserve or http-DAV. If DAV is used, you need to ensure that the load balancer keeps an entire transaction on the same FE (to allow transactions to be built up on local disk). The load balancer must be configured with "machine affinity" set, so that
all HTTP connections from a client will be routed to the same server. You should also configure apache to keep a single TCP connection for the entire transaction (see example below).

Apache configuration to maintain a TCP connection:
# 1. Enable HTTP persistent connections so a single transaction can
# be built up over a single connection.
KeepAlive on
# 2. Allow as many KeepAlives as required (0 => infinite) to keep
# the same connection alive.
MaxKeepAliveRequests 0
# 3. Limit a child to serving only this 1 connection.
MaxRequestsPerChild 1

The last one is counter-intuitive, but see the "Note" at http://httpd.apache.org/docs/2.2/mod/mpm_common.html#maxrequestsperchild.

== High storage system reliability ==

The purpose of a version control system is to store a sequence of file/directory versions so you can retreive them in the future. None of this matters if the storage system fails.

The simplest step is to do periodic backups of the repository. This limits the loss to the changes that happened since the last backup. If the repository is large and the commit rate is high, it may be impossible to backup frequently enough to prevent significant data
loss. For example, if your repository gets one commit per second and you do a backup every hour, you may lose 3600 revisions if the disk fails. This is a large scale example, but the point is to gather your own numbers and figure out how much you might lose.

The next step is to make the disk system redundant using RAID technology. This allows one (and sometimes more) disks to fail without losing data. This still won't help if additional disks fail
during recovery or the entire array is lost due to fire, theft, etc.

Advanced NFS servers can be configured to do synchronous mirroring and/or asynchronous mirroring (also known as snapshot replication).
These capabilities are available in some commercial servers, or you can find various free alternatives by searching for "NFS server high availability" or "NFS server snapshot replication".

Synchronous mirroring sends every write to two independent storage systems and requires a high bandwidth network (e.g. gigabit ethernet). It reduces performance, but the caching optimizations listed above can help. The primary and slave systems are usually located in different
rooms (or buildings) and on different electrical circuits.

Asynchronous mirroring periodically updates a second storage system with changes from the master. It periodically makes a "snapshot" of the system every few minutes and then transmits the difference between the previous snapshot and the current one to the slave. This uses
less bandwidth, but lags the main filesystem by a several minutes. It can allow a backup filesystem to be located in another geographic region.

Another approach is to use subversion tools to maintain a mirror. Setup svnsync to periodically sync a back up server off the main one. This can lag behind by the polling interval, but it is simple to setup.

You can eliminate the lag by setting up a post-commit script that runs "svnadmin dump --incremental -r N" of that commit onto a separate partition/server. This creates a transaction log of commits that can be replayed on a recent backup to restore full state.

(TODO: reference the SVN benchmarking capabilites in mstone http://mstone.sourceforge.net/)

[[User:Dchristian|Dchristian]] 13:42, 30 May 2008 (PDT)

User talk:Dchristian

2008-06-04T17:46:26Z

Dchristian: User talk:Dchristian moved to Server performance tuning for Linux and Unix: Make this widely available

#REDIRECT [[Server performance tuning for Linux and Unix]]

Server performance tuning for Linux and Unix

2008-06-04T17:46:26Z

Dchristian: User talk:Dchristian moved to Server performance tuning for Linux and Unix: Make this widely available

= Server performance tuning for Linux/Unix =
[[User:Dchristian|Dchristian]] 13:42, 30 May 2008 (PDT)

== General notes: ==

Most of these tips apply to both the Berkeley DB and the FSFS
repository formats. FSFS is generally considered to be faster.
Database servers have the same kinds of problems and are a good source
for disk and operating system (OS) tuning advice.
http://dev.mysql.com/doc/refman/5.0/en/innodb-configuration.html

These notes have a lot of Linux specific details, but the concepts should apply to most Unix based systems. Feel free to add details for other flavors of Unix here. Non-posix performance notes should probably go on another page.

== Making reads cheaper: ==

The Unix concept of "file access time" (commonly call atime) is a
performance problem. When filesystem sematics were being defined, it
seemed like a good idea to know when a file was last accessed. The
down side is that every file open call now causes a disk write. A few
utilites use this information (e.g. tmpwatch and mail), but subversion
never uses atime. Subversion performance is improved by avoiding the
access time writes.

For a local filesystem, you can disable this behavior with mount
options. On linux, it's the 'noatime' and 'nodiratime' options. On a
NFS filesystem, the atime recording happens on the server and must be
disabled in the server's configuration.

A lazy atime approach called "relatime" was introduced in Linux-2.6.20
and mount-2.13. This eliminates most atime writes without breaking
the few utilities that need it. This is most useful if the repository
must be on the same partition as the mail spool and/or temporary
files. See: http://kerneltrap.org/node/14148 and
http://kernelnewbies.org/Linux_2_6_20

== Making writes cheaper: ==

Subversion uses uses the fsync() call (or the equivalent on non-Unix
operating systems) to tell the operating system to write data to disk.
Up until that point, the data is usually only memory and the operating
system will write it to disk "when it gets around to it".

By calling fsync() before finishing a commit, subversion is trying to
guarantee that everything it said had been done would be there when
the machine re-boots. Waiting for data to write out to disk is often
the slowest part of a commit.

However, the operating system doesn't always hold up its end of the
bargain. On Linux, fsync() only ensures that the data is on its way
to the disk "as soon as possible". If write cache is enabled on the
drive, then it doesn't actually wait for the data to hit the disk
platter before returning. This means there is a window of time that a
power loss can cause the disk state to not match what subversion
returned.

One way to significantly increase fsync() performance is to use a RAID
controller with a battery backed write cache. The cache is treated as
part of the disk system. As soon as the data is in the cache, the
fsync() can safely return. This means you don't have to wait for the
disk head seek or the data transfer. If power is interrupted, the
RAID controller will finish writing out the cache when power is
restored.

A newer way to avoid this problem is a flash based disk. There is no
latency from head movement or waiting for the disk to rotate. This
becomes more significant when writing many small files (like many FSFS
writes). The current downsides of flash disks are high cost, limited
capacity, and low write bandwidth (but these problems are improving).

== Reducing the number of writes: ==

As of subversion-1.5, transactions can be built up on a different
filesystem than the one holding the repository. This is valuable when
the repository lives on a slower filesystem like NFS.

To implement this, do the following:
stop all servers that can write to the repository
cd REPO_PATH/db
mv transactions /LOCAL/DISK/PATH/
ln -s /LOCAL/DISK/PATH/transactions .
start the servers

== Reduce directory index size: ==

The subversion-1.5 repository format allows the revisions to be stored
in subdirectories that don't grow past a specified size. This allows
repositories to store many more revisions than can (efficiently) be
stored in one directory.

Modern filesystems can handle hundreds of thousands of files in a
single directory. However, performance can suffer as the directory
index starts to use multiple levels of indirection. Some
administration tools may also have trouble with very large
directories. Splitting the revision store into sub-directories avoids
all these problems.

The shard size can by adjusted by editing the "layout sharded" line in
"db/format" after 'svnadmin create' but before populating the
repository. The default is 1000 revisions per subdirectory.
Non-sharded repositories can be loaded into a new, sharded,
repository using "svnadmin load" or "svnsync".

== Make write-once portions of the repository skip NFS cache checks: ==

If the repository is on a NFS filesystem, then a cache consistency
check is made every time a file is opened. Since the revision files
in a FSFS repository never change, it is worthwhile to skip the cache
checks on these files. The subversion-1.5 repository format store
immutable files in specific subdirectories so that this can be done.

The NFS cache check can be disabled on Linux by passing the 'nocto'
option to the mount command (note: the man page claims this is
ignored, but it isn't on linux-2.6). You need coherency for some
files, so the NFS volume is also mounted without the option on a
different mount point. Symbolic links are made from the cache
coherent mount point to the 'nocto' mount for these directories: revs
and txn-protorevs.

Implementation example (not complete, just an outline of the key steps):
stop all servers that can write to the repository
sudo mount -t nfs nfs_server:/mount_point /mnt/svn -o \
rw,nosuid,tcp,rsize=32768,wsize=32768
sudo mount -t nfs nfs_server:/mount_point /mnt/svn-nocto -o \
rw,nosuid,tcp,rsize=32768,wsize=32768,nocto,actimeo=3600
cd /mnt/svn/repo_path
mv revs revs-nocto
mv txn-protorevs txn-protorevs-nocto
ln -s /mnt/svn-nocto/repo_path/db/revs-nocto revs
ln -s /mnt/svn-nocto/repo_path/db/txn-protorevs-nocto txn-protorevs
start the servers

== Increase NFS caching timeout: ==

On Linux, metadata on files from NFS is only kept for a finite period
of time. This can be changed by passing the actimeo option to the
mount command. The man page claims the default is 60 (seconds), but
some experimentation suggests it may be higher than that. For a
'nocto' mount point, this value can be raised to something much larger
(e.g. 3600). See the above example.

== Distributing CPU load: ==

The subversion communicates with the clients by transmitting
differences in state, so the CPU load to calculate the difference can
be significant. By storing the repository on NFS, you can have
multiple "front end" (FE) systems that share the computational load
and provide redundancy. A network load balancer makes all front ends
(FEs) appear as one server to users.

The FEs can either run svnserve or http-DAV. If DAV is used, you need
to ensure that the load balancer keeps an entire transaction on the
same FE (to allow transactions to be built up on local disk). The
load balancer must be configured with "machine affinity" set, so that
all HTTP connections from a client will be routed to the same server.
You should also configure apache to keep a single TCP connection for
the entire transaction (see example below).

Apache configuration to maintain a TCP connection:
# 1. Enable HTTP persistent connections so a single transaction can
# be built up over a single connection.
KeepAlive on
# 2. Allow as many KeepAlives as required (0 => infinite) to keep
# the same connection alive.
MaxKeepAliveRequests 0
# 3. Limit a child to serving only this 1 connection.
MaxRequestsPerChild 1

The last one is counter-intuitive, but see the "Note" at
http://httpd.apache.org/docs/2.2/mod/mpm_common.html#maxrequestsperchild.

== High storage system reliability: ==

The purpose of a version control system is to store a sequence of
file/directory versions so you can retreive them in the future. None
of this matters if the storage system fails.

The simplest step is to do periodic backups of the repository. This
limits the loss to the changes that happened since the last backup.
If the repository is large and the commit rate is high, it may be
impossible to backup frequently enough to prevent significant data
loss. For example, if your repository gets one commit per second and
you do a backup every hour, you may lose 3600 revisions if the disk
fails. This is a large scale example, but the point is to gather
your own numbers and figure out how much you might lose.

The next step is to make the disk system redundant using RAID
technology. This allows one (and sometimes more) disks to fail
without losing data. This still won't help if additional disks fail
during recovery or the entire array is lost due to fire, theft, etc.

Advanced NFS servers can be configured to do synchronous mirroring
and/or asynchronous mirroring (also known as snapshot replication).
These capabilities are available in some commercial servers, or you
can find various free alternatives by searching for "NFS server high
availability" or "NFS server snapshot replication".

Synchronous mirroring sends every write to two independent storage
systems and requires a high bandwidth network (e.g. gigabit ethernet).
It reduces performance, but the caching optimizations listed above can
help. The primary and slave systems are usually located in different
rooms (or buildings) and on different electrical circuits.

Asynchronous mirroring periodically updates a second storage system
with changes from the master. It periodically makes a "snapshot" of
the system every few minutes and then transmits the difference between
the previous snapshot and the current one to the slave. This uses
less bandwidth, but lags the main filesystem by a several minutes. It
can allow a backup filesystem to be located in another geographic
region.

Another approach is to use subversion tools to maintain a mirror.
Setup svnsync to periodically sync a back up server off the main one.
This can lag behind by the polling interval, but it is simple to setup.

You can eliminate the lag by setting up a post-commit script that runs
"svnadmin dump --incremental -r N" of that commit onto a separate
partition/server. This creates a transaction log of commits that can
be replayed on a recent backup to restore full state.

(TODO: reference the benchmarking capabilites in mstone)

Server performance tuning for Linux and Unix

2008-05-30T20:42:08Z

Dchristian: /* Server performance tuning for Linux/Unix */

Server performance tuning for Linux and Unix

2008-05-30T20:32:37Z

Dchristian: Server performance tuning for Linux/Unix

== Server performance tuning for Linux/Unix ==

Optimizing subversion server performance
Dan Christian
5 May 2008

General notes:

Most of these tips apply to both the Berkeley DB and the FSFS
repository formats. FSFS is generally considered to be faster.
Database servers have the same kinds of problems and are a good source
for disk and operating system (OS) tuning advice.
http://dev.mysql.com/doc/refman/5.0/en/innodb-configuration.html

Making reads cheaper:

The Unix concept of "file access time" (commonly call atime) is a
performance problem. When filesystem sematics were being defined, it
seemed like a good idea to know when a file was last accessed. The
down side is that every file open call now causes a disk write. A few
utilites use this information (e.g. tmpwatch and mail), but subversion
never uses atime. Subversion performance is improved by avoiding the
access time writes.

For a local filesystem, you can disable this behavior with mount
options. On linux, it's the 'noatime' and 'nodiratime' options. On a
NFS filesystem, the atime recording happens on the server and must be
disabled in the server's configuration.

A lazy atime approach called "relatime" was introduced in Linux-2.6.20
and mount-2.13. This eliminates most atime writes without breaking
the few utilities that need it. This is most useful if the repository
must be on the same partition as the mail spool and/or temporary
files. See: http://kerneltrap.org/node/14148 and
http://kernelnewbies.org/Linux_2_6_20

Making writes cheaper:

Subversion uses uses the fsync() call (or the equivalent on non-Unix
operating systems) to tell the operating system to write data to disk.
Up until that point, the data is usually only memory and the operating
system will write it to disk "when it gets around to it".

By calling fsync() before finishing a commit, subversion is trying to
guarantee that everything it said had been done would be there when
the machine re-boots. Waiting for data to write out to disk is often
the slowest part of a commit.

However, the operating system doesn't always hold up its end of the
bargain. On Linux, fsync() only ensures that the data is on its way
to the disk "as soon as possible". If write cache is enabled on the
drive, then it doesn't actually wait for the data to hit the disk
platter before returning. This means there is a window of time that a
power loss can cause the disk state to not match what subversion
returned.

One way to significantly increase fsync() performance is to use a RAID
controller with a battery backed write cache. The cache is treated as
part of the disk system. As soon as the data is in the cache, the
fsync() can safely return. This means you don't have to wait for the
disk head seek or the data transfer. If power is interrupted, the
RAID controller will finish writing out the cache when power is
restored.

A newer way to avoid this problem is a flash based disk. There is no
latency from head movement or waiting for the disk to rotate. This
becomes more significant when writing many small files (like many FSFS
writes). The current downsides of flash disks are high cost, limited
capacity, and low write bandwidth (but these problems are improving).

Reducing the number of writes:

As of subversion-1.5, transactions can be built up on a different
filesystem than the one holding the repository. This is valuable when
the repository lives on a slower filesystem like NFS.

To implement this, do the following:
stop all servers that can write to the repository
cd REPO_PATH/db
mv transactions /LOCAL/DISK/PATH/
ln -s /LOCAL/DISK/PATH/transactions .
start the servers

Reduce directory index size:

The subversion-1.5 repository format allows the revisions to be stored
in subdirectories that don't grow past a specified size. This allows
repositories to store many more revisions than can (efficiently) be
stored in one directory.

Modern filesystems can handle hundreds of thousands of files in a
single directory. However, performance can suffer as the directory
index starts to use multiple levels of indirection. Some
administration tools may also have trouble with very large
directories. Splitting the revision store into sub-directories avoids
all these problems.

The shard size can by adjusted by editing the "layout sharded" line in
"db/format" after 'svnadmin create' but before populating the
repository. The default is 1000 revisions per subdirectory.
Non-sharded repositories can be loaded into a new, sharded,
repository using "svnadmin load" or "svnsync".

Make write-once portions of the repository skip NFS cache checks:

If the repository is on a NFS filesystem, then a cache consistency
check is made every time a file is opened. Since the revision files
in a FSFS repository never change, it is worthwhile to skip the cache
checks on these files. The subversion-1.5 repository format store
immutable files in specific subdirectories so that this can be done.

The NFS cache check can be disabled on Linux by passing the 'nocto'
option to the mount command (note: the man page claims this is
ignored, but it isn't on linux-2.6). You need coherency for some
files, so the NFS volume is also mounted without the option on a
different mount point. Symbolic links are made from the cache
coherent mount point to the 'nocto' mount for these directories: revs
and txn-protorevs.

Implementation example (not complete, just an outline of the key steps):
stop all servers that can write to the repository
sudo mount -t nfs nfs_server:/mount_point /mnt/svn -o \
rw,nosuid,tcp,rsize=32768,wsize=32768
sudo mount -t nfs nfs_server:/mount_point /mnt/svn-nocto -o \
rw,nosuid,tcp,rsize=32768,wsize=32768,nocto,actimeo=3600
cd /mnt/svn/repo_path
mv revs revs-nocto
mv txn-protorevs txn-protorevs-nocto
ln -s /mnt/svn-nocto/repo_path/db/revs-nocto revs
ln -s /mnt/svn-nocto/repo_path/db/txn-protorevs-nocto txn-protorevs
start the servers

Increase NFS caching timeout:

On Linux, metadata on files from NFS is only kept for a finite period
of time. This can be changed by passing the actimeo option to the
mount command. The man page claims the default is 60 (seconds), but
some experimentation suggests it may be higher than that. For a
'nocto' mount point, this value can be raised to something much larger
(e.g. 3600). See the above example.

Distributing CPU load:

The subversion communicates with the clients by transmitting
differences in state, so the CPU load to calculate the difference can
be significant. By storing the repository on NFS, you can have
multiple "front end" (FE) systems that share the computational load
and provide redundancy. A network load balancer makes all front ends
(FEs) appear as one server to users.

The FEs can either run svnserve or http-DAV. If DAV is used, you need
to ensure that the load balancer keeps an entire transaction on the
same FE (to allow transactions to be built up on local disk). The
load balancer must be configured with "machine affinity" set, so that
all HTTP connections from a client will be routed to the same server.
You should also configure apache to keep a single TCP connection for
the entire transaction (see example below).

Apache configuration to maintain a TCP connection:
# 1. Enable HTTP persistent connections so a single transaction can
# be built up over a single connection.
KeepAlive on
# 2. Allow as many KeepAlives as required (0 => infinite) to keep
# the same connection alive.
MaxKeepAliveRequests 0
# 3. Limit a child to serving only this 1 connection.
MaxRequestsPerChild 1

The last one is counter-intuitive, but see the "Note" at
http://httpd.apache.org/docs/2.2/mod/mpm_common.html#maxrequestsperchild.

High storage system reliability:

The purpose of a version control system is to store a sequence of
file/directory versions so you can retreive them in the future. None
of this matters if the storage system fails.

The simplest step is to do periodic backups of the repository. This
limits the loss to the changes that happened since the last backup.
If the repository is large and the commit rate is high, it may be
impossible to backup frequently enough to prevent significant data
loss. For example, if your repository gets one commit per second and
you do a backup every hour, you may lose 3600 revisions if the disk
fails. This is a large scale example, but the point is to gather
your own numbers and figure out how much you might lose.

The next step is to make the disk system redundant using RAID
technology. This allows one (and sometimes more) disks to fail
without losing data. This still won't help if additional disks fail
during recovery or the entire array is lost due to fire, theft, etc.

Advanced NFS servers can be configured to do synchronous mirroring
and/or asynchronous mirroring (also known as snapshot replication).
These capabilities are available in some commercial servers, or you
can find various free alternatives by searching for "NFS server high
availability" or "NFS server snapshot replication".

Synchronous mirroring sends every write to two independent storage
systems and requires a high bandwidth network (e.g. gigabit ethernet).
It reduces performance, but the caching optimizations listed above can
help. The primary and slave systems are usually located in different
rooms (or buildings) and on different electrical circuits.

Asynchronous mirroring periodically updates a second storage system
with changes from the master. It periodically makes a "snapshot" of
the system every few minutes and then transmits the difference between
the previous snapshot and the current one to the slave. This uses
less bandwidth, but lags the main filesystem by a several minutes. It
can allow a backup filesystem to be located in another geographic
region.

Another approach is to use subversion tools to maintain a mirror.
Setup svnsync to periodically sync a back up server off the main one.
This can lag behind by the polling interval, but it is simple to setup.

You can eliminate the lag by setting up a post-commit script that runs
"svnadmin dump --incremental -r N" of that commit onto a separate
partition/server. This creates a transaction log of commits that can
be replayed on a recent backup to restore full state.

(TODO: reference the benchmarking capabilites in mstone)