Q: Is there any way around the 2GB file-size limit in Linux? Are there any stable patches to fix it?

A: Short answer: In a practical sense, no. The 2GB limit is deeply embedded in the versions of Linux for 32-bit CPUs: in GNU libc, in the Linux kernel's filesystem drivers and VFS layer, in the fundamental design of some filesystems, and in the function calls used in the compiled applications and utilities furnished in (and for) typical Linux systems. There is thus currently no commonly available configuration.

Long answer: All Linux ports for 32-bit CPUs (i.e., all architectures other than Alpha and IA-64) use 32-bit integers for file access and locking, yielding a maximum size of 2^31 - 1 = 2GB. Creating a workaround for this situation without breaking existing code is an obstacle for all 32-bit Unixes[1], whose creators met to design a standard 64-bit file-access programming interface, at a meeting sponsored by X/Open (later renamed The Open Group), called the Large File Summit (LFS)[2]. The resulting large-file standard and related system calls are likewise called LFS.

To implement LFS (and remove the 2GB limit) on a Linux system, you would have to:

1. Install glibc v. 2.2 or later, compiled against kernel headers from kernel 2.4.0test7 or later. At this date, current distributions are compiled using glibc 2.1.x and kernel 2.2.x. glibc 2.2 is considered experimental.

2. Install kernel 2.4.0test7 or later, or recent 2.2.x kernels with unofficial LFS patches, or Red Hat Software's "enterprise" kernel. Earlier versions omitted the necessary routines in the middle kernel layers (e.g., VFS) and improvements to the filesystem drivers. A number of build problems have been reported with experimental kernel 2.2.x LFS efforts, including incompatibility with the much-requested NFS v. 3 and ext3 patches.

3. Use suitable filesystems. These include ext2/ext3, ReiserFS v. 3.6.x and up, IBM's JFS, SGI's XFS, and recent NFSv3 client drivers (http://nfs.sourceforge.net). Problematic filesystems include NFS v. 2, ISO9660, earlier ReiserFS, AFS, coda, Intermezzo, Minix, UFS, SCO/SysV, msdos/umsdos/vfat, smbfs, NCPfs, jffs, jffs2, and romfs.

The new maximum size of an individual file is 8 terabytes (2^63) on ext2 or ReiserFS filesystems, on LFS-compatible hosts. For other suitable filesystems, a different limit may apply in each, but reported values are all quite large.

4. Revise and recompile all programs and utilities that will manipulate large files, to use 64-bit file-access and file-locking system calls, instead of 32-bit ones. Since you will not have access to source code for proprietary applications you may use (such as Oracle), this fix may be effectively impossible for them, and very daunting to carry out on the large number of relevant open-source userspace programs.

Please be aware that, even if you implement these fixes without error, and, even if the new 64-bit functions work perfectly, you are likely to encounter performance losses and greater memory usage, when using the revised and recompiled code.

Since implementing the above-described LFS regime isn't very practical in most cases, sites having problems with excessively large files should generally find other ways around the problem: Large ongoing records can be written to multiple files branched off periodically (e.g., daily logfiles), or can be back-ended into one of the SQL databases, such as Oracle, that handle (work around) filesystem limits within their table-storage mechanisms.

To test for basic LFS support in your system:

dd if=/dev/zero of=bigfile bs=1024 count=3145728
dd if=/dev/zero of=bigfile bs=1024 count=5145728
dd if=/dev/zero bs=1024 count=5145728 > bigfile

Further information at Andreas Jaeger's page: http://www.suse.de/~aj/linux_lfs.html

[1] I have heard it alleged that some 32-bit Unixes implemented fixes different from those recommended by the Large File Summit, in some cases earlier.

[2] The X/Open Large File Summit's standards are documented here: http://ftp.sas.com/standards/large.file/