Large file support

from Wikipedia, the free encyclopedia

Large file support is a property of operating systems or file systems to be able to open and edit very large files. The system used often specifies this limit value; in some (older) Linux versions, this is e.g. B. 2 GiB or 4 GiB for FAT32 . Large databases on servers, image editing or video editing but often require large files (Engl. Large files ) that are significantly larger, so must not be affected such an application from a small size limit for files.

description

The problem with the 32-bit operating systems that have been very widespread for decades is the size restriction of integer numbers . A 32-bit integer number can only represent values ​​up to 2 GiB (with sign ) or 4 GiB (unsigned). To support very large files, a new data type and associated operating system functions must therefore be introduced in 32-bit programs , which makes it necessary to rewrite old programs. Older versions and especially programs that are no longer maintained can therefore only process files with a maximum size of 2 or 4 GiB, even if large file support exists.

The development of an API with 64-bit properties came about through the development of hard disks , which broke the gigabyte limit in the early 1990s. Subsequently developed file systems adapted to it - among them FreeBSD UFS2 , Linux ext2 (1993) and Windows NTFS (1993). The functionality of the operating system kernel was passed on to the applications in various ways, until a common API was agreed in the Unix environment at the manufacturer-independent “Large File Summit” in 1996. This was established with the Single UNIX Specification Version 2 (UNIX 98).

The arithmetic in the associated C compilers was added as a new 64-bit "long long" data type in C with the standardization for C99 (from 1995). This followed from the development of the operating systems for 32-bit architectures to use an ILP32 programming model in which the traditional data types "int", "long", "pointer" are each 32-bit long. The traditional functions "ftell" and "fseek" were thus limited to 32 bits. For Posix "ftello" / "fseeko" and Windows "_ftelli64" / "_fseeki64" the manufacturers introduced a 64-bit data type - initially with different names.

implementation

The adoption of the LFS API in 32-bit programs remained incomplete for a long time. A study from 2002 showed that even the basic libraries of the operating system were still delivered without LFS support and thus indirectly restricted numerous applications. The widely used zlib library did not support the 64-bit addition on 32-bit platforms until 2006.

In the area of ​​PC / workstations, the problem was ultimately resolved because only 64-bit architectures were used. Microsoft Windows Server 2008 was the last server version that was delivered in 32 bit. Redhat Enterprise Linux 7 was only provided as a 64-bit operating system when it was first released in 2014. Ubuntu Linux stopped delivery as a 32-bit operating system in 2019. Nvidia stopped developing 32-bit drivers in 2018 and has not been providing any updates since January 2019. Mac OS from Apple stopped the development of 32-bit in 2018, so that macOS Mojave is only available as a 64-bit operating system. There is currently no known end-of-life for Microsoft Windows 10 on the desktop , as it only replaced the last old versions (Windows 7, Windows 8) at the beginning of 2020, some of which were still used on i386 architectures.

In the field of mobile devices, Google has been requiring native support for 64-bit applications since August 2019, so that 32-bit support in Android is being discontinued . The changeover to 64-bit began in 2014, when all newer processors were only announced in 64-bit, and with Android 5 ("Lollipop") a suitable operating system was made available this year. Apple had already started the transition with the 64-bit Apple A7 , which was presented in 2013. From 2015, Google only delivered the developer workstation under Linux for 64-bit. In May 2019, the spread of Android versions below 5 was still around ten percent. Since app developers concentrate on one compilation , many manufacturers have set version 5 as the minimum version from mid-2019, for example Niantic. A 32-bit version was then difficult to obtain.

Except for embedded platforms with their specialized programs, attention to large file support in the program code will wane from 2020.

Related problems

The year 2038 problem in particular shows that the traditional representation of time stamps as 32-bit "long" can lead to problems. These will also overtake each other with the transition to pure 64-bit systems. In the meantime, a 64-bit time stamp has also been made available on 32-bit systems. In the Win32 API, this meant that new functions with 64-bit time stamps were given the suffix "64" and the 64-bit file lengths were marked with an appended suffix "i64" - certainly in all four combinations (findfirst32, findfirst64 , findfirst32i64, findfirst64i32). The UNIX98 API on the other hand introduces additional functions with the suffix "64" with "_LARGEFILE64_SOURCE".

Block counters for mass storage devices are related to the large-file API. Due to the usual size of the data blocks of 512 bytes, the 32-bit numbers were only later limited. When hard disks reached the size of 2 terabytes (around 2010), the master boot record had to be replaced as a partition table by the GUID partition table , which then defined 64-bit counters for the LBA ( linear block address ). The inode counters used in Unix systems also had to be expanded, as did other file counters (for example with the stat64 / setrlimit64 functions). The revision of the Linux kernel to version 2.4 took place around 2001, together with the introduction of LFS support, which was then taken over by glibc. Since the changeover took place at the same time, with the activation of 64-bit LFS in 32-bit architectures in the GNU-C library for Linux, the inode block counters and related functions are also changed to 64-bit.

The ext3 file system from 2001 then took over some 64-bit values ​​in the driver, but remained limited to 32-bit block counters on the mass storage device. Since you mostly work in the advanced format of 4 kilobyte blocks, the maximum here is typically 8 or 16 terabytes. Larger mass storage devices in the range of tens of terabytes then had to be formatted with XFS , which also support 64-bit inodes in the data format, and thus penetrate into the exabyte range. The first 16 terabyte hard drives were shipped from mid-2019. As a solid-state drive , mass storage devices with 32 TiB were already available from 2016, and they were announced for 2020 beyond 100 TiB.

See also

  • RF64 as a 64-bit extension of RIFF WAVE audio files
  • FAT32 + as a backward compatible extension of the FAT file system (up to 256 GiB)
  • ext4 as a 48-bit extension of the ext3 file system (> 16 TiB from e2fsprogs version 1.42)

Web links

Individual evidence

  1. ^ Adding Large File Support to the Single UNIX® Specification . The Open Group. August 14, 1996.
  2. ↑ http://ac-archive.sourceforge.net/largefile/distros.html
  3. ↑ https://www.zlib.net/ChangeLog.txt
  4. ↑ Panagiotis Kolokythas: Windows Server 2008: Microsoft's last 32-bit operating system for servers . PC world. May 28, 2007.
  5. ↑ Are 32-bit applications supported in RHEL 7 or later releases? . Red hat. February 2014.
  6. ↑ Will Cooke: Intel 32bit packages on Ubuntu from 19.10 onwards . Canonical. 2nd June 2019.
  7. ^ Matthew Addams: Nvidia discontinues support for 32-bit Windows platforms . Windows Report. April 12, 2018.
  8. ↑ Steven Silver: Mojave is Apple's last version of macOS to support 32-bit apps . Apple Insider. 5th June 2018.
  9. ↑ Support for Windows 7 ends on January 14, 2020 . Microsoft. Retrieved February 9, 2020.
  10. ↑ Andreas Sebayang: On the way to pure 64-bit Android apps . Golem. 17th January 2019.
  11. ↑ a b mw: Google announces the end of 32-bit Android apps in 2021 . IT magazine. 17th January 2019.
  12. ↑ 64-bit Android: These processors are there, these changes are coming . Android user. August 26, 2014.
  13. ↑ Platform-tools 23.1.0 Linux changed to 64-bit without notice. . Android Public Tracker. December 11, 2015 .: "It turns out the android-sdk-linux / platform-tools content is 32-bit ELF in 23.0.1 but 64-bit ELF in 23.1_rc1 and 23.1.0. [..] I set ANDROID_EMULATOR_FORCE_32BIT = true [..] 23.0.1 is the last 32-bit Linux build. "
  14. ↑ F. Tenzer: Proportion of the different Android versions on all devices with Android OS worldwide in the period from May 1st to May 7th, 2019 . Statista. November 14, 2019.
  15. ↑ Elia Del Favero: Ingress and Pokémon Go will soon need at least Android 5 . June 10, 2019.
  16. ↑ Why is 32bit 0.159.0 version apk still not available? . Reddit TheSilphRoad /. December 2019.
  17. ↑ CTF function reference findfirst . Microsoft. Retrieved in 20200210.
  18. ↑ a b c Andreas Jaeger: Large File Support in Linux . SUSE GmbH. February 15, 2015.
  19. ↑ linux / bits / stat.h: / * Note stat64 has the same shape as stat for x86-64. * /
  20. ↑ MJ Rutter: The 64 bit inode problem . Retrieved February 10, 2020.
  21. ↑ a b Ext4 Howto . kernel.org. February 11, 2019 .: "Although very large file systems are on ext4's feature list, current e2fsprogs currently still limits the filesystem size to 2 ^ 32 blocks (16TiB for a 4KiB block filesystem). Allowing filesystems larger than 16T is one of the very next high-priority features to complete for ext4. "
  22. ↑ Thomas Scherer: Samsung's 32 TB SSD: The beginning of the end of the hard drive . Elector. 15th August 2016.