Letter frequency
The frequency of letters ( frequency of graphs ) is a statistical quantity that indicates how often a certain letter occurs in a text or a collection of texts (corpus) . It can be specified as an absolute number or in relation to the total number of letters in the text. The frequency distribution of the letters depends on the language in question. While earlier assumptions believed to predict the statistical distribution of the frequency of letters by Zipf's law , quantitative linguistics has shown that a number of other probability distributions must be taken into account. Counts of the frequency of letters or sounds in texts or text corpora have been traceable since the early 19th century at the latest. For some purposes it is also interesting how often a letter occurs at the beginning or at the end of a word.
application
The letter frequency is used in the decryption of substitution processes in cryptanalysis as well as in data compression and coding . With simple encryption methods such as the Caesar cipher , a ciphertext can be decrypted by frequency analysis alone . The frequencies of the individual characters in the ciphertext are determined and then compared with the frequency of the characters in a plain text of the assumed language. Now the letters of the ciphertext are replaced by the normal letters of the same frequency. The most common letter of the ciphertext then corresponds, for example, to the plaintext letter e. This method is obviously particularly well suited for longer texts to be deciphered, because the statistical deviation between the frequency of letters found and the frequency to be expected is smaller.
For typing lessons, it is important that the teacher is well informed about the frequency of letters in a language and that the lesson content is tailored accordingly. Frequent letters like the E or the I have to be trained sufficiently to achieve the highest possible number of keystrokes and good writing reliability. When creating ergonomic keyboard layouts , the frequency of letters also plays a major role. Manufacturers of letter games such as Boggle or Scrabble also take into account the frequency and, if available, the value of the letters in the national variants.
One of the earliest uses was the Morse alphabet , which uses short codes for common characters (for example, E = · ); for characters that are rarely used, however, longer codes (for example Q = - - · - ).
Continuation
The continuation of the letter frequency is the frequency of letter pairs and triples and the word frequency as well as of writing units that stand for a systematic sound unit (graphemes for phonemes). If you deal with the spoken language instead of the written language, you can also carry out surveys on the frequency of sounds or phonemes .
Frequency of letters in German-language texts
From the following table it can be mathematically derived that the five most common letters cover around half and the ten most common letters three quarters of the letter frequency in German-language texts. The umlauts ä, ö and ü were counted like ae, oe and ue , ß as a separate character.
| space | Letter | Relative frequency |
|---|---|---|
| 1. | E. | 17.40% |
| 2. | N | 9.78% |
| 3. | I. | 7.55% |
| 4th | S. | 7.27% |
| 5. | R. | 7.00% |
| 6th | A. | 6.51% |
| 7th | T | 6.15% |
| 8th. | D. | 5.08% |
| 9. | H | 4.76% |
| 10. | U | 4.35% |
| 11. | L. | 3.44% |
| 12. | C. | 3.06% |
| 13. | G | 3.01% |
| 14th | M. | 2.53% |
| 15th | O | 2.51% |
| 16. | B. | 1.89% |
| 17th | W. | 1.89% |
| 18th | F. | 1.66% |
| 19th | K | 1.21% |
| 20th | Z | 1.13% |
| 21st | P | 0.79% |
| 22nd | V | 0.67% |
| 23. | ẞ | 0.31% |
| 24. | J | 0.27% |
| 25th | Y | 0.04% |
| 26th | X | 0.03% |
| 27. | Q | 0.02% |
For comparison: If the 27 letters were evenly distributed , the relative frequency would be 3.704%.
For comparison, a file that contains 99,586 letters from a mixed corpus of one person's letters (correspondence with authorities, friends, colleagues, broadcasters, publishers ...; always only the current text, i.e. without letterhead, salutation and greeting; letters from 1996– 2004) is based. In contrast to the previous overview, the umlaut letters <ä>, <ö> and <ü> are each raised separately.
| space | Letter | Absolute frequency | Relative frequency |
|---|---|---|---|
| 1. | E. | 16,040 | 16.11% |
| 2. | N | 10,288 | 10.33% |
| 3. | I. | 9,011 | 9.05% |
| 4th | R. | 6,693 | 6.72% |
| 5. | T | 6.312 | 6.34% |
| 6th | S. | 6,203 | 6.23% |
| 7th | A. | 5,577 | 5.60% |
| 8th. | H | 5,177 | 5.20% |
| 9. | D. | 4.156 | 4.17% |
| 10. | U | 3,680 | 3.70% |
| 11. | C. | 3,384 | 3.40% |
| 12. | L. | 3.226 | 3.24% |
| 13. | G | 2,924 | 2.94% |
| 14th | M. | 2,784 | 2.80% |
| 15th | O | 2,312 | 2.32% |
| 16. | B. | 2.176 | 2.19% |
| 17th | F. | 1,701 | 1.71% |
| 18th | W. | 1,383 | 1.39% |
| 19th | Z | 1,351 | 1.36% |
| 20th | K | 1,329 | 1.33% |
| 21st | V | 912 | 0.92% |
| 22nd | P | 841 | 0.84% |
| 23. | Ü | 636 | 0.64% |
| 24. | Ä | 511 | 0.51% |
| 25th | Ö | 363 | 0.36% |
| 26th | ẞ | 189 | 0.19% |
| 27. | J | 186 | 0.19% |
| 28. | X | 112 | 0.11% |
| 29 | Q | 73 | 0.07% |
| 30th | Y | 56 | 0.06% |
The Institute for German Language in Mannheim offers various character and letter frequency lists for download on its website. The statistics are based on a text sample of almost 180 billion characters from the German reference corpus (as of 2018).
Duden offers an overview of the frequency of letters in the form of a bar chart on the basis of the Duden corpus, a full text collection with over 2 billion word forms; The umlaut letters are also listed individually in this overview. The graphic was revised in the 27th edition of the Spelling Duden, now on the basis of the Duden corpus with now 4 billion word forms (as of spring 2017).
first letters
The frequency of initial letters indicates how often a letter appears as the first letter of a word. It depends to a large extent on the type of text. The five most common initial letters for running text are:
| space | Letter | Relative frequency |
|---|---|---|
| 1. | D. | 14.2% |
| 2. | S. | 10.8% |
| 3. | E. | 7.8% |
| 4th | I. | 7.1% |
| 5. | W. | 6.8% |
There is a different distribution for lexica. The letters D, E, I and W appear much less often at the beginning of a word than in the running text, S is the most common with a clear margin:
| space | Letter | Relative frequency |
|---|---|---|
| 1. | S. | 11.8% |
| 2. | K | 7.3% |
| 3. | A. | 7.1% |
| 4th | P | 7.0% |
| 5. | B. | 5.7% |
| 6th | M. | 5.7% |
Final letters
The frequency of final letters indicates how often a letter occurs as the last letter of a word. (The novel Effi Briest by Theodor Fontane was evaluated as an example text basis , whereby ß was always counted as ss . The text basis comprises all 36 chapters of this work with a total of 572,849 characters.)
| space | Letter | Relative frequency |
|---|---|---|
| 1. | N | 21.0% |
| 2. | E. | 15.1% |
| 3. | R. | 13.0% |
| 4th | T | 10.3% |
| 5. | S. | 9.6% |
Frequency charts
Diagram showing the relative frequency of letters in German-language texts
Monogram frequency range: The letter frequency distribution of a longer German text.
Bigram frequency range: Distribution of the most common bigram in a German text.
Trigram frequency range: Distribution of the most common trigrams in a German text. The triples ER_ and EN_ are the most common [_ stands for the space].
3D bar chart : frequency of vowel letters in Serbo-Croatian texts
Frequency of letters in selected languages
| Letter | German | English | French | Spanish | Esperanto | Italian | Swedish | Polish |
|---|---|---|---|---|---|---|---|---|
| a | 6.51% | 8.167% | 7.636% | 12.53% | 12.12% | 11.74% | 9.3% | 8.0% |
| b | 1.89% | 1.492% | 0.901% | 1.42% | 0.98% | 0.92% | 1.3% | 1.3% |
| c | 3.06% | 2,782% | 3,260% | 4.68% | 0.78% | 4.5% | 1.3% | 3.8% |
| d | 5.08% | 4.253% | 3.669% | 5.86% | 3.04% | 3.73% | 4.5% | 3.0% |
| e | 17.40% | 12.702% | 14.715% | 13.68% | 8.99% | 11.79% | 9.9% | 6.9% |
| f | 1.66% | 2.228% | 1.066% | 0.69% | 1.03% | 0.95% | 2.0% | 0.1% |
| G | 3.01% | 2.015% | 0.866% | 1.01% | 1.17% | 1.64% | 3.3% | 1.0% |
| H | 4.76% | 6.094% | 0.737% | 0.70% | 0.38% | 1.54% | 2.1% | 1.0% |
| i | 7.55% | 6.966% | 7.529% | 6.25% | 10.01% | 11.28% | 5.1% | 7.0% |
| j | 0.27% | 0.153% | 0.545% | 0.44% | 3.50% | 0.00% | 0.7% | 1.9% |
| k | 1.21% | 0.772% | 0.049% | 0.00% | 4.16% | 0.00% | 3.2% | 2.7% |
| l | 3.44% | 4.025% | 5.456% | 4.97% | 6.14% | 6.51% | 5.2% | 3.1% |
| m | 2.53% | 2.406% | 2.968% | 3.15% | 2.99% | 2.51% | 3.5% | 2.4% |
| n | 9.78% | 6.749% | 7.095% | 6.71% | 7.96% | 6.88% | 8.8% | 4.7% |
| O | 2.51% | 7.507% | 5.378% | 8.68% | 8.78% | 9.83% | 4.1% | 7.1% |
| p | 0.79% | 1.929% | 3.021% | 2.51% | 2.74% | 3.05% | 1.7% | 2.4% |
| q | 0.02% | 0.095% | 1.362% | 0.88% | 0.00% | 0.51% | 0.007% | 0.00% |
| r | 7.00% | 5.987% | 6.553% | 6.87% | 5.91% | 6.37% | 8.3% | 3.5% |
| s | 7.27% | 6.327% | 7.948% | 7.98% | 6.09% | 4.98% | 6.3% | 3.8% |
| t | 6.15% | 9,056% | 7.244% | 4.63% | 5.27% | 5.62% | 8.7% | 2.4% |
| u | 4.35% | 2,758% | 6.311% | 3.93% | 3.18% | 3.01% | 1.8% | 1.8% |
| v | 0.67% | 0.978% | 1.628% | 0.90% | 1.90% | 2.10% | 2.4% | 0.00% |
| w | 1.89% | 2,360% | 0.114% | 0.02% | 0.00% | 0.00% | 0.03% | 3.6% |
| x | 0.03% | 0.150% | 0.387% | 0.22% | 0.00% | 0.00% | 0.1% | 0.00% |
| y | 0.04% | 1,974% | 0.308% | 0.90% | 0.00% | 0.00% | 0.6% | 3.2% |
| z | 1.13% | 0.074% | 0.136% | 0.52% | 0.50% | 0.49% | 0.02% | 5.1% |
| œ | 0.00% | 0.00% | 0.018% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% |
| ß | 0.31% | 0.00% | 0.00 % | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% |
| à | 0.00% | 0.00% | 0.486% | 0.00% | 0.00% | see a | 0.00% | 0.00% |
| ą | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | see a |
| ç | 0.00% | 0.00% | 0.085% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% |
| ĉ | 0.00% | 0.00% | 0.00 % | 0.00% | 0.66% | 0.00% | 0.00% | 0.00% |
| ć | 0.00% | 0.00% | 0.00 % | 0.00% | 0.00% | 0.00% | 0.00% | see c |
| è | 0.00% | 0.00% | 0.271% | 0.00% | 0.00% | see e | 0.00% | 0.00% |
| é | 0.01% | 0.00% | 1.904% | 0.00% | 0.00% | see e | 0.00% | 0.00% |
| ê | 0.00% | 0.00% | 0.225% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% |
| ë | 0.00% | 0.00% | 0.00 % | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% |
| ę | 0.00% | 0.00% | 0.00 % | 0.00% | 0.00% | 0.00% | 0.00% | see e |
| G | 0.00% | 0.00% | 0.00 % | 0.00% | 0.69% | 0.00% | 0.00% | 0.00% |
| H | 0.00% | 0.00% | 0.00 % | 0.00% | 0.02% | 0.00% | 0.00% | 0.00% |
| î | 0.00% | 0.00% | 0.045% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% |
| ì | 0.00% | 0.00% | 0.00 % | 0.00% | 0.00% | see i | 0.00% | 0.00% |
| ï | 0.00% | 0.01% | 0.005% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% |
| ĵ | 0.00% | 0.00% | 0.00 % | 0.00% | 0.12% | 0.00% | 0.00% | 0.00% |
| ł | 0.00% | 0.00% | 0.00 % | 0.00% | 0.00% | 0.00% | 0.00% | see l |
| ń | 0.00% | 0.00% | 0.00 % | 0.00% | 0.00% | 0.00% | 0.00% | see n |
| O | 0.00% | 0.00% | 0.00 % | 0.00% | 0.00% | 0.00% | 0.00% | see above |
| O | 0.00% | 0.00% | 0.00 % | 0.00% | 0.00% | see above | 0.00% | 0.00% |
| ŝ | 0.00% | 0.00% | 0.00 % | 0.00% | 0.38% | 0.00% | 0.00% | 0.00% |
| ś | 0.00% | 0.00% | 0.00 % | 0.00% | 0.00% | 0.00% | 0.00% | see p |
| ù | 0.00% | 0.00% | 0.058% | 0.00% | 0.00% | see u | 0.00% | 0.00% |
| ŭ | 0.00% | 0.00% | 0.00 % | 0.00% | 0.52% | 0.00% | 0.00% | 0.00% |
| ź | 0.00% | 0.00% | 0.00 % | 0.00% | 0.00% | 0.00% | 0.00% | see e.g. |
| ż | 0.00% | 0.00% | 0.00 % | 0.00% | 0.00% | 0.00% | 0.00% | 0.7% |
What is particularly noteworthy in the table is that the letter E is used significantly more frequently in German and the letter O is used significantly less often than in Romance and Slavic languages.
The table only shows the frequencies of letters in texts / corpora of languages for which the Latin script is used. For the frequency of letters in languages with the Cyrillic script, reference can be made to the description by Kempgen (1995) on Russian and the study by Grzybek & Kehlich (2005) on Ukrainian.
See also
- Benford's law (number frequency)
- Frequency class
- Sound frequency
- Phoneme frequency
- Word frequency
- N-grams
literature
- Friedrich L. Bauer : Deciphered Secrets. Methods and maxims of cryptology. Springer, Berlin et al. 1995, ISBN 3-540-58118-9 . (Contains letter frequencies in German and English with percentages on page 223.)
- Karl-Heinz Best : On the frequency of letters, spaces and other characters in German texts. In: Glottometrics 11, 2005, ISSN 1617-8351 , pages 9-31. (PDF full text ). (In addition to the frequency of letters, it also indicates the proportions of other characters in German texts.)
- Erich Mater: German verbs. 1. Alphabetical index. Bibliographisches Institut, Leipzig 1966. (Contains in the opening chapter an overview of the frequency of first letters in 6 different dictionaries as well as an overview. Unfortunately no page count.)
- Helmut Meier : German language statistics. (= Olms Paperbacks. 31). 2nd, enlarged and improved edition. Olms, Hildesheim 1967. (Letter statistics for German, English and Spanish on page 334.)
- Gustav Muthmann: Declining German dictionary. Handbook of word exits in German, taking into account the word and sound structure. (= German Linguistics series. 78). Niemeyer, Tübingen 1988, ISBN 3-484-31078-2 . (On page 36 there is a compilation of the frequencies of initial letters and page 65 of the final letters.)
- Gustav Muthmann: Phonological dictionary of the German language. (= German Linguistics series. 163). Niemeyer, Tübingen 1996, ISBN 3-484-31163-0 , pages 35-37. (Frequency of graphemes and phonemes .)
- Wolfgang Schönpflug : n-gram frequency in the German language. I. Monograms and digrams. In: Journal for Experimental and Applied Psychology. 16, 1969, ISSN 0044-2712 , pages 157-183. (On page 162f. Contains an overview of the frequency of letters in a text corpus of over 100,000 words, separated according to their position in the word.)
- Katja Siekmann, Günther Thomé: The spelling mistake. 2nd, updated edition Oldenburg 2018, ISBN 978-3-942122-07-8 (contains detailed overviews on pages 239 to 247 of the frequency of letters and letter combinations from a more recent 100,000 count of phoneme-grapheme correspondences in German. Isb-Verlag Oldenburg ).
- Dorothea Thomé, Günther Thomé: Phonemes and graphemes in German: three diagrams. 1. The sounds of German (according to the standard wording), 2. Basic graphemes (basic characters for phonemes), 3. All basic and orthographemes (what is how often?). isb-Fachverlag, Oldenburg 2014, ISBN 978-3-942122-15-3 .
- Günther Thomé, Dorothea Thomé: German words structured according to phonetic and written units. isb-Fachverlag, Oldenburg 2016, ISBN 978-3-942122-21-4 (reading samples from isb-Verlag Oldenburg ; With numerous tables on the frequency of phonetic and written units in German.)
Web links
- Corpus-based character frequency lists at the Institute for German Language, Mannheim
- Graz project for quantitative text analysis (QuanTA)
- Bibliographies and further information on the Göttingen Quantitative Linguistics project
- Portable freeware tool to determine the frequency of letters and / or syllables from texts of any length
Individual evidence
- ↑ See also: Archived copy ( Memento of the original from April 7, 2015 in the Internet Archive ) Info: The archive link was automatically inserted and not yet checked. Please check the original and archive link according to the instructions and then remove this notice. Letters, sounds and phonemes basically follow the same distributions.
- ^ Karl-Heinz Best: phonetic and letter counting in the early 19th century. In: Glottometrics 20, 2010, pages 110-114. (PDF full text ).
- ^ Albrecht Beutelspacher: Kryptologie. 7th edition. Vieweg Verlagsgesellschaft, Wiesbaden 2005, ISBN 3-8348-0014-7 , page 10.
- ^ Karl-Heinz Best: Letter frequencies in German and English. In: Naukovyj Visnyk Černivec'koho Universitetu. Vypusk 231, 2005, ZDB -ID 2390772-1 , pages 119-127.
- ↑ Institute for German Language: Corpus Linguistics: Corpus-based character and letter frequency lists. Retrieved on March 20, 2018 (German).
- ↑ Duden - German Universal Dictionary. 7th, revised and expanded edition. Dudenverlag, Mannheim / Zurich 2011, ISBN 978-3-411-05507-4 , page 2110.
- ↑ Duden. The German spelling. 27th, completely revised and expanded edition. Dudenverlag, Berlin 2017, ISBN 978-3-411-04017-9 , pages 148, 158.
- ↑ a b Peter Vogelgesang: Frequency of letters. ( Memento of February 9, 2006 in the Internet Archive ) 2003.
- ↑ Robert Edward Lewand: Relative Frequencies of Letters text in General English Plain.
- ↑ CorpusDeThomasTempé. ( Memento from February 13, 2008 in the Internet Archive )
- ^ Fletcher Pratt: Secret and Urgent: the Story of Codes and Ciphers Blue Ribbon Books , 1939, pp. 254-255.
- ↑ La Oftecoj de la Esperantaj Literoj. Retrieved September 14, 2007 .
- ↑ Simon Singh: Codici e Segreti. RCS, 1999, ISBN 88-17-12539-3 .
- ↑ Simon Singh: Brogren Margareta: Kodboken: konsten att skapa sekretess - från det gamla Egypt till kvantkryptering . Norstedt, Stockholm 1999, ISBN 91-1-300708-4 .
- ↑ Wstęp do kryptologii. ( MS Word ; 300 kB) Retrieved April 30, 2012 .
- ^ Sebastian Kempgen: Russian language statistics. Systematic overview and bibliography . Verlag Otto Sagner, Munich 1995, pages 19-22. ISBN 3-87690-617-2 .
- ↑ Peter Grzybek, Emmerich Kelih: Grapheme frequencies in Ukrainian. Part I: Without an apostrophe (') . In: Gabriel Altmann, Viktor Levickij, & Valentina Perebyinis (eds.): Problemy kvantytatyvnoi linhvistyky / Problems of Quantitative Linguistics: zbirnyk naukovych prac (pp. 159-179). Ruta, Cernivci 2005. ISBN 966-568-783-2 .
