Check out David Lakens’s github and, in particular, https://lakens.github.io/statistical_inferences/repository
Note: To remove the punctuation, you can use the following function:
import string def remove_punctuation(text): punct = string.punctuation + chr(10) return text.translate(str.maketrans(punct, " " * len(punct)))
Zipf law states that the product rank X frequency is roughly constant. This ‘law’ was discovered by Estoup and popularized by Zipf. See http://en.wikipedia.org/wiki/Zipf%27s_law. Create the Zipf plot for the text of Alice in Wonderland showing, on the y axis, the log of the frequency and on the x axis the word rank (sorting words from the most frequent to the least frequent).
Display the relationship between word length and word frequencies from the data in
Generate random text (each letter from a-z being equiprobable, and the spacecharacter being 8 times more probable) of 1 million characters. Compute the frequencies of each ‘pseudowords’ and plot the rank/frequency diagram.
To know more about lexical frequencies:
Read Harald Baayen (2001) Word Frequency Distributions Kluwer Academic Publishers.
Read Michel, Jean-Baptiste, Yuan Kui Shen, Aviva P. Aiden, Adrian Veres, Matthew K. Gray, The Google Books Team, Joseph P. Pickett, et al. 2010. “Quantitative Analysis of Culture Using Millions of Digitized Books.” Science, December. https://doi.org/10.1126/science.1199644. (use scholar.google.com to find a pdf copy). Check out google ngrams at https://books.google.com/ngrams. (Note that at the bottom of the page, there is a message “Raw data is available for download here”).