Have fun! 頑張ってね～～

Hit Chart Elegy (音楽番付哀歌)
Just Like Great Britain’s–Unfortunately Bibi on Fox News Две ракеты были выпущены по южной территории #Израиль из #СекторГаза
★ 12月30日 Awoi Instore at Zeal Link Shibuya
…fermented red beancurd or hong fu ru 红腐乳…
…to order their debut EP, 플레이걸의 24時.
Xi Lu Han is the full name of Lu Han (루한 / 鹿 晗)
..LOL we may be getting an adlut♥♥ I hope !
He loves YOU!(((: #His<3isbettter

Regular English by itself offers fast endless plenty of grammatical, syntactical and lexical forms and combinations with extremely rich semantic background. In its function of lingua franca in the times of globalization and Internet, it multiplies the variability of the lexical forms an analyst has to deal with.

The most challenging step in the course of analysis of a given text corpus HC corpus for the purposes of prediction was therefore preprocessing of the text, trying to remove the “garbage”, reduce the variability but avoiding extreme bias as a result. The corpus includes patterns from different segments of Internet (news, blogs, Twitter).

The main steps of text preprocessing in this project were:

• removing numbers
• trimming excessive white spaces
• removing lines with words from languages like Chinese, Japanese, Russian, Hebrew and so on
• converting the punctuation marks to standard where possible
• removing “words” consisting of non-letter characters (though, hash-tags and simple smileys accepted)
• reducing variability through leaving out articles (the, a , an)

N-Gramms model has been chosen as a basis for prediction engine. Such models are widely used in statistical natural language processing. They are typically collected from a text or speech corpus. So did we as well (HC Corpus, available over Coursera/SwiftKey).

The following principles were implemented for building n-gram databases:

• Building data bases of mono, bi-, tri-, four- and pentagrams.
• End of the sentence or of the line was used as a natural break point building the n-grams.
• Data bases were packed by the length in the R data tables, splited into predictor (all words of n-gram but last), outcome (the last word of n-gram) plus detected frequency. The format is directly usable by prediction engine.
• Because of high memory consumption only 15% of the text corpus could be processed at a time. With random sampling and 21 processing iterations the lines coverage in corpus of 97% has been reached (mean redundancy 3.15) . The n-gram databases were then then merged into one per n-gram length.
• Working data bases were created, trimming the over-represented predictors (like “to be”) to upper 150 outcomes by frequency and removing rare combinations.

Using the data bases created, following predictive model has been implemented:

• Maximal length of an expression took as a predictor is four words.
• As the first step, the decision is made towards either next-word or in-word prediction (trailing blanks in the input string?); a flag for possible article as a last word will be set.
• For next-word search, the line trimmed the same way as the text corpus is used as the subject for exact search in the indexed data tables, from longer to shorter predictors downwards. Each of the following steps is to be performed, when preceding one still has brought no suggestions.
• Exact search after trimming by endings like “ 'll”, “ 're” and so on.
• False input of the last word (typo?) assumed; the search is to be repeated iteratively, shortening the last word by one letter.
• The original predictor expression is shortened by the first word; the rest becomes the subject of exact search and other steps, listed above.
• The last word standing goes into exact search in the used words database, then into in-word prediction.
• The in-word prediction make the search in the data base of words used in text corpus, followed by the reference word list if needed. Exact search and search for the words, beginning with the given pattern.
• In in-word prediction, the relative frequency of the findings in the data bases as well as eventually preceding n-gram (for in-word prediction) count by sorting of the output.