Lec03: 出現頻度数

作業ディレクトリの設定

getwd()

[1] "/cloud/project"

Recap of the Lec02

テキストの頻度表作成

サンプルテキスト

• Greeting from President NISHIO Shojiro

手順

1. 一行ずつ読み込んで、リストに格納

txt<-readLines("ou_msg/ou_msg_en.txt")

2. 正規表現その３：メタ文字

• : アルファベット、数字やアンダーバー以外の文字
• : 空白文字

wordLst <- strsplit(txt,"\\s|\\W")

3. 各行のデータを一括化

wordLst<-unlist(wordLst)

4. 小文字に変換

wordLst<-tolower(wordLst)

途中結果(部分出力）

head(wordLst)

[1] "greetings" ""          "everyone"  "today"     ""          "i"        

5. 空白”“の削除

#wordLst<-wordLst[nchar(wordLst)>0]
wordLst<- wordLst[wordLst != ""]

結果(部分出力）

head(wordLst)

[1] "greetings" "everyone"  "today"     "i"         "was"       "appointed"

テキストの基本統計量

単語のToken数

tokens <- length(wordLst)

単語のTypes数

• unique()関数は，リストの重複しない要素を返す

types <- length(unique(wordLst))

結果出力

paste("Tokens =", tokens)

[1] "Tokens = 844"

paste("Types =", types)

[1] "Types = 360"

TTR: Type-Token Ratioの計算

\[TTR=\frac{types}{tokens} \times 100 \]

types/tokens*100

[1] 42.65403

小数点2桁で結果を出力

round(types/tokens*100,2)

[1] 42.65

練習: Guiraud値(RTTR: Root Type-Token Ratio)を求める

\[RTTR=\frac{types}{\sqrt{tokens}} \]

小数点2桁で結果を出力

[1] 12.39

集計: Table関数

Word Frequencies

freq <- table(wordLst)

結果(部分出力）

head(freq)

wordLst
100th    19  1931  2015  2021  2022 
    1     1     1     1     3     2 

Sort

freq_data<-sort(freq, decreasing=TRUE)

結果(部分出力）

head(freq_data)

wordLst
       the        and         of         to university    society 
        43         35         31         23         22         20 

カンマ区切り形式のファイルに出力

単語頻度数分布(単色)

las: label style

単語頻度数分布(複数色)

colors = c("orange", "lightblue", "green") 
barplot(freq_data[1:20], las=3,col=colors)

cleanNLPパッケージの利用

インストール

install.packages("cleanNLP")

ライブラリの読み込み

テキスト処理の言語設定（デフォルト値:en）

• UDPipe
• Universal Dependencies

cnlp_init_udpipe()

Registered S3 method overwritten by 'data.table':
  method           from
  print.data.table     
Downloading udpipe model from https://raw.githubusercontent.com/jwijffels/udpipe.models.ud.2.5/master/inst/udpipe-ud-2.5-191206/english-ewt-ud-2.5-191206.udpipe to /cloud/lib/x86_64-pc-linux-gnu-library/4.3/cleanNLP/extdata/english-ewt-ud-2.5-191206.udpipe
 - This model has been trained on version 2.5 of data from https://universaldependencies.org
 - The model is distributed under the CC-BY-SA-NC license: https://creativecommons.org/licenses/by-nc-sa/4.0
 - Visit https://github.com/jwijffels/udpipe.models.ud.2.5 for model license details.
 - For a list of all models and their licenses (most models you can download with this package have either a CC-BY-SA or a CC-BY-SA-NC license) read the documentation at ?udpipe_download_model. For building your own models: visit the documentation by typing vignette('udpipe-train', package = 'udpipe')
trying URL 'https://raw.githubusercontent.com/jwijffels/udpipe.models.ud.2.5/master/inst/udpipe-ud-2.5-191206/english-ewt-ud-2.5-191206.udpipe'
Content type 'application/octet-stream' length 16309608 bytes (15.6 MB)
==================================================
downloaded 15.6 MB

Downloading finished, model stored at '/cloud/lib/x86_64-pc-linux-gnu-library/4.3/cleanNLP/extdata/english-ewt-ud-2.5-191206.udpipe'

head(res$token$lemma)

[1] "greeting" ","        "everyone" "."        "today"    ","       

頻度集計

head(sort(freqBycnlp, decreasing=TRUE))

  , the and  of   .  to 
 50  43  35  31  30  23 

中国語テキスト

cnlp_init_udpipe(model_name = "chinese")

Downloading udpipe model from https://raw.githubusercontent.com/jwijffels/udpipe.models.ud.2.5/master/inst/udpipe-ud-2.5-191206/chinese-gsd-ud-2.5-191206.udpipe to /cloud/lib/x86_64-pc-linux-gnu-library/4.3/cleanNLP/extdata/chinese-gsd-ud-2.5-191206.udpipe
 - This model has been trained on version 2.5 of data from https://universaldependencies.org
 - The model is distributed under the CC-BY-SA-NC license: https://creativecommons.org/licenses/by-nc-sa/4.0
 - Visit https://github.com/jwijffels/udpipe.models.ud.2.5 for model license details.
 - For a list of all models and their licenses (most models you can download with this package have either a CC-BY-SA or a CC-BY-SA-NC license) read the documentation at ?udpipe_download_model. For building your own models: visit the documentation by typing vignette('udpipe-train', package = 'udpipe')
trying URL 'https://raw.githubusercontent.com/jwijffels/udpipe.models.ud.2.5/master/inst/udpipe-ud-2.5-191206/chinese-gsd-ud-2.5-191206.udpipe'
Content type 'application/octet-stream' length 13863905 bytes (13.2 MB)
==================================================
downloaded 13.2 MB

Downloading finished, model stored at '/cloud/lib/x86_64-pc-linux-gnu-library/4.3/cleanNLP/extdata/chinese-gsd-ud-2.5-191206.udpipe'

txt_ch<-readLines("ou_msg/ou_msg_ch.txt")
txt_ch
res$token[res$token$upos == "NOUN",]

条件抽出

• パイプ演算(%>%)は、次回紹介予定

日本語テキスト

cnlp_init_udpipe(model_name = "japanese")

res <- cnlp_annotate(input = txt_ja)

Processed document 10 of 19

res <- cnlp_annotate(input = txt_ja)

Processed document 10 of 19

res$token

複数条件抽出

• Adposition

res$token$upos
res$token[(res$token$upos == "NOUN")|(res$token$upos == "ADP"),]

スペイン語テキスト

cnlp_init_udpipe(model_name = "spanish")

Registered S3 method overwritten by 'data.table':
  method           from
  print.data.table     

条件抽出(記号以外)

res <- res$token[res$token$upos != "PUNCT",]
head(res)

頻度集計

sort(freqBycnlp, decreasing=TRUE)

             de              la              en              un               a              el             que              18 
             11               7               5               4               3               3               3               1 
            500         Al-Ahli       alrededor          armado          ataque     autoridades            Aviv          centro 
              1               1               1               1               1               1               1               1 
         cohete          contra         decenas           Desde        deslinda         después     diplomática              El 
              1               1               1               1               1               1               1               1 
       embajada  estadounidense      explotaron         fallido          Franja            Gaza           grupo              ha 
              1               1               1               1               1               1               1               1 
          Hamás           hasta           hecho        islámica          Israel         israelí        Jordania     lanzamiento 
              1               1               1               1               1               1               1               1 
         Líbano         llamado       madrugada manifestaciones          médico    movilización          muerte         mundial 
              1               1               1               1               1               1               1               1 
        octubre      palestinas        personas         provocó  representación      sanitarias              se           según 
              1               1               1               1               1               1               1               1 
      señalando              su        sufriera             Tel             vez           Yihad 
              1               1               1               1               1               1