Proof of Concept

# Proof of Concept
## ☩<br/>Cleaning, Matricing and Modeling a Chronology
### Dan Eness
### Truman State University
### 2020-12-09

---

background-image: url(https://upload.wikimedia.org/wikipedia/en/2/25/Foundation_gnome.jpg)

# Psychohistory - Social or Mathematical?

???

Image credit: [Wikipedia Commons](https://upload.wikimedia.org/wikipedia/en/2/25/Foundation_gnome.jpg)

---
class: inverse, left, top

# Psychohistory - Social or Mathematical?

Isaac Asimov's Foundation series, begun in 1939, is known for its unique take on *social science* in the application of "Psychohistory" but it is really a story about *data science.*
--

In the books, it is portrayed as a fictional science developed in the future Galactic Empire and combines: 
* history 
* sociology and 
* mathematical statistics
---
class: left, top
# The Cycles of History

Strauss and Howe popularized the idea of historical cycles, but the concept is an old one in the social sciences. Though the "linear" theory of history and time was an important innovation over traditional "seasonal" views of history, what was lost in Modern and Postmodern historical disciplines was critical pattern recognition: the ability to forecast the trajectory of future events based on past events.

>"...without some notion of historical recurrence, no one can meaningfully discuss the past at all. Why even talk about the founding (or decline) of a city, a victory (or defeat) in battle, the rise (or passing away) of a generation, **unless we accept that similar things have happened before and could happen again?"**(Strauss et al., 1998)

###### Strauss, W., & Howe, N. (1998). Winter Comes Again. In The fourth turning: an American prophecy (p. 13)., Broadway Books.
--

##### Social science has (typically) approached patterns in history qualitatively (even political polling tends to quantify "sentiment" rather than to predict policy.) Strauss and Howe examine histories, legends and cultural myths and look for "soft" and "intuitive" patterns as they chronologize human generations, using math and statistics as *logos* support for their narrative analysis.

---
class: left, middle

# The Data Source - Text
##James Ussher's *The Annals of the World*
* Published 1650 (Latin)
* Translated 1658 (English)
* A chronology of the Ancient World (4004BC-70AD)
* 18,000 Footnotes
* 100s of Primary, Secondary Sources
* Authors of Multiple Texts and Languages

but...

###### Ussher, J. Annals of the World : Ussher, James, 1581-1656. Retrieved October 26, 2020, from https://archive.org/details/AnnalsOfTheWorld/page/n27/mode/2up
---
class: inverse, center, middle

# ...Can You Graph It?

# ...Can You Predict It?

# ...Can You Detect the Cycles Statistically?

---

# Asimovian Data Principle #1

> "Gaal Dornick, using nonmathematical concepts, has defined psychohistory to be that branch of mathematics which deals with the reactions of human conglomerates to fixed social and economic stimuli… Implicit in all these definitions is the assumption that the human conglomerate being dealt with is sufficiently large for valid statistical treatment. The necessary size of such a conglomerate may be determined by Seldon’s First Theorem which …" - Foundation (Book 1 of The Foundation Trilogy)

Historical data must be of *sufficient size* to identify accurate patterns about the past and make predictions about future events.
---
background-image: url(https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcQCI34cii9MndRmPzfj2o1sMpiLjmRhyVlgq5lezMo40MwrjsZrf00IMxY5H2wZpOJHOwM5k_qs&usqp=CAc)
background-position: 50% 50%
class: left, top, inverse

### Sufficient Size?

--
class: left, bottom

# The Ussher text, printed, is nearly 1000 pages long in "Family Bible-sized" hardcover format

---
# Asimovian Data Principle #2
“That represents the condition of the Empire at present.” He waited.

>Gaal said finally, “Surely that is not a complete representation.”

>“No, not complete,” said Seldon. “I am glad you do not accept my word blindly. However, this is an approximation which will serve to demonstrate the proposition. Will you accept that?”

>“Subject to my later verification of the derivation of the function, yes.” Gaal was carefully avoiding a possible trap.

>“Good. Add to this the known probability of Imperial assassination, viceregal revolt, the contemporary recurrence of periods of economic depression, the declining rate of planetary explorations, the. . .”

>He proceeded. As each item was mentioned, new symbols sprang to life at his touch, and melted into the basic function which expanded and changed.

Historical data must include a **sufficient number of distinct variables** to refine the analysis of historical patterns. (The "Tidy" Principle)
---
###Raw Data (Begins at Line 471)
![The Raw Data begins nearly 500 lines in.](Raw Data Ussher.jpg)
---
###Same Data, Delimited, Regex, 1 Line equals 1 Row, NA to Dummy Text
![Every Row a Line](Every Row Data Ussher.jpg)
---
# Asimovian Data Principle #3
>“It will end well; almost certainly so for the project; and with reasonable probability for you.”

>“What are the figures?” demanded Gaal.

>“For the project, over 99.9%.”

>“And for myself?”

>"I am instructed that this probability is 77.2%.”

>“Then I’ve got better than one chance in five of being sentenced to prison or to death.”

>“The last is under one per cent.”

>“Indeed. Calculations upon one man mean nothing. You send Dr. Seldon to me.”

Historical analysis must include **sufficient modeled values to fairly represent** conclusions about mathematically derived patterns in history.
---
### This requires a "Tidy" Approach to Raw Text Data Cleaning, Framing and Modeling

* Find a text's "natural index."
* Every row is a line.
* Columns should be held to the minimum reasonable number.

--
#####Inspired by Hadley Wickham's Tidy Principles:

1. "Each variable forms a column.
2. Every observation forms a row.
3. Each type of observational unit forms a table."(Wickham, 2014)
###### Wickham, H. (2014). Tidy Data. Journal of Statistical Software, 59(10), 1 - 23. doi:http://dx.doi.org/10.18637/jss.v059.i10
---
class: inverse, middle, center

# Prove the Concept

---
# Initial Text Prep for .csv (delimited file) reading in R.

# Structure.

```
## tibble [7,005 x 3] (S3: tbl_df/tbl/data.frame)
##  $ Epoch : chr [1:7005] "The First Age of the World" "AgeBlank" "AgeBlank" "AgeBlank" ...
##  $ Dating: chr [1:7005] " 1a AM, 710 JP, 4004 BC" "DateBlank" "DateBlank" "DateBlank" ...
##  $ Event : chr [1:7005] "1. In the beginning God created the heaven and the earth. Ge 1:1 This beginning of time, according to our chron"| __truncated__ "2. On the first day Ge 1:1-5 of the world, on Sunday, October 23rd, God created the highest heaven and the ange"| __truncated__ "3. On the second day Ge 1:6-8 (Monday, October 24th) after the firmament or heaven was finished, the waters abo"| __truncated__ "4. On the third day Ge 1:9-13 (Tuesday, October 25th) when these waters below ran together into one place, the "| __truncated__ ...
```
---
# Tail.

<table>
 <thead>
  <tr>
   <th style="text-align:left;"> Epoch </th>
   <th style="text-align:left;"> Dating </th>
   <th style="text-align:left;"> Event </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> AgeBlank </td>
   <td style="text-align:left;"> DateBlank </td>
   <td style="text-align:left;"> 6995. Here Joseph ended the history of the destruction of Judea. He was captured in this war and made a freedman by Flarius Vespasian the Emperor and assumed the name of Flarius from his patron. </td>
  </tr>
  <tr>
   <td style="text-align:left;"> AgeBlank </td>
   <td style="text-align:left;"> DateBlank </td>
   <td style="text-align:left;"> 6996. Cornelius and Suetonius relate that there were 600,000 Jews killed in this war. Josephus a Jew, was a commander in that war and who deserved thanks and pardon from Vespasian for foretelling him that he should be emperor. He wrote that 1,000,000 perished by sword and famine, and of the rest of the Jews that were dispersed all over the world and put to death various ways to the number of 90,000. Orosius {Orosius, 1. 7. c. 9. } also stated the same. I cannot find the number of 600,000 of those who were killed in Suetonius' writings. In Josephus {Josephus, Wars, 1. 6. c. 17. &lt;c. 9. 1:749&gt;} the number of captives was 97,000 and the other number of 1,100,000 is only of those who perished in the six month siege of Jerusalem. Justius Lipsius made this catalogue from Josephus. {Lipsius, de. Constantia., 1. 2. c. 21.} of those who perished outside of Jerusalem during the whole seven years. </td>
  </tr>
  <tr>
   <td style="text-align:left;"> AgeBlank </td>
   <td style="text-align:left;"> DateBlank </td>
   <td style="text-align:left;"> 6997. At Jerusalem, first killed by the command of Floras 630 20,000 By the inhabitants of Caesarea in hatred to them and their religion At Scythopolis (a city of Syria) 30,000 At Askelon in Palestine by the inhabitants Likewise at Ptolemais 2,000 At Alexandria in Egypt under Tib. Alexander the President At Gamala as well killed as those who threw themselves down In the citadel of Masada, that slew themselves 2,500 50,000 8,400 2,000 At Damascus 10,000 At the taking of Joppa by Cesius Floras At a certain mountain called Cabulo In a fight at Askelon 10,000 By an ambush 8,000 At Aphaca when it was taken 15,000 Slain at mount Gerizim 1 1 ,600 At Jotapae, where Josephus was 30,000 At Joppa when it was taken were drowned Slain at Tarichaea 6,500 9,000 2,000 (The only survivors in the whole city were two women who were sisters.) When they forsook Giskala, killed in the flight Slain of the Gadarenes, besides an infinite number that leaped 13,000 into the river Slain in the villages of Idumea 10,000 At Gerizim 1,000 AtMacheron 1,700 In the wood of Jardes 3,000 960 In Cyrene by Catulus the Governor 3,000 Which number of the dead, being added to those who died &quot;&quot; v ' at the siege of Jerusalem Total 1,337,490&quot;&quot;&quot; </td>
  </tr>
  <tr>
   <td style="text-align:left;"> AgeBlank </td>
   <td style="text-align:left;"> DateBlank </td>
   <td style="text-align:left;"> 6998. An innumerable company were omitted who perished through famine, banishment and other miseries. (At the passover feast a few years earlier Josephus estimates there were about three million people in Jerusalem. See note on 4096b AM 6403. There were likely this many there for the passover when Titus started the seige. Most were unaccounted for. This would make the offical death toll low by at least two million. Editor.) </td>
  </tr>
  <tr>
   <td style="text-align:left;"> AgeBlank </td>
   <td style="text-align:left;"> DateBlank </td>
   <td style="text-align:left;"> 6999. Justus Tiberiensis in his chronicle of the Kings of the Jews showed that Agrippa the last king of the family of Herod, had his kingdom augmented by Vespasian, {Photius, Bibliotheca, cod. 33.} Dio related that he had praetorian honours given him. His sister Bernice who came to Rome with him lived in the palace. Titus was so in love with her that he made her believe he would marry her and she carried on all things as if she had been his wife. However, when Titus knew that the people of Rome did not take it well, he put her away. { Seutonius, Titus, c. 7 } {*Dio, 1. 65. 1:291 (Xiphiline ex Dio)} The observation of Josephus is very memorable {Josephus, Antiq., 1. 18. c. 7. &lt;c. 5. 1:485&gt;} about the rest of Herod's progeny that they all failed within one hundred years although they were very numerous. </td>
  </tr>
  <tr>
   <td style="text-align:left;"> AgeBlank </td>
   <td style="text-align:left;"> DateBlank </td>
   <td style="text-align:left;"> 7000. This was the end of the Jewish affairs and happened as predicted by Jesus in Mt 24. (See John Bray's book, &quot;&quot;Matthew 24 Fulfilled&quot;&quot;, for a most detailed discussion of these events. Editor.) vv And as he sat upon the mount of Olives, the disciples came unto him privately, saying, Tell us, when shall these things be? and what shall be the sign of thy coming, and of the end of the world? ... Verily I say unto you, THIS GENERATION shall not pass, till all these things be fulfilled.&quot;&quot; {Mt 24:3,34} FINIS &quot;&quot;&quot; </td>
  </tr>
</tbody>
</table>
---
###### Fill the "NA" (AgeBlank and DateBlank) with previous non-null date content. [Undated events are "between" or "in conjunction" with first non-null date.]Extract from Events an Index number (Index), Biblical Sources (TextSrc1) and Frequently Occurring Authors

```r
authorvar <- c("Josephus","Plutarch", "Herodotus","Aelian","Aeschylus","Agathias","Polyhistor","Ambrose","Marcellinus","Apion","Appian of Alexandria","Aristeas","Aristedes","Aristotle","Arrian","Asconius Pedianus","Athenaeus","Augustine","Aulus Gellius","Aurelius Victor","Ausonius","Basil the Great","Berosus","Cassidorus","Censorinus","Chrysostom","Cicero","Clement of Alexandria","Manetho","Livy","Lucan","Diodorus Siculous","Polybius","Suetonius","Valerius Maximus","Emilius Probus")
annals.data.na1 <- annals.data %>% dplyr::na_if("AgeBlank") %>% dplyr::na_if("DateBlank")
*annals.data.fill1 <- annals.data.na1 %>% fill(Epoch) %>% fill(Dating)
annals.index <- annals.data.fill1 %>%
  mutate(Index = as.numeric(str_extract(Event, "[0-9]+\\."))) %>%
  mutate(TextSrc1 = as.character(str_extract_all(Event, "[A-Z][a-z]+ [0-9]+:[0-9]++-[0-9]"))) %>%
  mutate(BibBk1 = as.character(str_extract(TextSrc1, "[A-Z][a-z]+"))) %>%
  mutate(AuthorSrc1 = as.character(str_extract(Event, (authorvar))))
```

```
## Warning in stri_extract_first_regex(string, pattern, opts_regex =
## opts(pattern)): longer object length is not a multiple of shorter object length
```
---

```r
ussh.ind <- annals.index %>% 
  mutate(EventTxt=(str_replace_all(Event, "[A-Z][a-z]+ [0-9]+:[0-9]++-[0-9]"," "))) %>%
  mutate(EventTxt=(str_replace_all(EventTxt, "[[:punct:]]"," "))) %>%
  mutate(EventTxt=(str_replace_all(EventTxt, "[0-9]+"," "))) %>%
  mutate(EventTxt=(str_replace_all(EventTxt, (authorvar)," "))) %>%
  mutate(Epoch=(str_replace_all(Epoch,"The Seventh Age of the World","7th Age"))) %>%
  mutate(Epoch=(str_replace_all(Epoch,"The Sixth Age of the World","6th Age"))) %>%
  mutate(Epoch=(str_replace_all(Epoch,"The Fifth Age of the World","5th Age"))) %>%
  mutate(Epoch=(str_replace_all(Epoch,"The Fourth Age of the World","4th Age"))) %>%  
  mutate(Epoch=(str_replace_all(Epoch,"The Third Age of the World","3rd Age"))) %>%
  mutate(Epoch=(str_replace_all(Epoch,"The Second Age of the World","2nd Age"))) %>%
  mutate(Epoch=(str_replace_all(Epoch,"The First Age of the World","1st Age"))) 
```

```
## Warning in stri_replace_all_regex(string, pattern,
## fix_replacement(replacement), : longer object length is not a multiple of
## shorter object length
```

```r
ussh.ind[ussh.ind == "character(0)" ] <- NA
```
---
#Tidied, Extracted, Replaced Dataframe

```r
knitr::kable(head(ussh.ind), format = 'html')
```

```r
knitr::kable(tail(ussh.ind), format = 'html')
```

<table>
 <thead>
  <tr>
   <th style="text-align:left;"> Epoch </th>
   <th style="text-align:left;"> Dating </th>
   <th style="text-align:left;"> Event </th>
   <th style="text-align:right;"> Index </th>
   <th style="text-align:left;"> TextSrc1 </th>
   <th style="text-align:left;"> BibBk1 </th>
   <th style="text-align:left;"> AuthorSrc1 </th>
   <th style="text-align:left;"> EventTxt </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> 7th Age </td>
   <td style="text-align:left;"> 4076c AM, 4786 JP, 73 AD </td>
   <td style="text-align:left;"> 6995. Here Joseph ended the history of the destruction of Judea. He was captured in this war and made a freedman by Flarius Vespasian the Emperor and assumed the name of Flarius from his patron. </td>
   <td style="text-align:right;"> 6995 </td>
   <td style="text-align:left;"> NA </td>
   <td style="text-align:left;"> NA </td>
   <td style="text-align:left;"> NA </td>
   <td style="text-align:left;"> Here Joseph ended the history of the destruction of Judea  He was captured in this war and made a freedman by Flarius Vespasian the Emperor and assumed the name of Flarius from his patron </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 7th Age </td>
   <td style="text-align:left;"> 4076c AM, 4786 JP, 73 AD </td>
   <td style="text-align:left;"> 6996. Cornelius and Suetonius relate that there were 600,000 Jews killed in this war. Josephus a Jew, was a commander in that war and who deserved thanks and pardon from Vespasian for foretelling him that he should be emperor. He wrote that 1,000,000 perished by sword and famine, and of the rest of the Jews that were dispersed all over the world and put to death various ways to the number of 90,000. Orosius {Orosius, 1. 7. c. 9. } also stated the same. I cannot find the number of 600,000 of those who were killed in Suetonius' writings. In Josephus {Josephus, Wars, 1. 6. c. 17. &lt;c. 9. 1:749&gt;} the number of captives was 97,000 and the other number of 1,100,000 is only of those who perished in the six month siege of Jerusalem. Justius Lipsius made this catalogue from Josephus. {Lipsius, de. Constantia., 1. 2. c. 21.} of those who perished outside of Jerusalem during the whole seven years. </td>
   <td style="text-align:right;"> 6996 </td>
   <td style="text-align:left;"> NA </td>
   <td style="text-align:left;"> NA </td>
   <td style="text-align:left;"> NA </td>
   <td style="text-align:left;"> Cornelius and Suetonius relate that there were     Jews killed in this war  Josephus a Jew  was a commander in that war and who deserved thanks and pardon from Vespasian for foretelling him that he should be emperor  He wrote that       perished by sword and famine  and of the rest of the Jews that were dispersed all over the world and put to death various ways to the number of      Orosius  Orosius        c       also stated the same  I cannot find the number of     of those who were killed in Suetonius  writings  In Josephus  Josephus  Wars        c     &lt;c        &gt;  the number of captives was     and the other number of       is only of those who perished in the six month siege of Jerusalem  Justius Lipsius made this catalogue from Josephus   Lipsius  de  Constantia         c      of those who perished outside of Jerusalem during the whole seven years </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 7th Age </td>
   <td style="text-align:left;"> 4076c AM, 4786 JP, 73 AD </td>
   <td style="text-align:left;"> 6997. At Jerusalem, first killed by the command of Floras 630 20,000 By the inhabitants of Caesarea in hatred to them and their religion At Scythopolis (a city of Syria) 30,000 At Askelon in Palestine by the inhabitants Likewise at Ptolemais 2,000 At Alexandria in Egypt under Tib. Alexander the President At Gamala as well killed as those who threw themselves down In the citadel of Masada, that slew themselves 2,500 50,000 8,400 2,000 At Damascus 10,000 At the taking of Joppa by Cesius Floras At a certain mountain called Cabulo In a fight at Askelon 10,000 By an ambush 8,000 At Aphaca when it was taken 15,000 Slain at mount Gerizim 1 1 ,600 At Jotapae, where Josephus was 30,000 At Joppa when it was taken were drowned Slain at Tarichaea 6,500 9,000 2,000 (The only survivors in the whole city were two women who were sisters.) When they forsook Giskala, killed in the flight Slain of the Gadarenes, besides an infinite number that leaped 13,000 into the river Slain in the villages of Idumea 10,000 At Gerizim 1,000 AtMacheron 1,700 In the wood of Jardes 3,000 960 In Cyrene by Catulus the Governor 3,000 Which number of the dead, being added to those who died &quot;&quot; v ' at the siege of Jerusalem Total 1,337,490&quot;&quot;&quot; </td>
   <td style="text-align:right;"> 6997 </td>
   <td style="text-align:left;"> NA </td>
   <td style="text-align:left;"> NA </td>
   <td style="text-align:left;"> NA </td>
   <td style="text-align:left;"> At Jerusalem  first killed by the command of Floras       By the inhabitants of Caesarea in hatred to them and their religion At Scythopolis  a city of Syria      At Askelon in Palestine by the inhabitants Likewise at Ptolemais     At Alexandria in Egypt under Tib  Alexander the President At Gamala as well killed as those who threw themselves down In the citadel of Masada  that slew themselves                 At Damascus     At the taking of Joppa by Cesius Floras At a certain mountain called Cabulo In a fight at Askelon     By an ambush     At Aphaca when it was taken     Slain at mount Gerizim        At Jotapae  where Josephus was     At Joppa when it was taken were drowned Slain at Tarichaea              The only survivors in the whole city were two women who were sisters   When they forsook Giskala  killed in the flight Slain of the Gadarenes  besides an infinite number that leaped     into the river Slain in the villages of Idumea     At Gerizim     AtMacheron     In the wood of Jardes       In Cyrene by Catulus the Governor     Which number of the dead  being added to those who died    v   at the siege of Jerusalem Total </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 7th Age </td>
   <td style="text-align:left;"> 4076c AM, 4786 JP, 73 AD </td>
   <td style="text-align:left;"> 6998. An innumerable company were omitted who perished through famine, banishment and other miseries. (At the passover feast a few years earlier Josephus estimates there were about three million people in Jerusalem. See note on 4096b AM 6403. There were likely this many there for the passover when Titus started the seige. Most were unaccounted for. This would make the offical death toll low by at least two million. Editor.) </td>
   <td style="text-align:right;"> 6998 </td>
   <td style="text-align:left;"> NA </td>
   <td style="text-align:left;"> NA </td>
   <td style="text-align:left;"> NA </td>
   <td style="text-align:left;"> An innumerable company were omitted who perished through famine  banishment and other miseries   At the passover feast a few years earlier Josephus estimates there were about three million people in Jerusalem  See note on  b AM    There were likely this many there for the passover when Titus started the seige  Most were unaccounted for  This would make the offical death toll low by at least two million  Editor </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 7th Age </td>
   <td style="text-align:left;"> 4076c AM, 4786 JP, 73 AD </td>
   <td style="text-align:left;"> 6999. Justus Tiberiensis in his chronicle of the Kings of the Jews showed that Agrippa the last king of the family of Herod, had his kingdom augmented by Vespasian, {Photius, Bibliotheca, cod. 33.} Dio related that he had praetorian honours given him. His sister Bernice who came to Rome with him lived in the palace. Titus was so in love with her that he made her believe he would marry her and she carried on all things as if she had been his wife. However, when Titus knew that the people of Rome did not take it well, he put her away. { Seutonius, Titus, c. 7 } {*Dio, 1. 65. 1:291 (Xiphiline ex Dio)} The observation of Josephus is very memorable {Josephus, Antiq., 1. 18. c. 7. &lt;c. 5. 1:485&gt;} about the rest of Herod's progeny that they all failed within one hundred years although they were very numerous. </td>
   <td style="text-align:right;"> 6999 </td>
   <td style="text-align:left;"> NA </td>
   <td style="text-align:left;"> NA </td>
   <td style="text-align:left;"> NA </td>
   <td style="text-align:left;"> Justus Tiberiensis in his chronicle of the Kings of the Jews showed that Agrippa the last king of the family of Herod  had his kingdom augmented by Vespasian   Photius  Bibliotheca  cod      Dio related that he had praetorian honours given him  His sister Bernice who came to Rome with him lived in the palace  Titus was so in love with her that he made her believe he would marry her and she carried on all things as if she had been his wife  However  when Titus knew that the people of Rome did not take it well  he put her away    Seutonius  Titus  c        Dio             Xiphiline ex Dio   The observation of Josephus is very memorable  Josephus  Antiq         c     &lt;c        &gt;  about the rest of Herod s progeny that they all failed within one hundred years although they were very numerous </td>
  </tr>
  <tr>
   <td style="text-align:left;"> 7th Age </td>
   <td style="text-align:left;"> 4076c AM, 4786 JP, 73 AD </td>
   <td style="text-align:left;"> 7000. This was the end of the Jewish affairs and happened as predicted by Jesus in Mt 24. (See John Bray's book, &quot;&quot;Matthew 24 Fulfilled&quot;&quot;, for a most detailed discussion of these events. Editor.) vv And as he sat upon the mount of Olives, the disciples came unto him privately, saying, Tell us, when shall these things be? and what shall be the sign of thy coming, and of the end of the world? ... Verily I say unto you, THIS GENERATION shall not pass, till all these things be fulfilled.&quot;&quot; {Mt 24:3,34} FINIS &quot;&quot;&quot; </td>
   <td style="text-align:right;"> 7000 </td>
   <td style="text-align:left;"> NA </td>
   <td style="text-align:left;"> NA </td>
   <td style="text-align:left;"> NA </td>
   <td style="text-align:left;"> This was the end of the Jewish affairs and happened as predicted by Jesus in Mt     See John Bray s book    Matthew   Fulfilled    for a most detailed discussion of these events  Editor   vv And as he sat upon the mount of Olives  the disciples came unto him privately  saying  Tell us  when shall these things be  and what shall be the sign of thy coming  and of the end of the world      Verily I say unto you  THIS GENERATION shall not pass  till all these things be fulfilled     Mt        FINIS </td>
  </tr>
</tbody>
</table>
---

# Tokenization of Dataframe

```r
tidy_annals <- ussh.ind %>%
  unnest_tokens(word, EventTxt)
data(stop_words)
tidy_annals <- tidy_annals %>%
  anti_join(stop_words)
```

```
## Joining, by = "word"
```

```r
tidy_annals[!(!is.na(tidy_annals$word) & tidy_annals$word=="" & tidy_annals$word==" "& tidy_annals$word=="   "), ]
```

```
## # A tibble: 264,579 x 8
##    Epoch  Dating     Event               Index TextSrc1 BibBk1 AuthorSrc1 word  
##    <chr>  <chr>      <chr>               <dbl> <chr>    <chr>  <chr>      <chr> 
##  1 1st A~ " 1a AM, ~ 1. In the beginnin~     1 <NA>     <NA>   <NA>       begin~
##  2 1st A~ " 1a AM, ~ 1. In the beginnin~     1 <NA>     <NA>   <NA>       god   
##  3 1st A~ " 1a AM, ~ 1. In the beginnin~     1 <NA>     <NA>   <NA>       creat~
##  4 1st A~ " 1a AM, ~ 1. In the beginnin~     1 <NA>     <NA>   <NA>       heaven
##  5 1st A~ " 1a AM, ~ 1. In the beginnin~     1 <NA>     <NA>   <NA>       earth 
##  6 1st A~ " 1a AM, ~ 1. In the beginnin~     1 <NA>     <NA>   <NA>       ge    
##  7 1st A~ " 1a AM, ~ 1. In the beginnin~     1 <NA>     <NA>   <NA>       begin~
##  8 1st A~ " 1a AM, ~ 1. In the beginnin~     1 <NA>     <NA>   <NA>       time  
##  9 1st A~ " 1a AM, ~ 1. In the beginnin~     1 <NA>     <NA>   <NA>       chron~
## 10 1st A~ " 1a AM, ~ 1. In the beginnin~     1 <NA>     <NA>   <NA>       happe~
## # ... with 264,569 more rows
```
---
# Head of Tokened Matrix

```r
tidy_annals %>%
  count(word, sort = TRUE) %>%
  filter(n %in% (300:100000))
```

```
## # A tibble: 109 x 2
##    word          n
##    <chr>     <int>
##  1 king       2725
##  2 army       1609
##  3 alexander  1528
##  4 city       1520
##  5 son        1488
##  6 time       1462
##  7 soldiers   1265
##  8 caesar     1261
##  9 war        1248
## 10 day        1174
## # ... with 99 more rows
```

```r
auth_counts = tidy_annals %>%
  group_by(AuthorSrc1, word) %>%
  count

auth_counts %>% 
  arrange(desc(n))
```

```
## # A tibble: 23,661 x 3
## # Groups:   AuthorSrc1, word [23,661]
##    AuthorSrc1 word          n
##    <chr>      <chr>     <int>
##  1 <NA>       king       2691
##  2 <NA>       army       1586
##  3 <NA>       alexander  1503
##  4 <NA>       city       1499
##  5 <NA>       son        1465
##  6 <NA>       time       1442
##  7 <NA>       soldiers   1250
##  8 <NA>       caesar     1231
##  9 <NA>       war        1228
## 10 <NA>       day        1167
## # ... with 23,651 more rows
```

```r
auth_dtm = auth_counts %>%
  cast_dfm(document=AuthorSrc1, term=word, value=n) %>%
  na.omit(auth_dtm)

auth_dtm
```

```
## Document-feature matrix of: 15 documents, 20,938 features (92.5% sparse).
##             features
## docs         abdera actors advice age ahead alex alexand alexander amanus
##   Arrian          1      1      1   1     1    1       1        13      1
##   Athenaeus       0      0      0   0     0    0       0         1      0
##   Berosus         0      0      0   0     0    0       0         2      0
##   Censorinus      0      0      0   0     0    0       0         0      0
##   Cicero          0      0      0   0     0    0       0         0      0
##   Herodotus       0      0      0   0     0    0       0         0      0
##             features
## docs         amounted
##   Arrian            1
##   Athenaeus         0
##   Berosus           0
##   Censorinus        0
##   Cicero            0
##   Herodotus         0
## [ reached max_ndoc ... 9 more documents, reached max_nfeat ... 20,928 more features ]
```

```r
ncol(auth_dtm)
```

```
## [1] 20938
```

```r
top100 = topfeatures(auth_dtm, 10)
top100
```

```
##      king      army alexander      city       son      time  soldiers    caesar 
##      2725      1609      1528      1520      1488      1462      1265      1261 
##       war       day 
##      1248      1174
```

```r
ussh_10 = LDA(convert(auth_dtm, to = "topicmodels"), k = 5, control=list(seed=1234))
ushauth <- get_terms(ussh_10, 10)

ussh_dtm = auth_dtm %>%
  dfm_wordstem()

ussh_dtm
```

```
## Document-feature matrix of: 15 documents, 15,807 features (92.2% sparse).
##             features
## docs         abdera actor advic age ahead alex alexand amanus amount anaxarchus
##   Arrian          1     1     1   1     1    1      14      1      1          1
##   Athenaeus       0     0     0   0     0    0       1      0      0          0
##   Berosus         0     0     0   0     0    0       2      0      0          0
##   Censorinus      0     0     0   0     0    0       0      0      0          0
##   Cicero          0     0     0   0     0    0       0      0      0          0
##   Herodotus       0     0     0   0     0    0       0      0      0          0
## [ reached max_ndoc ... 9 more documents, reached max_nfeat ... 15,797 more features ]
```

```r
ncol(ussh_dtm)
```

```
## [1] 15807
```

```r
ushauth
```

```
##       Topic 1     Topic 2     Topic 3     Topic 4     Topic 5      
##  [1,] "king"      "king"      "king"      "city"      "king"       
##  [2,] "city"      "kingdom"   "day"       "day"       "time"       
##  [3,] "time"      "caesar"    "alexander" "alexander" "son"        
##  [4,] "army"      "son"       "jews"      "army"      "antony"     
##  [5,] "left"      "romans"    "war"       "josephus"  "called"     
##  [6,] "soldiers"  "herod"     "caesar"    "dio"       "mithridates"
##  [7,] "caesar"    "war"       "romans"    "rome"      "ships"      
##  [8,] "alexander" "called"    "vv"        "brother"   "army"       
##  [9,] "ships"     "plutarch"  "herod"     "death"     "egypt"      
## [10,] "country"   "alexander" "friends"   "ptolemy"   "god"
```

```r
t(topics(ussh_10, 10))
```

```
##                  [,1] [,2] [,3] [,4] [,5]
## Arrian              4    1    3    5    2
## Athenaeus           4    3    2    5    1
## Berosus             4    2    1    5    3
## Censorinus          3    1    5    4    2
## Cicero              5    1    4    2    3
## Herodotus           2    4    5    1    3
## Josephus            4    2    1    5    3
## Livy                3    5    2    4    1
## Lucan               3    2    1    4    5
## Manetho             1    5    4    2    3
## Plutarch            4    1    5    2    3
## Polybius            4    3    5    2    1
## Suetonius           4    1    2    3    5
## Valerius Maximus    3    4    1    2    5
## <NA>                2    1    5    3    4
```
---

```
## # A tibble: 18 x 3
## # Groups:   AuthorSrc1, Epoch [18]
##    AuthorSrc1       Epoch       n
##    <chr>            <chr>   <int>
##  1 Josephus         7th Age   210
##  2 Suetonius        7th Age    70
##  3 Arrian           6th Age   329
##  4 Athenaeus        6th Age    87
##  5 Berosus          6th Age    57
##  6 Censorinus       6th Age    66
##  7 Cicero           6th Age   370
##  8 Herodotus        6th Age    85
##  9 Josephus         6th Age   434
## 10 Livy             6th Age   719
## 11 Lucan            6th Age    76
## 12 Plutarch         6th Age  1186
## 13 Polybius         6th Age   125
## 14 Suetonius        6th Age    44
## 15 Valerius Maximus 6th Age   115
## 16 Berosus          5th Age    12
## 17 Herodotus        5th Age    91
## 18 Manetho          2nd Age     3
```

```
## Document-feature matrix of: 15 documents, 7 features (76.2% sparse).
##             features
## docs         6th Age 5th Age 7th Age 2nd Age 1st Age 3rd Age 4th Age
##   Arrian         329       0       0       0       0       0       0
##   Athenaeus       87       0       0       0       0       0       0
##   Berosus         57      12       0       0       0       0       0
##   Censorinus      66       0       0       0       0       0       0
##   Cicero         370       0       0       0       0       0       0
##   Herodotus       85      91       0       0       0       0       0
## [ reached max_ndoc ... 9 more documents ]
```

```
## [1] 7
```

---

```
## Document-feature matrix of: 42 documents, 7 features (76.9% sparse).
##      features
## docs  2nd Age 7th Age 5th Age 4th Age 6th Age 3rd Age 1st Age
##   Ac        1      78       0       0       0       0       0
##   Am        0       0       3       0       0       0       0
##   Bar       0       0       2       0       0       0       0
##   Ch        0       0      27       6       0       0       0
##   Co        0       5       0       0       0       0       0
##   Col       0       1       0       0       0       0       0
## [ reached max_ndoc ... 36 more documents ]
```

```
## [1] 7
```
---

---
### Prove the Concept (Authors Can Be Clustered By Era)
![](PDAT-620G-Capstone-Slideshow-Eness-Proof-of-Concept_files/figure-html/gridarr-1.png)
---
### Prove the Concept (Texts Can Be Clustered by Era)
![](PDAT-620G-Capstone-Slideshow-Eness-Proof-of-Concept_files/figure-html/textarr-1.png)
---
# Tasks to Resolve Before Truly Modeling "Cycles in Psychohistory"

1. To _all or not?, the issue of extraction strategy

1. Source Strategies - Authors and Books confusion for regex search patterns

1. Dating Decisions - AM? JP? BC? NK? SK? AD?

1. Topic Clustering assessment and decisions

1. Prediction insights
]

1. **Extraction and replacement results differ**

1. **Abbreviations, different spellings and historic similarities**

1. **Different timestamping yields different variables**

1. **Number of Clusters, Meaning, Boundaries**

1. **ML opportunities**
]

---

# Thanks!

References:

###### Asimov, I. (1951). Foundation. Doubleday.

###### Strauss, W., & Howe, N. (1998). Winter Comes Again. In The fourth turning: an American prophecy (pp. 1-382)., Broadway Books.

###### Ussher, J. Annals of the World : Ussher, James, 1581-1656. Retrieved October 26, 2020, from https://archive.org/details/AnnalsOfTheWorld/page/n27/mode/2up

###### Wickham, H. (2014). Tidy Data. Journal of Statistical Software, 59(10), 1 - 23. doi:http://dx.doi.org/10.18637/jss.v059.i10

Slides created via the R package [**xaringan**](https://github.com/yihui/xaringan).