Analysis of tf-idf in Civil Cases
in Federal District Court of New Hampshire

Import data and merge

Create data for nature of suit, data_NOS

The idea of tf-idf is to find the important words for the content of each document 1) by decreasing the weight for commonly used words and 2) increasing the weight for words that are not used very much in a collection or corpus of documents, in this case, the group of nature of suits as a whole.

tf-idf by nature of suit

## 
##  CIVIL RIGHTS ADA OTHER       CIVIL RIGHTS JOBS               INSURANCE 
##                       3                       8                       4 
##      OTHER CIVIL RIGHTS  OTHER CONTRACT ACTIONS   OTHER PERSONAL INJURY 
##                       9                       8                       7 
## OTHER STATUTORY ACTIONS 
##                       8

Interpretation

• What measuring tf-idf has done here is show us that similar language was used across the four nature of suits, and what distinguishes one nature of suit from the rest within the collection of the cases are the proper nouns, the names of people and places.
• This is the point of tf-idf; it identifies words that are important to one document within a collection of documents.
• All the legal jargons that are common in all legal documents (such as defendants, plaintiffs, and courts) will be ranked low, while terms that appear freqent in a single nature of suit but not common in few other nature of suits will be ranked high. In other words, we could identify risk factors (frequent words) for the nature of suit after removing the names of people and businesses.
• For example, it’s telling to see “police” and “arrest” in the Civil Rights: Other category!

Create data for causes of action, data_COA

The idea of tf-idf is to find the important words for the content of each document 1) by decreasing the weight for commonly used words and 2) increasing the weight for words that are not used very much in a collection or corpus of documents, in this case, the group of causes of actions as a whole.

tf-idf by causes of action

## 
##               28:1332 Diversity-Insurance Contract 
##                                                  2 
## 28:1441  Petition for Removal - Employment Discrim 
##                                                  2 
##                   28:1332 Diversity-Other Contract 
##                                                  1 
##                           42:1983 Civil Rights Act 
##                                                  6 
##               15:0045 Federal Trade Commission Act 
##                                                  1 
##                  28:1332 Diversity-Personal Injury 
##                                                  4 
##                   15:1692 Fair Debt Collection Act 
##                                                  3 
##               28:1332 Diversity-Breach of Contract 
##                                                  3 
##   28:1331 Fed. Question: Employment Discrimination 
##                                                  2 
##           42:12101 Americans With Disabilities Act 
##                                                  4 
##             28:1332  Diversity - Legal Malpractice 
##                                                  2 
##                 28:1332 Diversity-Contract Dispute 
##                                                  2 
##           47:227 Telephone Consumer Protection Act 
##                                                  2 
##                 28:1332 Diversity-Contract Default 
##                                                  1 
##                  28:1343 Violation of Civil Rights 
##                                                  1 
##             28:1331  Federal Question:  Bivens Act 
##                                                  1 
##   28:1441 Petition for Removal- Breach of Contract 
##                                                  1 
##                   42:2000 Job Discrimination (Sex) 
##                                                  2 
##                               28:1983 Civil Rights 
##                                                  1 
##                       28:2201 Declaratory Judgment 
##                                                  1 
##   28:1441 Petition for Removal- Insurance Contract 
##                                                  2 
##                              28:1331 Fed. Question 
##                                                  1 
##     28:1441 Petition for Removal- Civil Rights Act 
##                                                  1 
##                          42:1396 - Tort Negligence 
##                                                  1

Interpretation

• What measuring tf-idf has done here is show us that similar language was used across the four causes of actions, and what distinguishes one causes of action from the rest within the collection of the cases are the proper nouns, the names of people and places.
• This is the point of tf-idf; it identifies words that are important to one document within a collection of documents.
• All the legal jargons that are common in all legal documents (such as defendants, plaintiffs, and courts) will be ranked low, while terms that appear freqent in a single causes of action but not common in few other causes of actions will be ranked high. In other words, we could identify risk factors (frequent words) for the causes of action after removing the names of people and businesses.
• For example, it’s telling to see “police” and “arrest” in the Civil Rights: Other category!