Introduction to R, RStudio and Python
Overview I
Learning a scripting language for statistical analysis is important for reproducible and documentable, workflows.
R and Python are similar but definitions somewhat differ on some fundemental definitions. We will not go into details.
R is written with C, Fortran and R itself.
It is mainly a high level imperative language.
The order you write a line of code matters.
This is a bit more nuanced but it will help you understand the scripts better.
Overview II
Before we dive into statistical concepts we need to clarify a set of concepts.
Classes, Objects, Methods, functions, data structures and types, for loops, vectorization and broadcasting are the main topics we will discuss. Recall these definitions can be different given the scripting language but if you learn one set of definitions it will be easier to learn the other(s).
You will not have to memorize any of these concepts for multiple choice tests!
What are classes and objects
- In general:
Classes are descriptions of how data is organized and what methods are available to call from it.
An examples of a class is data.frame, which represents a rectangular structure that can hold different types of data. matrix is another rectangular structure but it can only hold one type of data.
A helpful object oriented definition for an object is an instance of a class. This is a bit different in R compared to Java or Python.
What are classes and objects
- "In R, class information is usually stored as an attribute. An object can have multiple class labels, and these labels can be used by R to determine which method should handle the object."
What are classes and objects
What are functions and methods?
A function is a piece of code that performs an operation (i.e.: mean, median, print).
A method in R is a function designed to handle objects of a particular class. “In Python, it is generally a function defined within a class and called through an object.”
F.Name L.Name
1 NA NA
2 NA NA
3 NA NA
4 NA NA
5 NA NA [1] print.acf*
[2] print.activeConcordance*
[3] print.AES*
[4] print.anova*
[5] print.aov*
[6] print.aovlist*
[7] print.ar*
[8] print.Arima*
[9] print.arima0*
[10] print.AsIs
[11] print.aspell*
[12] print.aspell_inspect_context*
[13] print.bibentry*
[14] print.Bibtex*
[15] print.bitstring*
[16] print.browseVignettes*
[17] print.by
[18] print.changedFiles*
[19] print.check_bogus_return*
[20] print.check_code_usage_in_package*
[21] print.check_compiled_code*
[22] print.check_demo_index*
[23] print.check_depdef*
[24] print.check_details*
[25] print.check_details_changes*
[26] print.check_doi_db*
[27] print.check_dotInternal*
[28] print.check_make_vars*
[29] print.check_nonAPI_calls*
[30] print.check_package_code_assign_to_globalenv*
[31] print.check_package_code_attach*
[32] print.check_package_code_data_into_globalenv*
[33] print.check_package_code_startup_functions*
[34] print.check_package_code_syntax*
[35] print.check_package_code_unload_functions*
[36] print.check_package_compact_datasets*
[37] print.check_package_CRAN_incoming*
[38] print.check_package_datalist*
[39] print.check_package_datasets*
[40] print.check_package_depends*
[41] print.check_package_description*
[42] print.check_package_description_encoding*
[43] print.check_package_license*
[44] print.check_packages_in_dir*
[45] print.check_packages_used*
[46] print.check_po_files*
[47] print.check_pragmas*
[48] print.check_Rd_line_widths*
[49] print.check_Rd_metadata*
[50] print.check_Rd_xrefs*
[51] print.check_RegSym_calls*
[52] print.check_S3_methods_needing_delayed_registration*
[53] print.check_so_symbols*
[54] print.check_T_and_F*
[55] print.check_url_db*
[56] print.check_vignette_index*
[57] print.checkDocFiles*
[58] print.checkDocStyle*
[59] print.checkFF*
[60] print.checkRd*
[61] print.checkRdContents*
[62] print.checkReplaceFuns*
[63] print.checkS3methods*
[64] print.checkTnF*
[65] print.checkVignettes*
[66] print.citation*
[67] print.cli_ansi_html_style*
[68] print.cli_ansi_string*
[69] print.cli_ansi_style*
[70] print.cli_boxx*
[71] print.cli_diff_chr*
[72] print.cli_doc*
[73] print.cli_progress_demo*
[74] print.cli_rule*
[75] print.cli_sitrep*
[76] print.cli_spark*
[77] print.cli_spinner*
[78] print.cli_tree*
[79] print.codoc*
[80] print.codocClasses*
[81] print.codocData*
[82] print.colorConverter*
[83] print.compactPDF*
[84] print.condition
[85] print.connection
[86] print.CRAN_package_reverse_dependencies_and_views*
[87] print.data.frame
[88] print.Date
[89] print.default
[90] print.dendrogram*
[91] print.density*
[92] print.difftime
[93] print.dist*
[94] print.Dlist
[95] print.DLLInfo
[96] print.DLLInfoList
[97] print.DLLRegisteredRoutines
[98] print.document_context*
[99] print.document_position*
[100] print.document_range*
[101] print.document_selection*
[102] print.dummy_coef*
[103] print.dummy_coef_list*
[104] print.ecdf*
[105] print.eigen
[106] print.evaluate_evaluation*
[107] print.factanal*
[108] print.factor
[109] print.family*
[110] print.fileSnapshot*
[111] print.findLineNumResult*
[112] print.FontVariation*
[113] print.formula*
[114] print.free1way*
[115] print.ftable*
[116] print.function
[117] print.getAnywhere*
[118] print.glm*
[119] print.hashtab*
[120] print.hclust*
[121] print.help_files_with_topic*
[122] print.hexmode
[123] print.HoltWinters*
[124] print.hsearch*
[125] print.hsearch_db*
[126] print.htest*
[127] print.html*
[128] print.html_dependency*
[129] print.htmltools.selector*
[130] print.htmltools.selector.list*
[131] print.infl*
[132] print.integrate*
[133] print.isoreg*
[134] print.json*
[135] print.key_missing*
[136] print.kmeans*
[137] print.knitr_kable*
[138] print.Latex*
[139] print.LaTeX*
[140] print.libraryIQR
[141] print.listof
[142] print.lm*
[143] print.loadings*
[144] print.loess*
[145] print.logLik*
[146] print.ls_str*
[147] print.medpolish*
[148] print.MethodsFunction*
[149] print.mtable*
[150] print.NativeRoutineList
[151] print.news_db*
[152] print.nls*
[153] print.noquote
[154] print.numeric_version
[155] print.object_size*
[156] print.octmode
[157] print.otel_counter*
[158] print.otel_gauge*
[159] print.otel_histogram*
[160] print.otel_logger*
[161] print.otel_logger_provider*
[162] print.otel_meter*
[163] print.otel_meter_provider*
[164] print.otel_span*
[165] print.otel_span_context*
[166] print.otel_tracer*
[167] print.otel_tracer_provider*
[168] print.otel_up_down_counter*
[169] print.packageDescription*
[170] print.packageInfo
[171] print.packageIQR*
[172] print.packageStatus*
[173] print.paged_df*
[174] print.pairwise.htest*
[175] print.person*
[176] print.POSIXct
[177] print.POSIXlt
[178] print.power.htest*
[179] print.ppr*
[180] print.prcomp*
[181] print.princomp*
[182] print.proc_time
[183] print.quosure*
[184] print.quosures*
[185] print.raster*
[186] print.Rconcordance*
[187] print.Rd*
[188] print.recordedplot*
[189] print.restart
[190] print.RGBcolorConverter*
[191] print.RGlyphFont*
[192] print.rlang:::list_of_conditions*
[193] print.rlang_box_done*
[194] print.rlang_box_splice*
[195] print.rlang_data_pronoun*
[196] print.rlang_dict*
[197] print.rlang_dyn_array*
[198] print.rlang_envs*
[199] print.rlang_error*
[200] print.rlang_fake_data_pronoun*
[201] print.rlang_lambda_function*
[202] print.rlang_message*
[203] print.rlang_trace*
[204] print.rlang_warning*
[205] print.rlang_zap*
[206] print.rle
[207] print.rlib_bytes*
[208] print.rlib_error_3_0*
[209] print.rlib_trace_3_0*
[210] print.roman*
[211] print.SavedPlots*
[212] print.scalar*
[213] print.sessionInfo*
[214] print.shiny.tag*
[215] print.shiny.tag.env*
[216] print.shiny.tag.list*
[217] print.shiny.tag.query*
[218] print.simple.list
[219] print.smooth.spline*
[220] print.socket*
[221] print.srcfile
[222] print.srcref
[223] print.stepfun*
[224] print.stl*
[225] print.StructTS*
[226] print.subdir_tests*
[227] print.summarize_CRAN_check_status*
[228] print.summary.aov*
[229] print.summary.aovlist*
[230] print.summary.ecdf*
[231] print.summary.free1way*
[232] print.summary.glm*
[233] print.summary.lm*
[234] print.summary.loess*
[235] print.summary.manova*
[236] print.summary.nls*
[237] print.summary.packageStatus*
[238] print.summary.ppr*
[239] print.summary.prcomp*
[240] print.summary.princomp*
[241] print.summary.table
[242] print.summary.warnings
[243] print.summaryDefault
[244] print.table
[245] print.tables_aov*
[246] print.terms*
[247] print.ts*
[248] print.tskernel*
[249] print.TukeyHSD*
[250] print.tukeyline*
[251] print.tukeysmooth*
[252] print.undoc*
[253] print.vignette*
[254] print.warnings
[255] print.xfun_md_viewable*
[256] print.xfun_raw_string*
[257] print.xfun_record_results*
[258] print.xfun_rename_seq*
[259] print.xfun_strict_list*
[260] print.xgettext*
[261] print.xngettext*
[262] print.xtabs*
see '?methods' for accessing help and source code- Functions are also a set of instructions but are not necessarily associated with classes.
Example Function
- Below, the rnorm function generates 5 normally distributed with mean (\(\mu\)) 0 and standard deviation (\(\sigma\)) 1.
What are functions and methods?
In R many of the functions have default parameters such as the rnorm has mean and standard deviation as 0 and 1.
To change it, we need to specify new parameter values within the parenthesis of the function.
Data types in R? Atomic and Recursive Classes
Data in R can be either atomic, containing only one type of data limited to character, logical, numeric, integer (vectors, matrix, arrays)
or it can be recursive (lists, data frames) that can contain any type of data.
Atomic Classes: Character
- Character: These are strings. You can do count operations on them.
- Familiarize yourself with error messages.
Atomic Classes: Numeric/Double
- Numeric/Double: real numbers, in addition to counting you can do arithmetic operations with.
Atomic Classes: Integer
- Integer: Integers that can be stored with less space than double. Arithmetic operations on objects stored as integers are usually coerced into double.
Atomic Classes: Int vs Double
Coerce object y to be an integer and compare sizes.
[1] TRUE[1] FALSE96 bytes80 bytesHow about when we increase the number of values from 5 to 10.
Atomic Classes: Int vs Double
Recall that when we had 5 single digit numbers, the object defined with double was 96 bytes and the object defined with integers is 80 bytes. When we increase the size of these objects to 10 single digit numbers the respective sizes jump to 176 and and 96. This is a good reason why knowing about these differences are important when you are building models with large/big data.
Atomic Classes: Logical
- Logical: TRUE/FALSE values, usually obtained as a result of comparison p>0.05. These logical values translate to 1 for TRUE and 0 for FALSE.
[1] FALSE FALSE TRUE FALSE TRUE80 bytes[1] 2- See above how the vector of logical values summed up to 2. We will use this quite often.
Atomic Classes: Others
Atomic classes that we will not use in this class.
Complex: A number with an imaginary component. For instance “i”.
https://www.geeksforgeeks.org/imaginary-numbers/
- Raw: Byte representation of data. We will not be using this in class.
https://stat.ethz.ch/R-manual/R-devel/library/base/html/raw.html
Objects: Data Structures
Vectors: Single dimensional collection of data in the same type
lists: Single dimensional collection of data in any type.
array: Single or more dimensional data of same type
matrix: Two dimensional arrays.
data.frame: Two dimensional table composed of vectors
Factors: Factor is a data structure that can be used to represent nominal values via a set of fixed number of levels and can be used as a variable in a data.frame. https://www.geeksforgeeks.org/r-factors/ https://r4ds.hadley.nz/factors.html
Objects: Others
- Regression/Hypothesis: These objects are going to hold information about the statistical procedures. We can obtain the information directly or use methods and functions to access this information.
(Intercept) X
0.04649317 0.41296550 [1] 0.2752237What type of a language is R?
Imperative vs Declarative (ignoring functional and object oriented)
- R is mainly an imperative language.
- Each line has an instruction that has an effect on the program flow.
- SQL in contrast is a declarative language. You declare what you want, from where you want it.
Definitions for language types are not black and white and most languages have a bit of both.
RStudio (Posit) is a GUI
- Supports multiple languages not only R.
Basic Concepts in Scripting- Control Flows
- for loops – for(i in 1:END) { Statements to be Repeated Written Here }
– for(i in START:END){ i is the index variable it will take values from START (say 1) to the value END (say 10) represents values that i is going to take, unless otherwise scripted, iterations happen by 1 unit at a time }
- Other control structures exist such as While
Basic Concepts in Scripting: For Loops
A for loop in declares a variable to index. In the example below this is “i”.
Everything within brackets is run for the times specified within the parenthesis. In the example below we specify there will be a total of 3 runs.
As the statements run, the value “i” changes from 1 to 3.
Basic Concepts in Scripting: For Loops
- We use loops to repetitively run statements. Below is a hypothetical set of statements.
- In the example below
The data frame ccFraud contains the dependent variable in the very last column, rest are independent variables.AIC is defined as a single dimension array with the number of elements equivalent to the number of independent variables.We use as independent variables each column separately, run a logistic regression and obtain model comparison measure AIC, store it in the object AIC.
Basic Concepts in Scripting: For Loops
#AIC object is defined
AIC=array(dim=c(ncol(ccFraud)-1))
END=ncol(ccFraud)-1
#For loop is going to run from i being equal to 1 to number of columns - 1.
for(i in 1:END){
#Store summary logistic regression information
#The log_regression object is going to be rewritten at every iteration.
log_regression= summary(glm(Fraud~ccFraud[,i],family=binomial(link='logit'),data=ccFraud))
#AIC is populated at i'th element
AIC[i]=log_regression$aic
}Basic Concepts in Scripting: Linear Algebra Vectorization
“Vectorization means applying an operation to an entire vector or other data structure without explicitly writing a loop over its individual elements.”
https://www.sciencedirect.com/topics/computer-science/vectorization#
Basic Concepts in Scripting: Vectorization
So what happens in the chunk of code above? A vector is added to another vector (in r, a scalar is a vector)
| 1 | 3 | 5 |
|---|---|---|
| + | + | + |
| 6 | 4 | 2 |
| = | = | = |
| 7 | 7 | 7 |
Basic Concepts in Scripting: Vectorization
How do we calculate residuals in regression (Observed - Predicted)
This is a vectorized operation, operations are made elementwise without having to write an explicit loop.
- It would be useful to dig into this topic https://www.noamross.net/archives/2014-04-16-vectorization-in-r-why/
Basic Concepts in Scripting: For Loops vs Vectorization
- R “For” loops are not as efficient as vectorization but vectorization can not be used for every operation not every function is vectorized.
user system elapsed
0.58 0.01 0.61 user system elapsed
0.55 0.02 0.55 Basic Concepts in Scripting: For Loops vs Vectorization
- However vectorized functions, can provide efficiencies.
Basic Concepts in Scripting: Recycling/Broadcasting
When you operate on two objects, matching the dimension of one object to the other for the operations is called broadcasting. Recycling works in a similar fashion. In R, for vectors, the dimensions of the smaller object is extended until it has the same dimension as the larger object.
Base R uses recycling https://stackoverflow.com/questions/42893238/recycling-higher-dimensional-arrays
Basic Concepts in Scripting: Recycling
- When you add a vector to a scalar, the scalar is recycled to match the dimensions of the vector. Note that in R, there is no separate scalar data type. A single number is a numeric vector of length 1
| 1 | 3 | 5 |
|---|---|---|
| + | + | + |
| 1 | 1 | 1 |
| = | = | = |
| 2 | 4 | 6 |
Basic Concepts in Scripting: Recycling
Adding two vectors of sizes that do not conform to each other. Objects a and b are vectors. Read the warning and think about what that means. Would you get the same warning if the length of a was 4?
Warning in a + b: longer object length is not a multiple of shorter object
length[1] 6 10 10| 1 | 3 | 5 |
|---|---|---|
| + | + | + |
| 5 | 7 | 5 |
| = | = | = |
| 6 | 10 | 10 |
Basic Concepts in Scripting: Recycling
How does this work with multiplication?
Warning in b * X: longer object length is not a multiple of shorter object
length[1] 25 70 10 -35 15| 5 | 7 | 5 | 7 | 5 |
|---|---|---|---|---|
| \(\times\) | \(\times\) | \(\times\) | \(\times\) | \(\times\) |
| 5 | 10 | 2 | -5 | 3 |
| = | = | = | = | = |
| 25 | 70 | 10 | -35 | 15 |
Arrays of Higher Dimensions
Create 2, 3 dimensional arrays with each dimension having 2,3 and 2 as sizes of these dimensions. We do not specify any data in the arrays therefore it will have NA values.
3-d Arrays
Print a_1 before we populate it.
Nested Loops
We will populate the arrays arbitrarily with a nested for loop structure.
What are the values populating the arrays?
Populated 3-d Arrays
The object a_1 has the following elements when k=1 and k=2:
Recycling for Higher Dimensions
Add to the array, vector d.
The vector d is recycled by filling columns to conform to the dimensions of a_1 the following structure for k=1 and k=2:
| 1 | 10 | 5 |
| 5 | 1 | 10 |
Resulting in:
3-d array + Vector Result:
The result is:
Adding Higher Dimensional Arrays
- We can add arrays of conforming dimensions
Errors in Recycling
What happens when we add two arrays which do not have the same dimensions.
If we convert object d to an array, it does not get recycled from a dimension of 1 to dimensions of 3 and you get an error.
[1] 1[1] 3Error in `a_1 + d`:
! non-conformable arraysThe smaller array is not reshaped and populated to conform to the dimensions of the larger object.
Coercion in R
Coercion is the conversion of the type or structure of a data/object due to operations done on them. Another way to describe it is: the conversion of an object type to another if an operation requires it.
We actually have already seen an example of this conversion.
Coercion in R
- In the example above, in order for sum function to work the logical types have been coerced into integers of 1 (TRUE) or 0 (FALSE).
https://www2.stat.duke.edu/courses/Fall20/sta523/slides/lecture/lec_01.html#9
- After going over these basic concepts we are ready to learn how to simulate coin tosses from probability distributions, revisit Central Limit Theorem (my favorite after Evolution), relearn Hypothesis Testing, type 1 and type 2 errors.
References
https://www.geeksforgeeks.org/classes-in-r-programming/
https://rpubs.com/Thinklabz/data_types_and_objects
https://caml.inria.fr/pub/docs/oreilly-book/html/book-ora140.html
Python specific discussion:
Practically is a more general purpose use but in my experience for the tasks in this class it requires more code for the similar operations.
Python can handle larger quantities of data by using explicitly declared functions for incrementally reading lines or chunks into the RAM instead of putting it all as one big block of data. Similar approaches are available in R, although they may be less convenient in some workflows.
How do we operate Python from RStudio? (needs work will come back to this before september 3rd.)
Python in RStudio
- I used ANACONDA GUI with CONDA before which allowed centrally managed
- Application and Package installations.
- RStudio for R and Spyder for Python (I do despise Jupyter but use it if you must)
- What was simple became too complicated for class purposes that focuses on methods rather than software.
- So now we will configure RStudio (Later I will change to Positron) for Python.
Python in RStudio
I am assuming you have already installed current version of R, Python and RStudio.
For Python/R needs a mechanism to execute scripts you write.
An interpreter is one mechanism used to execute programming language code. You will need to identify where the interpreter is in RStudio (this would already be done for R)
The Python interpreter in Windows have .exe extensions for instance python.exe, in LINUX and MAC we do not have .exe extensions so you will be looking for python to find where the interpreter is.
Configuring Python
Microsoft aliasing has to be changed for Python
- Windows START>>Manage app execution aliases, click on it.
- Python Python3 toggle them off.
To create a virtual environment we need to have a folder path and install a folder there.
Do NOT use One Drive. Its automatic file synchronization can create problems for virtual environments. Use a local folder instead.
“C:\Users\Administrator\Desktop\Teaching\3339\Fall26”
Global Options==>Python
Python Interpreter selected
Python in RStudio
Python in RStudio
- Restart RStudio.
- In the console of RStudio run: install.packages(“reticulate”)
- Setting the PATH for interpreter as below, is optional if you did the steps above in RStudio.
Python in RStudio
- In R run:
- You can now use RStudio for Python however we first need to create and activate virtual environments.
Virtual Environments
As mentioned above both R and Python has packages. For many common statistical tasks Python requires a lot more packages.
These packages have functions and methods that you need for your tasks.
All live languages are updated at intervals that the maintainers of those packages determine.
Python packages have historically been updated much more frequently compared to R.
Tinkering to make things is cool but scripts/programs break if you do not keep up.
Virtual Environments
Virtual environments allow you to manage the versions of the packages you use.
Once a program runs in the virtual environment, as long as you do not update the packages in it, it will likely run again.
We could optionally use a tool (GitBash) in terminal to create VEs.
GitBash
If you have Windows you will have to install Git-Bash, if you have MAC or Linux you already have a system that will operate with the same set of instructions.
Before installation learn whether your CPU architecture is X64 or ARM64.
Start>>Settings>>System>>About
X64
ARM64
Optional Installing GitBash
Go to: https://git-scm.com/ and install for windows.
Use defaults, if you remember change the VIM editor (one of the prompts) to Nano but not crucial if you forget it. Git is a highly involved program we will use it for basic tasks and therefore do not need to learn the details in this particular course.
GitBash
Opening a new terminal from Tools
Creating virtual environments
- First navigate to your folder you create for this class from the terminal.
- cd “C:/Users/Administrator/Desktop/Teaching/3339/Fall26”
![]()
- Note the forward slashes instead of backward. Also note in MAC we do not have C:
Creating virtual environments
python -m venv .venv
venv is the command to create a virtual environment.
.venv is a default name and will contain a python interpreter.
Once the virtual environment is created we need to activate it:
Windows (Git Bash): source .venv/Scripts/activate
MACOS and Linux : source .venv/bin/activate
Activated virtual environment
Reset the File Location for VE
Reset the File Location for VE
For Windows: C:/Users/Administrator/Desktop/Teaching/3339/Fall26/.venv/Scripts/python.exe
For MAC OS and Linux: /Users/Administrator/Desktop/Teaching/3339/Fall26/.venv/bin/python
Installing Python Packages via Pip
Once you activate .venv you can now pip install Python packages.
pip install pandas numpy matplotlib
Recall CONDA is another package manager but we will not use it unless we have to.
One more step
- We can configure the Renviron to not have to worry about activating the VE every time however lets worry about that later.