• The C++ preprocessor replaces text in C++ source files before compilation
• Java and Python allow “annotations” to provide syntactic metaprocessing of code (“reflection”)
• Features like this allow programmers alter code as it is being processed (by the compiler or at runtime)
• For exmaple, in Python you can literally construct python code inside your code and execute it on the fly
• There are several ways to do this in Julia, but the most straightforward way are with macros
• Macros start with the @ symbol

x = 6
@show x^3
@assert x>1000 "x show be bigger than 1000"
@time sum(sqrt.(rand(10000)))

using BenchmarkTools
@benchmark sum(sqrt.(rand(10000)))

using ProgressMeter
@showprogress for idx in 1:216
         sleep(0.1);
       end

• @. makes every operation in an expression dotted (vectorized)
• @threads before a loop makes the loop multithreaded
• @simd before a loop allows a loop to be fully vectorized
• @async wraps an expression in a task that can be run asyncronously

• Using quote … end defines a block of code that will not be executed until explicitly evaluated
• Alternatively: :( .. )
• You can evaluate later using eval(...)
• You can build up expressions, store them, the evaluate them later… in code
• Or you can modify code on the fly
• That is … your code can re-write itself as it is running
• Using @macroexpand will display the macro

More Info: https://docs.julialang.org/en/v1/manual/metaprogramming/#man-macros

• It’s pretty complicated, but you can write your own macros
• Here’s a very simple example for timing something:

macro time(ex)
    return quote
        local start = time_ns()
        local val = $ex  # Evaluate the expression
        local stop = time_ns()
        println("elapsed time: ", (stop-start)/1e9, " seconds")
        val
    end
end

• Most realistic data pipeline workflows involve using multiple languages
• Of course, we could have separate steps between them, storing out temporary structures in one to read into another
• Or we could find a complicated way to execute remotely shelled commands from one lanuage to another
• But Julia lets you natively call R and Python from inside Julia code
• You just have to be careful about type conversions since Python and R objects are not the same as Julia datatypes

• Julia has an external package called PyCall that let’s you call Python directly

using Pkg
Pkg.add("PyCall")    # Install, just once

More Info: https://github.com/JuliaPy/PyCall.jl

• By default Julia mantains it’s own version of Python, not your system Python
• So you won’t necessarily have access to Python packages you’ve already installed
• You can install Python packages with Conda

Pkg.build("PyCall") 
using Conda
Conda.add(["numpy", "pandas", "scipy", "matplotlib"])  

• But you can use your system python

ENV["PYTHON"] = "/opt/local/bin/python" # Or Wherever yours is ...
Pkg.build("PyCall")

• Once you have PyCall loaded, you can import any installed Python package you like with pyimport
• Afterward, a lot of Python code can be directly called

np = pyimport("numpy");
x = np.random.normal(size=10, scale=2.5);
println("x = $x");

pd = pyimport("pandas");
foo = pd.read_csv("foo.csv")
foo["Name"]

• Julia has an external package called RCall that let’s you use an R environment from inside Julia

using Pkg
Pkg.add("RCall")    # Install, just once
using RCall         # Whenever you want to use R

More Info: https://juliainterop.github.io/RCall.jl/stable/gettingstarted/

• You can drop into an R console from inside Julia by typing the $
• You get back to the Julia prompt by pressing backspace (Windows) or delete (MacOS)
• You can copy variables into the R environment from Julia using @rput

x = 3.1415;
@rput x;

• You can copy variables out of the R environment from Julia using @rget

@rget y;
println("y = $y");

• You can make a direct call using R strings:

x = R"rnorm(10)"

• But results from R calls are special Robjects
• If you want to convert R structures into Julia structures, use Rcopy

z = rcopy(R"rnorm(10)")

• Or the other direction with robject

rdf = robject(df);
@rput rdf;
R"""
  library(ggplot2)
  ggplot(rdf, aes(x=sepal_length, y=sepal_width, fill=species)) +
    geom_point(size=4, shape=21, color="black") +
    xlab("Sepal Length of Iris") + ylab("Sepal Width of Iris")
  """

@rimport base as rbase
rbase.sum(df.sepal_length)
rcall(:summary, rdf)

• REPL: Read-Eval Print Loop – the interactive command line
• Julia has one of the most powerful REPLs I’ve seen
• It has different modes, and you can move between these modes very easily
• Modes include: Julian, Help, Pkg, Shell, Search, and R
• Julian mode is the normal Julia interactive command line
• From Julian mode, pressing the ? key puts you in Help mode
• Pressing the ] key puts you in Pkg mode (for package management)

More Info: https://docs.julialang.org/en/v1/stdlib/REPL/

• You can go back to your OS command-line shell by pressing the ; key from the Julia REPL
• This is better than exiting Julia because you don’t lose state in your command-line interactions
• You get back to the Julian mode by just hitting the delete key
• This lets you easily whip in and out of the OS while debugging, for example

• From Julian mode, you can enter Search mode using ^R
• This will allow you to perform a reverse search through your command history
• You can also just use the arrow keys to move through you history, and the Julia REPL remembers which mode you were in

• The RCall package lets get to an R prompt from inside Julia
• Once you are using that module, the $ key will switch the REPL to R mode
• It’s more or less the same as being at the R command line in general
• State is retained, so you can verify state information with @rput and @rget
• As far as I know, there is no Python model for the Julia REPL

• Don’t forget about Julia’s powerful vectorized operations (broadcast dispatch)
• These are operations that are done concurrently, possibly in parallel
• They are much faster, and far less cumbersome

x = [1,2,3];
y = [4,5,6];
z = x .* y;
println("z = $z");

• https://docs.julialang.org/en/v1/manual/functions/#man-vectorized

string([1,2,3])
string.([1,2,3])
string([1,2,3], ["a", "b", "c"])
string.([1,2,3], ["a", "b", "c"])

fib(n) = n ≤ 2 ? one(n) : fib(n-1) + fib(n-2)     # Fibonacci number
dist(X,Y) = .√sum((X-Y).^2, dims=2)               # Distance between every row vector betwen X and Y
norm(X) = X./sum(X, dims=2)                       # Normalize each row vector in a matrix
avglag(X) = sum(X[2:end] - X[1:end-1], dims=1) / (size(X)[1]-1)  # Average lag between numbers in a list

# Here's a cooler estimator for pi (Bailey-Borwein-Plouffe)
bppEstimate(n) = sum([(1/16^k) * sum([4,-2,-1,-1] ./ (8*k .+ [1,4,5,6]))  for k in 0:n])

M1 = M = [1 2 3; 4 5 6; 7 8 9];
M2 = rand(Float64, (3,3));
zero(M1)  # What is zero for this data type?
zero(M2)
zero(6)
zero(-1.77)

one(M1)   # What is identity for this data type?
one(M2)
one(6)
one(-1.77)

similar(M2) # Give me an uninitialized structure just like this one

≻(a,b) = all(a.>=b) && any(a.>b);
x = [1,2,3];
y = [0,2,2];
z = [1,4,1];

x ≻ y
x ≻ z

# Users can only send float matrices and vectors to this
function Print1(A::Matrix{Float64}, x::Vector{Float64})
    println("A = $A");
    println("x = $x");
end

# Users can send any matrix or vector that's an int, float, or complex
function Print2(A::Matrix{<:Real}, x::Vector{<:Real})
    println("A = $A");
    println("x = $x");
end
         
# Users can send any kind of matrix or vector to this
function Print2(A::AbstractMatrix, x::AbstractVector)
    println("A = $A");
    println("x = $x");
end