Regularization Techniques

I. Try to run an OLS where the number of predictors (p) is much larger than the number of observations (n). Can you estimate all the betas (slope parameters), and their associated standard errors? Explain why this is happening.

You can see I just simulate the data for this question from different distributions, but you can find any interesting data and confirm the thery.

In cases where the number of predictors p exceeds the number of observations n, the (X’X) matrix in the normal equation β=(X’X)^-1 X’ y becomes non-invertible (singular), which means it’s impossible to compute the inverse. This leads to an inability to solve for the beta coefficients uniquely, resulting in an undetermined system. The degrees of freedom, calculated as n−(p+1), turn negative or zero, implying there is not enough data to estimate the variance and standard errors accurately, leading to undefined standard errors. In addition, with more predictors than observations, some predictors are bound to be linear combinations of others, leading to an infinite number of solutions for the coefficients, making the model unstable.

These problems result in a failure to compute the OLS estimations properly when p > n. This is why alternative methods like ridge and lasso regression, which can handle cases where p > n by introducing a penalty to shrink the coefficients, are used.

We want to make one data to run OLS that p>>n

# Simulate data where p > n
set.seed(123)
# number of observations
n <- 10  
 # number of predictors > n
p <- 500 

# Generate predictors and response
X <- matrix(rnorm(n * p), ncol = p)
# Randomly generated true coefficients
a <- rnorm(p)  
# response variable
y <- X %*% a + rnorm(n)  
summary(y)

       V1          
 Min.   :-16.1837  
 1st Qu.: -0.4383  
 Median : 12.8079  
 Mean   : 14.6549  
 3rd Qu.: 23.0786  
 Max.   : 62.9920  

# Fit OLS model
fit <- lm(y ~ X - 1)
summary(fit)  # Expected to fail due to p > n

Call:
lm(formula = y ~ X - 1)

Residuals:
ALL 10 residuals are 0: no residual degrees of freedom!

Coefficients: (490 not defined because of singularities)
     Estimate Std. Error t value Pr(>|t|)
X1     45.329        NaN     NaN      NaN
X2     -1.930        NaN     NaN      NaN
X3     30.093        NaN     NaN      NaN
X4     -3.969        NaN     NaN      NaN
X5     18.959        NaN     NaN      NaN
X6     39.668        NaN     NaN      NaN
X7     51.674        NaN     NaN      NaN
X8    -29.853        NaN     NaN      NaN
X9    -39.409        NaN     NaN      NaN
X10    28.014        NaN     NaN      NaN
X11        NA         NA      NA       NA
X12        NA         NA      NA       NA
X13        NA         NA      NA       NA
X14        NA         NA      NA       NA
X15        NA         NA      NA       NA
X16        NA         NA      NA       NA
X17        NA         NA      NA       NA
X18        NA         NA      NA       NA
X19        NA         NA      NA       NA
X20        NA         NA      NA       NA
X21        NA         NA      NA       NA
X22        NA         NA      NA       NA
X23        NA         NA      NA       NA
X24        NA         NA      NA       NA
X25        NA         NA      NA       NA
X26        NA         NA      NA       NA
X27        NA         NA      NA       NA
X28        NA         NA      NA       NA
X29        NA         NA      NA       NA
X30        NA         NA      NA       NA
X31        NA         NA      NA       NA
X32        NA         NA      NA       NA
X33        NA         NA      NA       NA
X34        NA         NA      NA       NA
X35        NA         NA      NA       NA
X36        NA         NA      NA       NA
X37        NA         NA      NA       NA
X38        NA         NA      NA       NA
X39        NA         NA      NA       NA
X40        NA         NA      NA       NA
X41        NA         NA      NA       NA
X42        NA         NA      NA       NA
X43        NA         NA      NA       NA
X44        NA         NA      NA       NA
X45        NA         NA      NA       NA
X46        NA         NA      NA       NA
X47        NA         NA      NA       NA
X48        NA         NA      NA       NA
X49        NA         NA      NA       NA
X50        NA         NA      NA       NA
X51        NA         NA      NA       NA
X52        NA         NA      NA       NA
X53        NA         NA      NA       NA
X54        NA         NA      NA       NA
X55        NA         NA      NA       NA
X56        NA         NA      NA       NA
X57        NA         NA      NA       NA
X58        NA         NA      NA       NA
X59        NA         NA      NA       NA
X60        NA         NA      NA       NA
X61        NA         NA      NA       NA
X62        NA         NA      NA       NA
X63        NA         NA      NA       NA
X64        NA         NA      NA       NA
X65        NA         NA      NA       NA
X66        NA         NA      NA       NA
X67        NA         NA      NA       NA
X68        NA         NA      NA       NA
X69        NA         NA      NA       NA
X70        NA         NA      NA       NA
X71        NA         NA      NA       NA
X72        NA         NA      NA       NA
X73        NA         NA      NA       NA
X74        NA         NA      NA       NA
X75        NA         NA      NA       NA
X76        NA         NA      NA       NA
X77        NA         NA      NA       NA
X78        NA         NA      NA       NA
X79        NA         NA      NA       NA
X80        NA         NA      NA       NA
X81        NA         NA      NA       NA
X82        NA         NA      NA       NA
X83        NA         NA      NA       NA
X84        NA         NA      NA       NA
X85        NA         NA      NA       NA
X86        NA         NA      NA       NA
X87        NA         NA      NA       NA
X88        NA         NA      NA       NA
X89        NA         NA      NA       NA
X90        NA         NA      NA       NA
X91        NA         NA      NA       NA
X92        NA         NA      NA       NA
X93        NA         NA      NA       NA
X94        NA         NA      NA       NA
X95        NA         NA      NA       NA
X96        NA         NA      NA       NA
X97        NA         NA      NA       NA
X98        NA         NA      NA       NA
X99        NA         NA      NA       NA
X100       NA         NA      NA       NA
X101       NA         NA      NA       NA
X102       NA         NA      NA       NA
X103       NA         NA      NA       NA
X104       NA         NA      NA       NA
X105       NA         NA      NA       NA
X106       NA         NA      NA       NA
X107       NA         NA      NA       NA
X108       NA         NA      NA       NA
X109       NA         NA      NA       NA
X110       NA         NA      NA       NA
X111       NA         NA      NA       NA
X112       NA         NA      NA       NA
X113       NA         NA      NA       NA
X114       NA         NA      NA       NA
X115       NA         NA      NA       NA
X116       NA         NA      NA       NA
X117       NA         NA      NA       NA
X118       NA         NA      NA       NA
X119       NA         NA      NA       NA
X120       NA         NA      NA       NA
X121       NA         NA      NA       NA
X122       NA         NA      NA       NA
X123       NA         NA      NA       NA
X124       NA         NA      NA       NA
X125       NA         NA      NA       NA
X126       NA         NA      NA       NA
X127       NA         NA      NA       NA
X128       NA         NA      NA       NA
X129       NA         NA      NA       NA
X130       NA         NA      NA       NA
X131       NA         NA      NA       NA
X132       NA         NA      NA       NA
X133       NA         NA      NA       NA
X134       NA         NA      NA       NA
X135       NA         NA      NA       NA
X136       NA         NA      NA       NA
X137       NA         NA      NA       NA
X138       NA         NA      NA       NA
X139       NA         NA      NA       NA
X140       NA         NA      NA       NA
X141       NA         NA      NA       NA
X142       NA         NA      NA       NA
X143       NA         NA      NA       NA
X144       NA         NA      NA       NA
X145       NA         NA      NA       NA
X146       NA         NA      NA       NA
X147       NA         NA      NA       NA
X148       NA         NA      NA       NA
X149       NA         NA      NA       NA
X150       NA         NA      NA       NA
X151       NA         NA      NA       NA
X152       NA         NA      NA       NA
X153       NA         NA      NA       NA
X154       NA         NA      NA       NA
X155       NA         NA      NA       NA
X156       NA         NA      NA       NA
X157       NA         NA      NA       NA
X158       NA         NA      NA       NA
X159       NA         NA      NA       NA
X160       NA         NA      NA       NA
X161       NA         NA      NA       NA
X162       NA         NA      NA       NA
X163       NA         NA      NA       NA
X164       NA         NA      NA       NA
X165       NA         NA      NA       NA
X166       NA         NA      NA       NA
X167       NA         NA      NA       NA
X168       NA         NA      NA       NA
X169       NA         NA      NA       NA
X170       NA         NA      NA       NA
X171       NA         NA      NA       NA
X172       NA         NA      NA       NA
X173       NA         NA      NA       NA
X174       NA         NA      NA       NA
X175       NA         NA      NA       NA
X176       NA         NA      NA       NA
X177       NA         NA      NA       NA
X178       NA         NA      NA       NA
X179       NA         NA      NA       NA
X180       NA         NA      NA       NA
X181       NA         NA      NA       NA
X182       NA         NA      NA       NA
X183       NA         NA      NA       NA
X184       NA         NA      NA       NA
X185       NA         NA      NA       NA
X186       NA         NA      NA       NA
X187       NA         NA      NA       NA
X188       NA         NA      NA       NA
X189       NA         NA      NA       NA
X190       NA         NA      NA       NA
X191       NA         NA      NA       NA
X192       NA         NA      NA       NA
X193       NA         NA      NA       NA
X194       NA         NA      NA       NA
X195       NA         NA      NA       NA
X196       NA         NA      NA       NA
X197       NA         NA      NA       NA
X198       NA         NA      NA       NA
X199       NA         NA      NA       NA
X200       NA         NA      NA       NA
X201       NA         NA      NA       NA
X202       NA         NA      NA       NA
X203       NA         NA      NA       NA
X204       NA         NA      NA       NA
X205       NA         NA      NA       NA
X206       NA         NA      NA       NA
X207       NA         NA      NA       NA
X208       NA         NA      NA       NA
X209       NA         NA      NA       NA
X210       NA         NA      NA       NA
X211       NA         NA      NA       NA
X212       NA         NA      NA       NA
X213       NA         NA      NA       NA
X214       NA         NA      NA       NA
X215       NA         NA      NA       NA
X216       NA         NA      NA       NA
X217       NA         NA      NA       NA
X218       NA         NA      NA       NA
X219       NA         NA      NA       NA
X220       NA         NA      NA       NA
X221       NA         NA      NA       NA
X222       NA         NA      NA       NA
X223       NA         NA      NA       NA
X224       NA         NA      NA       NA
X225       NA         NA      NA       NA
X226       NA         NA      NA       NA
X227       NA         NA      NA       NA
X228       NA         NA      NA       NA
X229       NA         NA      NA       NA
X230       NA         NA      NA       NA
X231       NA         NA      NA       NA
X232       NA         NA      NA       NA
X233       NA         NA      NA       NA
X234       NA         NA      NA       NA
X235       NA         NA      NA       NA
X236       NA         NA      NA       NA
X237       NA         NA      NA       NA
X238       NA         NA      NA       NA
X239       NA         NA      NA       NA
X240       NA         NA      NA       NA
X241       NA         NA      NA       NA
X242       NA         NA      NA       NA
X243       NA         NA      NA       NA
X244       NA         NA      NA       NA
X245       NA         NA      NA       NA
X246       NA         NA      NA       NA
X247       NA         NA      NA       NA
X248       NA         NA      NA       NA
X249       NA         NA      NA       NA
X250       NA         NA      NA       NA
X251       NA         NA      NA       NA
X252       NA         NA      NA       NA
X253       NA         NA      NA       NA
X254       NA         NA      NA       NA
X255       NA         NA      NA       NA
X256       NA         NA      NA       NA
X257       NA         NA      NA       NA
X258       NA         NA      NA       NA
X259       NA         NA      NA       NA
X260       NA         NA      NA       NA
X261       NA         NA      NA       NA
X262       NA         NA      NA       NA
X263       NA         NA      NA       NA
X264       NA         NA      NA       NA
X265       NA         NA      NA       NA
X266       NA         NA      NA       NA
X267       NA         NA      NA       NA
X268       NA         NA      NA       NA
X269       NA         NA      NA       NA
X270       NA         NA      NA       NA
X271       NA         NA      NA       NA
X272       NA         NA      NA       NA
X273       NA         NA      NA       NA
X274       NA         NA      NA       NA
X275       NA         NA      NA       NA
X276       NA         NA      NA       NA
X277       NA         NA      NA       NA
X278       NA         NA      NA       NA
X279       NA         NA      NA       NA
X280       NA         NA      NA       NA
X281       NA         NA      NA       NA
X282       NA         NA      NA       NA
X283       NA         NA      NA       NA
X284       NA         NA      NA       NA
X285       NA         NA      NA       NA
X286       NA         NA      NA       NA
X287       NA         NA      NA       NA
X288       NA         NA      NA       NA
X289       NA         NA      NA       NA
X290       NA         NA      NA       NA
X291       NA         NA      NA       NA
X292       NA         NA      NA       NA
X293       NA         NA      NA       NA
X294       NA         NA      NA       NA
X295       NA         NA      NA       NA
X296       NA         NA      NA       NA
X297       NA         NA      NA       NA
X298       NA         NA      NA       NA
X299       NA         NA      NA       NA
X300       NA         NA      NA       NA
X301       NA         NA      NA       NA
X302       NA         NA      NA       NA
X303       NA         NA      NA       NA
X304       NA         NA      NA       NA
X305       NA         NA      NA       NA
X306       NA         NA      NA       NA
X307       NA         NA      NA       NA
X308       NA         NA      NA       NA
X309       NA         NA      NA       NA
X310       NA         NA      NA       NA
X311       NA         NA      NA       NA
X312       NA         NA      NA       NA
X313       NA         NA      NA       NA
X314       NA         NA      NA       NA
X315       NA         NA      NA       NA
X316       NA         NA      NA       NA
X317       NA         NA      NA       NA
X318       NA         NA      NA       NA
X319       NA         NA      NA       NA
X320       NA         NA      NA       NA
X321       NA         NA      NA       NA
X322       NA         NA      NA       NA
X323       NA         NA      NA       NA
X324       NA         NA      NA       NA
X325       NA         NA      NA       NA
X326       NA         NA      NA       NA
X327       NA         NA      NA       NA
X328       NA         NA      NA       NA
X329       NA         NA      NA       NA
X330       NA         NA      NA       NA
X331       NA         NA      NA       NA
X332       NA         NA      NA       NA
X333       NA         NA      NA       NA
X334       NA         NA      NA       NA
X335       NA         NA      NA       NA
X336       NA         NA      NA       NA
X337       NA         NA      NA       NA
X338       NA         NA      NA       NA
X339       NA         NA      NA       NA
X340       NA         NA      NA       NA
X341       NA         NA      NA       NA
X342       NA         NA      NA       NA
X343       NA         NA      NA       NA
X344       NA         NA      NA       NA
X345       NA         NA      NA       NA
X346       NA         NA      NA       NA
X347       NA         NA      NA       NA
X348       NA         NA      NA       NA
X349       NA         NA      NA       NA
X350       NA         NA      NA       NA
X351       NA         NA      NA       NA
X352       NA         NA      NA       NA
X353       NA         NA      NA       NA
X354       NA         NA      NA       NA
X355       NA         NA      NA       NA
X356       NA         NA      NA       NA
X357       NA         NA      NA       NA
X358       NA         NA      NA       NA
X359       NA         NA      NA       NA
X360       NA         NA      NA       NA
X361       NA         NA      NA       NA
X362       NA         NA      NA       NA
X363       NA         NA      NA       NA
X364       NA         NA      NA       NA
X365       NA         NA      NA       NA
X366       NA         NA      NA       NA
X367       NA         NA      NA       NA
X368       NA         NA      NA       NA
X369       NA         NA      NA       NA
X370       NA         NA      NA       NA
X371       NA         NA      NA       NA
X372       NA         NA      NA       NA
X373       NA         NA      NA       NA
X374       NA         NA      NA       NA
X375       NA         NA      NA       NA
X376       NA         NA      NA       NA
X377       NA         NA      NA       NA
X378       NA         NA      NA       NA
X379       NA         NA      NA       NA
X380       NA         NA      NA       NA
X381       NA         NA      NA       NA
X382       NA         NA      NA       NA
X383       NA         NA      NA       NA
X384       NA         NA      NA       NA
X385       NA         NA      NA       NA
X386       NA         NA      NA       NA
X387       NA         NA      NA       NA
X388       NA         NA      NA       NA
X389       NA         NA      NA       NA
X390       NA         NA      NA       NA
X391       NA         NA      NA       NA
X392       NA         NA      NA       NA
X393       NA         NA      NA       NA
X394       NA         NA      NA       NA
X395       NA         NA      NA       NA
X396       NA         NA      NA       NA
X397       NA         NA      NA       NA
X398       NA         NA      NA       NA
X399       NA         NA      NA       NA
X400       NA         NA      NA       NA
X401       NA         NA      NA       NA
X402       NA         NA      NA       NA
X403       NA         NA      NA       NA
X404       NA         NA      NA       NA
X405       NA         NA      NA       NA
X406       NA         NA      NA       NA
X407       NA         NA      NA       NA
X408       NA         NA      NA       NA
X409       NA         NA      NA       NA
X410       NA         NA      NA       NA
X411       NA         NA      NA       NA
X412       NA         NA      NA       NA
X413       NA         NA      NA       NA
X414       NA         NA      NA       NA
X415       NA         NA      NA       NA
X416       NA         NA      NA       NA
X417       NA         NA      NA       NA
X418       NA         NA      NA       NA
X419       NA         NA      NA       NA
X420       NA         NA      NA       NA
X421       NA         NA      NA       NA
X422       NA         NA      NA       NA
X423       NA         NA      NA       NA
X424       NA         NA      NA       NA
X425       NA         NA      NA       NA
X426       NA         NA      NA       NA
X427       NA         NA      NA       NA
X428       NA         NA      NA       NA
X429       NA         NA      NA       NA
X430       NA         NA      NA       NA
X431       NA         NA      NA       NA
X432       NA         NA      NA       NA
X433       NA         NA      NA       NA
X434       NA         NA      NA       NA
X435       NA         NA      NA       NA
X436       NA         NA      NA       NA
X437       NA         NA      NA       NA
X438       NA         NA      NA       NA
X439       NA         NA      NA       NA
X440       NA         NA      NA       NA
X441       NA         NA      NA       NA
X442       NA         NA      NA       NA
X443       NA         NA      NA       NA
X444       NA         NA      NA       NA
X445       NA         NA      NA       NA
X446       NA         NA      NA       NA
X447       NA         NA      NA       NA
X448       NA         NA      NA       NA
X449       NA         NA      NA       NA
X450       NA         NA      NA       NA
X451       NA         NA      NA       NA
X452       NA         NA      NA       NA
X453       NA         NA      NA       NA
X454       NA         NA      NA       NA
X455       NA         NA      NA       NA
X456       NA         NA      NA       NA
X457       NA         NA      NA       NA
X458       NA         NA      NA       NA
X459       NA         NA      NA       NA
X460       NA         NA      NA       NA
X461       NA         NA      NA       NA
X462       NA         NA      NA       NA
X463       NA         NA      NA       NA
X464       NA         NA      NA       NA
X465       NA         NA      NA       NA
X466       NA         NA      NA       NA
X467       NA         NA      NA       NA
X468       NA         NA      NA       NA
X469       NA         NA      NA       NA
X470       NA         NA      NA       NA
X471       NA         NA      NA       NA
X472       NA         NA      NA       NA
X473       NA         NA      NA       NA
X474       NA         NA      NA       NA
X475       NA         NA      NA       NA
X476       NA         NA      NA       NA
X477       NA         NA      NA       NA
X478       NA         NA      NA       NA
X479       NA         NA      NA       NA
X480       NA         NA      NA       NA
X481       NA         NA      NA       NA
X482       NA         NA      NA       NA
X483       NA         NA      NA       NA
X484       NA         NA      NA       NA
X485       NA         NA      NA       NA
X486       NA         NA      NA       NA
X487       NA         NA      NA       NA
X488       NA         NA      NA       NA
X489       NA         NA      NA       NA
X490       NA         NA      NA       NA
X491       NA         NA      NA       NA
X492       NA         NA      NA       NA
X493       NA         NA      NA       NA
X494       NA         NA      NA       NA
X495       NA         NA      NA       NA
X496       NA         NA      NA       NA
X497       NA         NA      NA       NA
X498       NA         NA      NA       NA
X499       NA         NA      NA       NA
X500       NA         NA      NA       NA

Residual standard error: NaN on 0 degrees of freedom
Multiple R-squared:      1, Adjusted R-squared:    NaN 
F-statistic:   NaN on 10 and 0 DF,  p-value: NA

library(stargazer)

Please cite as: 

 Hlavac, Marek (2022). stargazer: Well-Formatted Regression and Summary Statistics Tables.

 R package version 5.2.3. https://CRAN.R-project.org/package=stargazer 

stargazer(fit, type="text")

========================================
                 Dependent variable:    
             ---------------------------
                          y             
----------------------------------------
X1                     45.329           
                                        
                                        
X2                     -1.930           
                                        
                                        
X3                     30.093           
                                        
                                        
X4                     -3.969           
                                        
                                        
X5                     18.959           
                                        
                                        
X6                     39.668           
                                        
                                        
X7                     51.674           
                                        
                                        
X8                     -29.853          
                                        
                                        
X9                     -39.409          
                                        
                                        
X10                    28.014           
                                        
                                        
X11                                     
                                        
                                        
X12                                     
                                        
                                        
X13                                     
                                        
                                        
X14                                     
                                        
                                        
X15                                     
                                        
                                        
X16                                     
                                        
                                        
X17                                     
                                        
                                        
X18                                     
                                        
                                        
X19                                     
                                        
                                        
X20                                     
                                        
                                        
X21                                     
                                        
                                        
X22                                     
                                        
                                        
X23                                     
                                        
                                        
X24                                     
                                        
                                        
X25                                     
                                        
                                        
X26                                     
                                        
                                        
X27                                     
                                        
                                        
X28                                     
                                        
                                        
X29                                     
                                        
                                        
X30                                     
                                        
                                        
X31                                     
                                        
                                        
X32                                     
                                        
                                        
X33                                     
                                        
                                        
X34                                     
                                        
                                        
X35                                     
                                        
                                        
X36                                     
                                        
                                        
X37                                     
                                        
                                        
X38                                     
                                        
                                        
X39                                     
                                        
                                        
X40                                     
                                        
                                        
X41                                     
                                        
                                        
X42                                     
                                        
                                        
X43                                     
                                        
                                        
X44                                     
                                        
                                        
X45                                     
                                        
                                        
X46                                     
                                        
                                        
X47                                     
                                        
                                        
X48                                     
                                        
                                        
X49                                     
                                        
                                        
X50                                     
                                        
                                        
X51                                     
                                        
                                        
X52                                     
                                        
                                        
X53                                     
                                        
                                        
X54                                     
                                        
                                        
X55                                     
                                        
                                        
X56                                     
                                        
                                        
X57                                     
                                        
                                        
X58                                     
                                        
                                        
X59                                     
                                        
                                        
X60                                     
                                        
                                        
X61                                     
                                        
                                        
X62                                     
                                        
                                        
X63                                     
                                        
                                        
X64                                     
                                        
                                        
X65                                     
                                        
                                        
X66                                     
                                        
                                        
X67                                     
                                        
                                        
X68                                     
                                        
                                        
X69                                     
                                        
                                        
X70                                     
                                        
                                        
X71                                     
                                        
                                        
X72                                     
                                        
                                        
X73                                     
                                        
                                        
X74                                     
                                        
                                        
X75                                     
                                        
                                        
X76                                     
                                        
                                        
X77                                     
                                        
                                        
X78                                     
                                        
                                        
X79                                     
                                        
                                        
X80                                     
                                        
                                        
X81                                     
                                        
                                        
X82                                     
                                        
                                        
X83                                     
                                        
                                        
X84                                     
                                        
                                        
X85                                     
                                        
                                        
X86                                     
                                        
                                        
X87                                     
                                        
                                        
X88                                     
                                        
                                        
X89                                     
                                        
                                        
X90                                     
                                        
                                        
X91                                     
                                        
                                        
X92                                     
                                        
                                        
X93                                     
                                        
                                        
X94                                     
                                        
                                        
X95                                     
                                        
                                        
X96                                     
                                        
                                        
X97                                     
                                        
                                        
X98                                     
                                        
                                        
X99                                     
                                        
                                        
X100                                    
                                        
                                        
X101                                    
                                        
                                        
X102                                    
                                        
                                        
X103                                    
                                        
                                        
X104                                    
                                        
                                        
X105                                    
                                        
                                        
X106                                    
                                        
                                        
X107                                    
                                        
                                        
X108                                    
                                        
                                        
X109                                    
                                        
                                        
X110                                    
                                        
                                        
X111                                    
                                        
                                        
X112                                    
                                        
                                        
X113                                    
                                        
                                        
X114                                    
                                        
                                        
X115                                    
                                        
                                        
X116                                    
                                        
                                        
X117                                    
                                        
                                        
X118                                    
                                        
                                        
X119                                    
                                        
                                        
X120                                    
                                        
                                        
X121                                    
                                        
                                        
X122                                    
                                        
                                        
X123                                    
                                        
                                        
X124                                    
                                        
                                        
X125                                    
                                        
                                        
X126                                    
                                        
                                        
X127                                    
                                        
                                        
X128                                    
                                        
                                        
X129                                    
                                        
                                        
X130                                    
                                        
                                        
X131                                    
                                        
                                        
X132                                    
                                        
                                        
X133                                    
                                        
                                        
X134                                    
                                        
                                        
X135                                    
                                        
                                        
X136                                    
                                        
                                        
X137                                    
                                        
                                        
X138                                    
                                        
                                        
X139                                    
                                        
                                        
X140                                    
                                        
                                        
X141                                    
                                        
                                        
X142                                    
                                        
                                        
X143                                    
                                        
                                        
X144                                    
                                        
                                        
X145                                    
                                        
                                        
X146                                    
                                        
                                        
X147                                    
                                        
                                        
X148                                    
                                        
                                        
X149                                    
                                        
                                        
X150                                    
                                        
                                        
X151                                    
                                        
                                        
X152                                    
                                        
                                        
X153                                    
                                        
                                        
X154                                    
                                        
                                        
X155                                    
                                        
                                        
X156                                    
                                        
                                        
X157                                    
                                        
                                        
X158                                    
                                        
                                        
X159                                    
                                        
                                        
X160                                    
                                        
                                        
X161                                    
                                        
                                        
X162                                    
                                        
                                        
X163                                    
                                        
                                        
X164                                    
                                        
                                        
X165                                    
                                        
                                        
X166                                    
                                        
                                        
X167                                    
                                        
                                        
X168                                    
                                        
                                        
X169                                    
                                        
                                        
X170                                    
                                        
                                        
X171                                    
                                        
                                        
X172                                    
                                        
                                        
X173                                    
                                        
                                        
X174                                    
                                        
                                        
X175                                    
                                        
                                        
X176                                    
                                        
                                        
X177                                    
                                        
                                        
X178                                    
                                        
                                        
X179                                    
                                        
                                        
X180                                    
                                        
                                        
X181                                    
                                        
                                        
X182                                    
                                        
                                        
X183                                    
                                        
                                        
X184                                    
                                        
                                        
X185                                    
                                        
                                        
X186                                    
                                        
                                        
X187                                    
                                        
                                        
X188                                    
                                        
                                        
X189                                    
                                        
                                        
X190                                    
                                        
                                        
X191                                    
                                        
                                        
X192                                    
                                        
                                        
X193                                    
                                        
                                        
X194                                    
                                        
                                        
X195                                    
                                        
                                        
X196                                    
                                        
                                        
X197                                    
                                        
                                        
X198                                    
                                        
                                        
X199                                    
                                        
                                        
X200                                    
                                        
                                        
X201                                    
                                        
                                        
X202                                    
                                        
                                        
X203                                    
                                        
                                        
X204                                    
                                        
                                        
X205                                    
                                        
                                        
X206                                    
                                        
                                        
X207                                    
                                        
                                        
X208                                    
                                        
                                        
X209                                    
                                        
                                        
X210                                    
                                        
                                        
X211                                    
                                        
                                        
X212                                    
                                        
                                        
X213                                    
                                        
                                        
X214                                    
                                        
                                        
X215                                    
                                        
                                        
X216                                    
                                        
                                        
X217                                    
                                        
                                        
X218                                    
                                        
                                        
X219                                    
                                        
                                        
X220                                    
                                        
                                        
X221                                    
                                        
                                        
X222                                    
                                        
                                        
X223                                    
                                        
                                        
X224                                    
                                        
                                        
X225                                    
                                        
                                        
X226                                    
                                        
                                        
X227                                    
                                        
                                        
X228                                    
                                        
                                        
X229                                    
                                        
                                        
X230                                    
                                        
                                        
X231                                    
                                        
                                        
X232                                    
                                        
                                        
X233                                    
                                        
                                        
X234                                    
                                        
                                        
X235                                    
                                        
                                        
X236                                    
                                        
                                        
X237                                    
                                        
                                        
X238                                    
                                        
                                        
X239                                    
                                        
                                        
X240                                    
                                        
                                        
X241                                    
                                        
                                        
X242                                    
                                        
                                        
X243                                    
                                        
                                        
X244                                    
                                        
                                        
X245                                    
                                        
                                        
X246                                    
                                        
                                        
X247                                    
                                        
                                        
X248                                    
                                        
                                        
X249                                    
                                        
                                        
X250                                    
                                        
                                        
X251                                    
                                        
                                        
X252                                    
                                        
                                        
X253                                    
                                        
                                        
X254                                    
                                        
                                        
X255                                    
                                        
                                        
X256                                    
                                        
                                        
X257                                    
                                        
                                        
X258                                    
                                        
                                        
X259                                    
                                        
                                        
X260                                    
                                        
                                        
X261                                    
                                        
                                        
X262                                    
                                        
                                        
X263                                    
                                        
                                        
X264                                    
                                        
                                        
X265                                    
                                        
                                        
X266                                    
                                        
                                        
X267                                    
                                        
                                        
X268                                    
                                        
                                        
X269                                    
                                        
                                        
X270                                    
                                        
                                        
X271                                    
                                        
                                        
X272                                    
                                        
                                        
X273                                    
                                        
                                        
X274                                    
                                        
                                        
X275                                    
                                        
                                        
X276                                    
                                        
                                        
X277                                    
                                        
                                        
X278                                    
                                        
                                        
X279                                    
                                        
                                        
X280                                    
                                        
                                        
X281                                    
                                        
                                        
X282                                    
                                        
                                        
X283                                    
                                        
                                        
X284                                    
                                        
                                        
X285                                    
                                        
                                        
X286                                    
                                        
                                        
X287                                    
                                        
                                        
X288                                    
                                        
                                        
X289                                    
                                        
                                        
X290                                    
                                        
                                        
X291                                    
                                        
                                        
X292                                    
                                        
                                        
X293                                    
                                        
                                        
X294                                    
                                        
                                        
X295                                    
                                        
                                        
X296                                    
                                        
                                        
X297                                    
                                        
                                        
X298                                    
                                        
                                        
X299                                    
                                        
                                        
X300                                    
                                        
                                        
X301                                    
                                        
                                        
X302                                    
                                        
                                        
X303                                    
                                        
                                        
X304                                    
                                        
                                        
X305                                    
                                        
                                        
X306                                    
                                        
                                        
X307                                    
                                        
                                        
X308                                    
                                        
                                        
X309                                    
                                        
                                        
X310                                    
                                        
                                        
X311                                    
                                        
                                        
X312                                    
                                        
                                        
X313                                    
                                        
                                        
X314                                    
                                        
                                        
X315                                    
                                        
                                        
X316                                    
                                        
                                        
X317                                    
                                        
                                        
X318                                    
                                        
                                        
X319                                    
                                        
                                        
X320                                    
                                        
                                        
X321                                    
                                        
                                        
X322                                    
                                        
                                        
X323                                    
                                        
                                        
X324                                    
                                        
                                        
X325                                    
                                        
                                        
X326                                    
                                        
                                        
X327                                    
                                        
                                        
X328                                    
                                        
                                        
X329                                    
                                        
                                        
X330                                    
                                        
                                        
X331                                    
                                        
                                        
X332                                    
                                        
                                        
X333                                    
                                        
                                        
X334                                    
                                        
                                        
X335                                    
                                        
                                        
X336                                    
                                        
                                        
X337                                    
                                        
                                        
X338                                    
                                        
                                        
X339                                    
                                        
                                        
X340                                    
                                        
                                        
X341                                    
                                        
                                        
X342                                    
                                        
                                        
X343                                    
                                        
                                        
X344                                    
                                        
                                        
X345                                    
                                        
                                        
X346                                    
                                        
                                        
X347                                    
                                        
                                        
X348                                    
                                        
                                        
X349                                    
                                        
                                        
X350                                    
                                        
                                        
X351                                    
                                        
                                        
X352                                    
                                        
                                        
X353                                    
                                        
                                        
X354                                    
                                        
                                        
X355                                    
                                        
                                        
X356                                    
                                        
                                        
X357                                    
                                        
                                        
X358                                    
                                        
                                        
X359                                    
                                        
                                        
X360                                    
                                        
                                        
X361                                    
                                        
                                        
X362                                    
                                        
                                        
X363                                    
                                        
                                        
X364                                    
                                        
                                        
X365                                    
                                        
                                        
X366                                    
                                        
                                        
X367                                    
                                        
                                        
X368                                    
                                        
                                        
X369                                    
                                        
                                        
X370                                    
                                        
                                        
X371                                    
                                        
                                        
X372                                    
                                        
                                        
X373                                    
                                        
                                        
X374                                    
                                        
                                        
X375                                    
                                        
                                        
X376                                    
                                        
                                        
X377                                    
                                        
                                        
X378                                    
                                        
                                        
X379                                    
                                        
                                        
X380                                    
                                        
                                        
X381                                    
                                        
                                        
X382                                    
                                        
                                        
X383                                    
                                        
                                        
X384                                    
                                        
                                        
X385                                    
                                        
                                        
X386                                    
                                        
                                        
X387                                    
                                        
                                        
X388                                    
                                        
                                        
X389                                    
                                        
                                        
X390                                    
                                        
                                        
X391                                    
                                        
                                        
X392                                    
                                        
                                        
X393                                    
                                        
                                        
X394                                    
                                        
                                        
X395                                    
                                        
                                        
X396                                    
                                        
                                        
X397                                    
                                        
                                        
X398                                    
                                        
                                        
X399                                    
                                        
                                        
X400                                    
                                        
                                        
X401                                    
                                        
                                        
X402                                    
                                        
                                        
X403                                    
                                        
                                        
X404                                    
                                        
                                        
X405                                    
                                        
                                        
X406                                    
                                        
                                        
X407                                    
                                        
                                        
X408                                    
                                        
                                        
X409                                    
                                        
                                        
X410                                    
                                        
                                        
X411                                    
                                        
                                        
X412                                    
                                        
                                        
X413                                    
                                        
                                        
X414                                    
                                        
                                        
X415                                    
                                        
                                        
X416                                    
                                        
                                        
X417                                    
                                        
                                        
X418                                    
                                        
                                        
X419                                    
                                        
                                        
X420                                    
                                        
                                        
X421                                    
                                        
                                        
X422                                    
                                        
                                        
X423                                    
                                        
                                        
X424                                    
                                        
                                        
X425                                    
                                        
                                        
X426                                    
                                        
                                        
X427                                    
                                        
                                        
X428                                    
                                        
                                        
X429                                    
                                        
                                        
X430                                    
                                        
                                        
X431                                    
                                        
                                        
X432                                    
                                        
                                        
X433                                    
                                        
                                        
X434                                    
                                        
                                        
X435                                    
                                        
                                        
X436                                    
                                        
                                        
X437                                    
                                        
                                        
X438                                    
                                        
                                        
X439                                    
                                        
                                        
X440                                    
                                        
                                        
X441                                    
                                        
                                        
X442                                    
                                        
                                        
X443                                    
                                        
                                        
X444                                    
                                        
                                        
X445                                    
                                        
                                        
X446                                    
                                        
                                        
X447                                    
                                        
                                        
X448                                    
                                        
                                        
X449                                    
                                        
                                        
X450                                    
                                        
                                        
X451                                    
                                        
                                        
X452                                    
                                        
                                        
X453                                    
                                        
                                        
X454                                    
                                        
                                        
X455                                    
                                        
                                        
X456                                    
                                        
                                        
X457                                    
                                        
                                        
X458                                    
                                        
                                        
X459                                    
                                        
                                        
X460                                    
                                        
                                        
X461                                    
                                        
                                        
X462                                    
                                        
                                        
X463                                    
                                        
                                        
X464                                    
                                        
                                        
X465                                    
                                        
                                        
X466                                    
                                        
                                        
X467                                    
                                        
                                        
X468                                    
                                        
                                        
X469                                    
                                        
                                        
X470                                    
                                        
                                        
X471                                    
                                        
                                        
X472                                    
                                        
                                        
X473                                    
                                        
                                        
X474                                    
                                        
                                        
X475                                    
                                        
                                        
X476                                    
                                        
                                        
X477                                    
                                        
                                        
X478                                    
                                        
                                        
X479                                    
                                        
                                        
X480                                    
                                        
                                        
X481                                    
                                        
                                        
X482                                    
                                        
                                        
X483                                    
                                        
                                        
X484                                    
                                        
                                        
X485                                    
                                        
                                        
X486                                    
                                        
                                        
X487                                    
                                        
                                        
X488                                    
                                        
                                        
X489                                    
                                        
                                        
X490                                    
                                        
                                        
X491                                    
                                        
                                        
X492                                    
                                        
                                        
X493                                    
                                        
                                        
X494                                    
                                        
                                        
X495                                    
                                        
                                        
X496                                    
                                        
                                        
X497                                    
                                        
                                        
X498                                    
                                        
                                        
X499                                    
                                        
                                        
X500                                    
                                        
                                        
----------------------------------------
Observations             10             
R2                      1.000           
========================================
Note:        *p<0.1; **p<0.05; ***p<0.01

This R code snippet generates a scenario where p>n and tries fitting an OLS model, which leads to errors due to the reasons explained above. SO we cannot uniquely estimate all coefficients

II. On a dataset of your choice where where p >> n, apply the lasso or ridge method after scaling all the X variables. Which method do you prefer, and why? Do they give similar coefficients?

Lasso might be preferable if feature selection (zeroing out coefficients) is important, especially in high-dimensional data to reduce model complexity and avoid overfitting. Ridge might be preferred if multicollinearity is a concern and shrinking coefficients without setting them to zero helps retain more information about the predictors.

cross-validation with standardizes

Standardization transforms data to have a mean of zero and a standard deviation of one. This process is done by subtracting the mean of each feature and then dividing by the standard deviation z= (x-miu)/sd

Standardization: When the features have different units or different scales, standardization ensures they are all on a comparable level, making it easier for the algorithm to learn patterns effectively.

# Lasso applies when only a subset of predictors is relevant so Lasso will be preferred in this case, but I put cross-validation here

library(glmnet)

Warning: package 'glmnet' was built under R version 4.3.3

Loading required package: Matrix

Loaded glmnet 4.1-8

# Scale predictors

# Standardize predictors (z-score normalization)
X_scaled <- scale(X)  # Standardizes each predictor to have mean = 0 and sd = 1

# Optionally standardize the response if it helps model performance
y_scaled <- scale(y)  # Standardizing the response variable

# Confirm standardization (optional in production, more useful during development)
cat("Means of scaled predictors (should be close to 0):", mean(colMeans(X_scaled)), "\n")

Means of scaled predictors (should be close to 0): -1.677478e-19 

cat("Standard deviations of scaled predictors (should be close to 1):", mean(apply(X_scaled, 2, sd)), "\n")

Standard deviations of scaled predictors (should be close to 1): 1 

# Fit Lasso Model
lasso_model <- glmnet(X_scaled, y_scaled, alpha = 1)
summary(lasso_model)

          Length Class     Mode   
a0           79  -none-    numeric
beta      39500  dgCMatrix S4     
df           79  -none-    numeric
dim           2  -none-    numeric
lambda       79  -none-    numeric
dev.ratio    79  -none-    numeric
nulldev       1  -none-    numeric
npasses       1  -none-    numeric
jerr          1  -none-    numeric
offset        1  -none-    logical
call          4  -none-    call   
nobs          1  -none-    numeric

ridge_model <- glmnet(X_scaled, y_scaled, alpha = 0)
summary(ridge_model)

          Length Class     Mode   
a0          100  -none-    numeric
beta      50000  dgCMatrix S4     
df          100  -none-    numeric
dim           2  -none-    numeric
lambda      100  -none-    numeric
dev.ratio   100  -none-    numeric
nulldev       1  -none-    numeric
npasses       1  -none-    numeric
jerr          1  -none-    numeric
offset        1  -none-    logical
call          4  -none-    call   
nobs          1  -none-    numeric

# Choosing lambda using cross-validation
cv_lasso <- cv.glmnet(X_scaled, y_scaled, type.measure = "mse", alpha = 1)

Warning: Option grouped=FALSE enforced in cv.glmnet, since < 3 observations per
fold

cv_ridge <- cv.glmnet(X_scaled, y_scaled, type.measure = "mse", alpha = 0)

Warning: Option grouped=FALSE enforced in cv.glmnet, since < 3 observations per
fold

# Plotting CV results
plot(cv_lasso)

plot(cv_ridge)

Given that the Lasso regression has effectively zeroed out almost all the predictors except the intercept, this suggests a very sparse model. Lasso is generally preferred if the goal is feature selection, as it can shrink less important predictor coefficients to zero, simplifying the model and potentially improving interpretability.

# Extract and print coefficients at the optimal lambda
lasso_coefs <- coef(cv_lasso, s = "lambda.min")
ridge_coefs <- coef(cv_ridge, s = "lambda.min")
print("Lasso Coefficients:")

[1] "Lasso Coefficients:"

summary(lasso_coefs)

501 x 1 sparse Matrix of class "dgCMatrix", with 1 entries 
  i j            x
1 1 1 5.551115e-17

print("Ridge Coefficients:")

[1] "Ridge Coefficients:"

summary(ridge_coefs)

501 x 1 sparse Matrix of class "dgCMatrix", with 501 entries 
      i j             x
1     1 1  5.720596e-17
2     2 1 -5.725959e-03
3     3 1 -6.674863e-03
4     4 1  1.200671e-02
5     5 1 -9.612440e-03
6     6 1 -6.894179e-04
7     7 1 -1.192161e-03
8     8 1  9.596732e-03
9     9 1 -3.888309e-03
10   10 1  1.130968e-02
11   11 1 -2.122073e-03
12   12 1  5.381441e-03
13   13 1  1.119427e-03
14   14 1  1.165882e-03
15   15 1 -5.720679e-03
16   16 1 -1.394292e-03
17   17 1 -4.349060e-03
18   18 1 -1.330240e-03
19   19 1 -4.007724e-03
20   20 1  2.597017e-03
21   21 1 -4.233040e-03
22   22 1 -4.898094e-03
23   23 1 -6.957761e-03
24   24 1 -1.042621e-02
25   25 1 -5.806870e-03
26   26 1 -4.518429e-04
27   27 1 -4.499004e-03
28   28 1 -2.769063e-03
29   29 1 -1.726939e-03
30   30 1  6.020618e-03
31   31 1  1.285895e-04
32   32 1  8.662310e-03
33   33 1  3.461845e-03
34   34 1  3.075611e-03
35   35 1 -3.692794e-03
36   36 1 -5.280422e-03
37   37 1  7.602133e-03
38   38 1 -7.606931e-03
39   39 1 -9.141602e-03
40   40 1  1.648575e-03
41   41 1 -5.011225e-04
42   42 1  3.800844e-03
43   43 1 -4.211361e-03
44   44 1 -7.717406e-03
45   45 1  8.370638e-03
46   46 1 -2.368058e-03
47   47 1 -4.396623e-03
48   48 1  5.804031e-03
49   49 1 -1.589705e-04
50   50 1 -8.944200e-03
51   51 1  1.336728e-03
52   52 1 -5.916765e-03
53   53 1 -2.583883e-03
54   54 1 -2.144967e-03
55   55 1  2.355156e-03
56   56 1 -3.850315e-03
57   57 1 -8.410747e-03
58   58 1  5.052536e-03
59   59 1  1.546395e-03
60   60 1 -4.694737e-03
61   61 1  4.185249e-03
62   62 1 -6.145540e-03
63   63 1  1.085147e-03
64   64 1  7.176155e-03
65   65 1 -1.969050e-03
66   66 1  4.013433e-03
67   67 1  3.840578e-03
68   68 1 -1.019555e-03
69   69 1 -5.889045e-03
70   70 1 -4.802631e-03
71   71 1 -5.808693e-03
72   72 1  5.059258e-03
73   73 1 -6.290904e-04
74   74 1  6.218522e-03
75   75 1  4.953252e-03
76   76 1 -4.607209e-03
77   77 1 -2.606945e-03
78   78 1  6.761228e-03
79   79 1  1.117368e-03
80   80 1 -1.851893e-03
81   81 1 -2.358629e-03
82   82 1 -3.034990e-03
83   83 1  3.128778e-04
84   84 1 -6.326090e-03
85   85 1 -6.336509e-03
86   86 1  2.283927e-03
87   87 1  1.786935e-03
88   88 1  7.474498e-03
89   89 1  3.610576e-03
90   90 1 -1.726346e-03
91   91 1  2.705900e-03
92   92 1 -8.097609e-04
93   93 1  1.071824e-03
94   94 1 -1.692983e-04
95   95 1  4.986900e-03
96   96 1  1.022095e-03
97   97 1  5.018807e-03
98   98 1 -4.975451e-03
99   99 1  4.262328e-03
100 100 1  4.152662e-03
101 101 1 -5.267966e-03
102 102 1  2.207519e-03
103 103 1  3.200334e-03
104 104 1  1.512557e-03
105 105 1  1.231216e-02
106 106 1 -5.546565e-04
107 107 1  3.589963e-04
108 108 1  6.923013e-04
109 109 1 -1.604367e-03
110 110 1 -6.094559e-03
111 111 1 -4.927809e-03
112 112 1  4.760054e-03
113 113 1 -1.008129e-02
114 114 1 -6.625990e-03
115 115 1 -1.227640e-03
116 116 1  4.279388e-03
117 117 1 -5.717013e-03
118 118 1  7.320695e-03
119 119 1  3.199021e-04
120 120 1 -9.931511e-03
121 121 1 -1.738299e-04
122 122 1 -1.970538e-04
123 123 1 -8.010164e-03
124 124 1  6.717285e-03
125 125 1  5.903785e-03
126 126 1  3.265459e-04
127 127 1 -3.406483e-03
128 128 1  1.075108e-02
129 129 1 -1.435622e-03
130 130 1  3.002415e-03
131 131 1  1.054942e-03
132 132 1  4.940466e-03
133 133 1 -5.021992e-03
134 134 1 -8.993336e-03
135 135 1 -8.154791e-04
136 136 1  6.139097e-03
137 137 1  5.731708e-04
138 138 1 -3.075986e-03
139 139 1  1.085910e-03
140 140 1 -8.519859e-03
141 141 1 -4.980467e-03
142 142 1  2.346996e-03
143 143 1  6.286857e-03
144 144 1 -6.711919e-04
145 145 1 -4.021175e-03
146 146 1 -2.475709e-03
147 147 1 -4.045361e-03
148 148 1 -5.081101e-03
149 149 1  4.033795e-03
150 150 1 -2.314342e-03
151 151 1 -1.520314e-03
152 152 1  3.893420e-03
153 153 1 -4.904559e-03
154 154 1  4.252511e-03
155 155 1  1.089153e-03
156 156 1 -1.420236e-03
157 157 1  2.743545e-03
158 158 1  2.997072e-03
159 159 1 -8.506451e-04
160 160 1  4.884221e-03
161 161 1  9.891121e-03
162 162 1  6.276267e-03
163 163 1 -1.148773e-03
164 164 1  4.073230e-03
165 165 1  8.041672e-03
166 166 1 -6.971573e-03
167 167 1 -7.746449e-03
168 168 1  6.991882e-03
169 169 1  8.597891e-03
170 170 1 -2.055586e-03
171 171 1 -1.508482e-03
172 172 1  6.644518e-03
173 173 1  1.622962e-03
174 174 1  1.066794e-03
175 175 1 -3.428953e-03
176 176 1 -3.375397e-03
177 177 1  7.553925e-03
178 178 1  2.551259e-04
179 179 1 -3.309500e-03
180 180 1  3.306808e-03
181 181 1 -1.005508e-02
182 182 1 -5.962231e-03
183 183 1  3.365226e-03
184 184 1  7.908982e-03
185 185 1  4.431936e-04
186 186 1 -1.145962e-02
187 187 1 -4.235912e-03
188 188 1  4.423383e-03
189 189 1  1.830691e-04
190 190 1 -6.147389e-03
191 191 1  2.948536e-03
192 192 1  9.574042e-04
193 193 1 -3.971463e-03
194 194 1  3.597648e-03
195 195 1 -2.808551e-03
196 196 1  4.775179e-03
197 197 1  1.131632e-02
198 198 1  2.420451e-03
199 199 1 -3.249784e-04
200 200 1  1.028501e-02
201 201 1  1.258755e-03
202 202 1 -4.562147e-03
203 203 1  2.892568e-03
204 204 1 -9.773652e-03
205 205 1 -1.541983e-03
206 206 1  1.342071e-03
207 207 1 -3.749747e-03
208 208 1 -6.046563e-03
209 209 1 -3.595075e-03
210 210 1 -3.123942e-03
211 211 1  1.895497e-03
212 212 1 -1.668052e-03
213 213 1  3.401756e-03
214 214 1 -2.901083e-03
215 215 1 -7.250066e-03
216 216 1  4.834930e-03
217 217 1 -2.446157e-03
218 218 1  4.176527e-03
219 219 1  4.223658e-03
220 220 1  3.456718e-04
221 221 1  2.426388e-03
222 222 1 -1.282914e-03
223 223 1  3.865921e-03
224 224 1  1.584402e-04
225 225 1 -7.432167e-03
226 226 1 -6.026731e-03
227 227 1  8.516264e-03
228 228 1  6.383997e-03
229 229 1 -4.987909e-03
230 230 1  1.049439e-02
231 231 1 -2.202403e-04
232 232 1 -3.664136e-03
233 233 1 -3.191069e-03
234 234 1 -1.567334e-04
235 235 1  1.498278e-03
236 236 1 -1.287566e-03
237 237 1  3.986599e-03
238 238 1 -3.243800e-03
239 239 1 -3.896451e-03
240 240 1  6.155845e-03
241 241 1  1.574478e-03
242 242 1 -6.961934e-04
243 243 1  3.061722e-05
244 244 1  6.093492e-03
245 245 1 -4.111977e-04
246 246 1  1.884016e-03
247 247 1 -7.100817e-03
248 248 1  3.737819e-03
249 249 1  4.755153e-03
250 250 1 -3.549229e-03
251 251 1 -6.019844e-04
252 252 1  1.277942e-02
253 253 1 -2.492574e-03
254 254 1 -3.778881e-03
255 255 1  8.075398e-03
256 256 1  1.336259e-03
257 257 1 -8.314177e-04
258 258 1  6.687359e-03
259 259 1  3.085510e-03
260 260 1 -3.935253e-03
261 261 1 -2.753948e-03
262 262 1 -1.500169e-03
263 263 1 -5.092707e-04
264 264 1 -6.461339e-03
265 265 1  4.163228e-03
266 266 1  7.580552e-03
267 267 1 -7.741127e-03
268 268 1 -8.316270e-03
269 269 1  6.781295e-03
270 270 1  2.594530e-03
271 271 1 -2.264787e-03
272 272 1  6.661944e-03
273 273 1 -1.127134e-04
274 274 1 -2.375886e-03
275 275 1 -4.067951e-03
276 276 1 -3.042321e-03
277 277 1 -8.838274e-03
278 278 1  1.299492e-02
279 279 1 -1.849217e-03
280 280 1 -2.521473e-03
281 281 1 -7.228483e-03
282 282 1  2.037844e-03
283 283 1  3.073682e-03
284 284 1  8.270327e-03
285 285 1  6.316869e-03
286 286 1 -4.115636e-03
287 287 1  2.355461e-03
288 288 1 -5.576889e-03
289 289 1 -5.814529e-03
290 290 1 -6.443134e-03
291 291 1 -2.491909e-06
292 292 1 -2.920332e-03
293 293 1  5.153074e-03
294 294 1 -4.996047e-03
295 295 1 -3.668400e-04
296 296 1 -5.139914e-03
297 297 1  1.971093e-03
298 298 1 -2.731482e-03
299 299 1 -1.172690e-03
300 300 1 -1.081527e-04
301 301 1 -2.122043e-04
302 302 1  2.732440e-03
303 303 1  1.210865e-03
304 304 1 -4.588708e-03
305 305 1  3.974340e-04
306 306 1 -1.090837e-02
307 307 1  4.430670e-03
308 308 1  7.205148e-04
309 309 1  3.494751e-03
310 310 1 -6.338446e-04
311 311 1  1.030540e-02
312 312 1 -2.802985e-03
313 313 1 -4.472562e-03
314 314 1  1.601771e-03
315 315 1  9.106985e-03
316 316 1  4.484708e-03
317 317 1  8.073787e-03
318 318 1 -8.794442e-03
319 319 1  6.896121e-03
320 320 1 -9.527618e-03
321 321 1  6.815900e-05
322 322 1  5.519384e-03
323 323 1  2.713887e-03
324 324 1  7.332251e-03
325 325 1 -1.189264e-03
326 326 1  5.152776e-03
327 327 1 -2.764322e-03
328 328 1 -4.381371e-03
329 329 1  8.984933e-04
330 330 1  5.141153e-03
331 331 1 -2.697766e-03
332 332 1  4.564233e-03
333 333 1  8.296926e-04
334 334 1  5.209721e-03
335 335 1 -5.786606e-03
336 336 1  1.929039e-03
337 337 1 -3.663703e-03
338 338 1 -2.920104e-03
339 339 1  5.112390e-03
340 340 1  9.540380e-04
341 341 1 -5.277400e-03
342 342 1  5.638801e-03
343 343 1 -1.752860e-03
344 344 1 -1.037760e-03
345 345 1  3.793329e-03
346 346 1 -3.229546e-03
347 347 1  1.779262e-03
348 348 1 -3.712508e-03
349 349 1  5.605030e-03
350 350 1  4.682767e-03
351 351 1  5.012276e-03
352 352 1  7.517489e-03
353 353 1  3.970509e-03
354 354 1  4.319402e-03
355 355 1 -2.985801e-03
356 356 1 -2.675527e-03
357 357 1  1.633016e-03
358 358 1  4.983904e-03
359 359 1  2.272784e-03
360 360 1 -4.166665e-03
361 361 1  6.078812e-03
362 362 1 -5.655947e-03
363 363 1  5.969985e-03
364 364 1 -6.047992e-03
365 365 1  6.013206e-03
366 366 1  1.708502e-03
367 367 1  2.315502e-03
368 368 1 -3.477405e-03
369 369 1 -3.448137e-03
370 370 1 -1.267524e-03
371 371 1  4.728874e-03
372 372 1  1.644635e-03
373 373 1 -5.759358e-03
374 374 1 -2.781631e-03
375 375 1  2.308570e-03
376 376 1  7.748706e-03
377 377 1 -6.385896e-03
378 378 1  5.153537e-03
379 379 1  6.211136e-03
380 380 1  4.227793e-03
381 381 1  4.816518e-04
382 382 1 -1.539526e-03
383 383 1 -6.472664e-03
384 384 1 -6.255969e-03
385 385 1 -5.055861e-03
386 386 1 -2.642936e-03
387 387 1  2.263372e-03
388 388 1  7.599551e-04
389 389 1 -1.108238e-02
390 390 1 -1.943153e-03
391 391 1 -3.611814e-03
392 392 1  8.644939e-04
393 393 1  8.337669e-04
394 394 1 -4.512355e-03
395 395 1  6.205540e-03
396 396 1  2.053988e-03
397 397 1  5.452377e-03
398 398 1 -8.243326e-03
399 399 1 -3.339610e-03
400 400 1 -2.920425e-03
401 401 1  1.183527e-02
402 402 1  5.215340e-03
403 403 1 -1.009155e-03
404 404 1 -1.451998e-03
405 405 1  1.142506e-02
406 406 1 -7.333939e-03
407 407 1  4.177702e-04
408 408 1 -9.721280e-03
409 409 1 -1.231549e-03
410 410 1  2.492806e-03
411 411 1 -6.119629e-03
412 412 1 -3.515516e-03
413 413 1 -3.596183e-03
414 414 1  5.736394e-03
415 415 1 -2.349973e-03
416 416 1  9.429431e-03
417 417 1  9.132512e-03
418 418 1  5.708970e-03
419 419 1 -5.820411e-03
420 420 1 -9.041124e-03
421 421 1 -1.070361e-02
422 422 1  9.437785e-04
423 423 1 -1.398746e-03
424 424 1 -5.928614e-03
425 425 1 -8.970734e-03
426 426 1 -6.051239e-03
427 427 1  3.031232e-03
428 428 1  1.450519e-03
429 429 1  4.610585e-03
430 430 1  3.045866e-03
431 431 1  5.346196e-03
432 432 1 -1.614350e-03
433 433 1  9.346788e-03
434 434 1 -4.119257e-03
435 435 1  5.707966e-03
436 436 1  6.985731e-03
437 437 1 -6.134296e-03
438 438 1  3.379678e-03
439 439 1 -3.656488e-03
440 440 1 -7.333520e-03
441 441 1 -7.188280e-03
442 442 1  3.801964e-03
443 443 1 -3.520185e-03
444 444 1 -1.194841e-03
445 445 1 -9.937793e-03
446 446 1 -6.315110e-03
447 447 1 -9.311176e-03
448 448 1  5.175795e-03
449 449 1  3.432057e-03
450 450 1 -2.110297e-03
451 451 1 -2.686962e-03
452 452 1 -1.032232e-02
453 453 1 -1.114946e-02
454 454 1  2.451092e-03
455 455 1  4.365215e-03
456 456 1 -7.171983e-03
457 457 1  3.833804e-03
458 458 1  4.441320e-03
459 459 1  7.916086e-03
460 460 1  6.117981e-03
461 461 1  5.532847e-03
462 462 1  4.477788e-03
463 463 1 -5.706532e-04
464 464 1  2.764609e-03
465 465 1 -4.611028e-04
466 466 1  7.873330e-03
467 467 1  6.381709e-03
468 468 1  3.337498e-03
469 469 1 -1.913549e-03
470 470 1 -3.006609e-03
471 471 1 -9.014251e-04
472 472 1  4.222849e-03
473 473 1 -5.274096e-03
474 474 1  9.030968e-03
475 475 1  9.064897e-03
476 476 1  1.143490e-02
477 477 1  8.853548e-03
478 478 1  7.712550e-03
479 479 1  1.235461e-03
480 480 1  4.035600e-03
481 481 1 -3.247770e-03
482 482 1  1.069396e-03
483 483 1 -2.467288e-03
484 484 1  2.387484e-03
485 485 1 -1.026152e-02
486 486 1 -1.565222e-03
487 487 1 -3.973713e-03
488 488 1  8.514115e-03
489 489 1 -4.569701e-03
490 490 1  7.356157e-03
491 491 1 -5.480259e-03
492 492 1 -8.793487e-03
493 493 1  5.262687e-03
494 494 1 -9.087084e-03
495 495 1 -2.022139e-03
496 496 1 -2.735161e-03
497 497 1  3.461726e-03
498 498 1 -3.368337e-03
499 499 1 -5.075186e-05
500 500 1 -5.667755e-03
501 501 1  3.612740e-03

So they do not produce similar coefficients. Lasso results in a model with fewer variables, potentially only the intercept or very few others having non-zero coefficients. On the other hand, Ridge modifies the coefficients to be smaller yet still retains them all, preventing any from becoming exactly zero unless inherently so due to lack of variability or influence.

Now, take any independent variable that has not been eliminated, and change its units (transform it). In lots of instances, this may make sense (EG if the variable is income measured in dollars, you may want to measure it in thousands of dollars instead). Rerun the lasso/ridge now (without scaling the variables) - does this rescaled variable that was not previously eliminated get eliminated as a resulting change in scale? Explain why this could happen?

# Transform and refit Lasso without rescaling
X_transformed <- X_scaled
X_transformed[,1] <- X_transformed[,1] * 1000  # Significantly scale up x1

# Refit Lasso on the transformed data
lasso_transformed <- glmnet(X_transformed, y_scaled, alpha = 1)
coefl_transformed <- coef(lasso_transformed, s = cv_lasso$lambda.min)
print("Transformed Lasso Coefficients:")

[1] "Transformed Lasso Coefficients:"

summary(coefl_transformed)

501 x 1 sparse Matrix of class "dgCMatrix", with 1 entries 
  i j            x
1 1 1 5.551115e-17

ridge_transformed <- glmnet(X_transformed, y_scaled, alpha = 0)
coefr_transformed <- coef(ridge_transformed, s = cv_lasso$lambda.min)
print("Transformed Ridge Coefficients:")

[1] "Transformed Ridge Coefficients:"

summary(coefr_transformed)

501 x 1 sparse Matrix of class "dgCMatrix", with 501 entries 
      i j             x
1     1 1  5.732134e-17
2     2 1 -5.725959e-06
3     3 1 -6.674863e-03
4     4 1  1.200671e-02
5     5 1 -9.612440e-03
6     6 1 -6.894179e-04
7     7 1 -1.192161e-03
8     8 1  9.596732e-03
9     9 1 -3.888309e-03
10   10 1  1.130968e-02
11   11 1 -2.122073e-03
12   12 1  5.381441e-03
13   13 1  1.119427e-03
14   14 1  1.165882e-03
15   15 1 -5.720679e-03
16   16 1 -1.394292e-03
17   17 1 -4.349060e-03
18   18 1 -1.330240e-03
19   19 1 -4.007724e-03
20   20 1  2.597017e-03
21   21 1 -4.233040e-03
22   22 1 -4.898094e-03
23   23 1 -6.957761e-03
24   24 1 -1.042621e-02
25   25 1 -5.806870e-03
26   26 1 -4.518429e-04
27   27 1 -4.499004e-03
28   28 1 -2.769063e-03
29   29 1 -1.726939e-03
30   30 1  6.020618e-03
31   31 1  1.285895e-04
32   32 1  8.662310e-03
33   33 1  3.461845e-03
34   34 1  3.075611e-03
35   35 1 -3.692794e-03
36   36 1 -5.280422e-03
37   37 1  7.602133e-03
38   38 1 -7.606931e-03
39   39 1 -9.141602e-03
40   40 1  1.648575e-03
41   41 1 -5.011225e-04
42   42 1  3.800844e-03
43   43 1 -4.211361e-03
44   44 1 -7.717406e-03
45   45 1  8.370638e-03
46   46 1 -2.368058e-03
47   47 1 -4.396623e-03
48   48 1  5.804031e-03
49   49 1 -1.589705e-04
50   50 1 -8.944200e-03
51   51 1  1.336728e-03
52   52 1 -5.916765e-03
53   53 1 -2.583883e-03
54   54 1 -2.144967e-03
55   55 1  2.355156e-03
56   56 1 -3.850315e-03
57   57 1 -8.410747e-03
58   58 1  5.052536e-03
59   59 1  1.546395e-03
60   60 1 -4.694737e-03
61   61 1  4.185249e-03
62   62 1 -6.145540e-03
63   63 1  1.085147e-03
64   64 1  7.176155e-03
65   65 1 -1.969050e-03
66   66 1  4.013433e-03
67   67 1  3.840578e-03
68   68 1 -1.019555e-03
69   69 1 -5.889045e-03
70   70 1 -4.802631e-03
71   71 1 -5.808693e-03
72   72 1  5.059258e-03
73   73 1 -6.290904e-04
74   74 1  6.218522e-03
75   75 1  4.953252e-03
76   76 1 -4.607209e-03
77   77 1 -2.606945e-03
78   78 1  6.761228e-03
79   79 1  1.117368e-03
80   80 1 -1.851893e-03
81   81 1 -2.358629e-03
82   82 1 -3.034990e-03
83   83 1  3.128778e-04
84   84 1 -6.326090e-03
85   85 1 -6.336509e-03
86   86 1  2.283927e-03
87   87 1  1.786935e-03
88   88 1  7.474498e-03
89   89 1  3.610576e-03
90   90 1 -1.726346e-03
91   91 1  2.705900e-03
92   92 1 -8.097609e-04
93   93 1  1.071824e-03
94   94 1 -1.692983e-04
95   95 1  4.986900e-03
96   96 1  1.022095e-03
97   97 1  5.018807e-03
98   98 1 -4.975451e-03
99   99 1  4.262328e-03
100 100 1  4.152662e-03
101 101 1 -5.267966e-03
102 102 1  2.207519e-03
103 103 1  3.200334e-03
104 104 1  1.512557e-03
105 105 1  1.231216e-02
106 106 1 -5.546565e-04
107 107 1  3.589963e-04
108 108 1  6.923013e-04
109 109 1 -1.604367e-03
110 110 1 -6.094559e-03
111 111 1 -4.927809e-03
112 112 1  4.760054e-03
113 113 1 -1.008129e-02
114 114 1 -6.625990e-03
115 115 1 -1.227640e-03
116 116 1  4.279388e-03
117 117 1 -5.717013e-03
118 118 1  7.320695e-03
119 119 1  3.199021e-04
120 120 1 -9.931511e-03
121 121 1 -1.738299e-04
122 122 1 -1.970538e-04
123 123 1 -8.010164e-03
124 124 1  6.717285e-03
125 125 1  5.903785e-03
126 126 1  3.265459e-04
127 127 1 -3.406483e-03
128 128 1  1.075108e-02
129 129 1 -1.435622e-03
130 130 1  3.002415e-03
131 131 1  1.054942e-03
132 132 1  4.940466e-03
133 133 1 -5.021992e-03
134 134 1 -8.993336e-03
135 135 1 -8.154791e-04
136 136 1  6.139097e-03
137 137 1  5.731708e-04
138 138 1 -3.075986e-03
139 139 1  1.085910e-03
140 140 1 -8.519859e-03
141 141 1 -4.980467e-03
142 142 1  2.346996e-03
143 143 1  6.286857e-03
144 144 1 -6.711919e-04
145 145 1 -4.021175e-03
146 146 1 -2.475709e-03
147 147 1 -4.045361e-03
148 148 1 -5.081101e-03
149 149 1  4.033795e-03
150 150 1 -2.314342e-03
151 151 1 -1.520314e-03
152 152 1  3.893420e-03
153 153 1 -4.904559e-03
154 154 1  4.252511e-03
155 155 1  1.089153e-03
156 156 1 -1.420236e-03
157 157 1  2.743545e-03
158 158 1  2.997072e-03
159 159 1 -8.506451e-04
160 160 1  4.884221e-03
161 161 1  9.891121e-03
162 162 1  6.276267e-03
163 163 1 -1.148773e-03
164 164 1  4.073230e-03
165 165 1  8.041672e-03
166 166 1 -6.971573e-03
167 167 1 -7.746449e-03
168 168 1  6.991882e-03
169 169 1  8.597891e-03
170 170 1 -2.055586e-03
171 171 1 -1.508482e-03
172 172 1  6.644518e-03
173 173 1  1.622962e-03
174 174 1  1.066794e-03
175 175 1 -3.428953e-03
176 176 1 -3.375397e-03
177 177 1  7.553925e-03
178 178 1  2.551259e-04
179 179 1 -3.309500e-03
180 180 1  3.306808e-03
181 181 1 -1.005508e-02
182 182 1 -5.962231e-03
183 183 1  3.365226e-03
184 184 1  7.908982e-03
185 185 1  4.431936e-04
186 186 1 -1.145962e-02
187 187 1 -4.235912e-03
188 188 1  4.423383e-03
189 189 1  1.830691e-04
190 190 1 -6.147389e-03
191 191 1  2.948536e-03
192 192 1  9.574042e-04
193 193 1 -3.971463e-03
194 194 1  3.597648e-03
195 195 1 -2.808551e-03
196 196 1  4.775179e-03
197 197 1  1.131632e-02
198 198 1  2.420451e-03
199 199 1 -3.249784e-04
200 200 1  1.028501e-02
201 201 1  1.258755e-03
202 202 1 -4.562147e-03
203 203 1  2.892568e-03
204 204 1 -9.773652e-03
205 205 1 -1.541983e-03
206 206 1  1.342071e-03
207 207 1 -3.749747e-03
208 208 1 -6.046563e-03
209 209 1 -3.595075e-03
210 210 1 -3.123942e-03
211 211 1  1.895497e-03
212 212 1 -1.668052e-03
213 213 1  3.401756e-03
214 214 1 -2.901083e-03
215 215 1 -7.250066e-03
216 216 1  4.834930e-03
217 217 1 -2.446157e-03
218 218 1  4.176527e-03
219 219 1  4.223658e-03
220 220 1  3.456718e-04
221 221 1  2.426388e-03
222 222 1 -1.282914e-03
223 223 1  3.865921e-03
224 224 1  1.584402e-04
225 225 1 -7.432167e-03
226 226 1 -6.026731e-03
227 227 1  8.516264e-03
228 228 1  6.383997e-03
229 229 1 -4.987909e-03
230 230 1  1.049439e-02
231 231 1 -2.202403e-04
232 232 1 -3.664136e-03
233 233 1 -3.191069e-03
234 234 1 -1.567334e-04
235 235 1  1.498278e-03
236 236 1 -1.287566e-03
237 237 1  3.986599e-03
238 238 1 -3.243800e-03
239 239 1 -3.896451e-03
240 240 1  6.155845e-03
241 241 1  1.574478e-03
242 242 1 -6.961934e-04
243 243 1  3.061722e-05
244 244 1  6.093492e-03
245 245 1 -4.111977e-04
246 246 1  1.884016e-03
247 247 1 -7.100817e-03
248 248 1  3.737819e-03
249 249 1  4.755153e-03
250 250 1 -3.549229e-03
251 251 1 -6.019844e-04
252 252 1  1.277942e-02
253 253 1 -2.492574e-03
254 254 1 -3.778881e-03
255 255 1  8.075398e-03
256 256 1  1.336259e-03
257 257 1 -8.314177e-04
258 258 1  6.687359e-03
259 259 1  3.085510e-03
260 260 1 -3.935253e-03
261 261 1 -2.753948e-03
262 262 1 -1.500169e-03
263 263 1 -5.092707e-04
264 264 1 -6.461339e-03
265 265 1  4.163228e-03
266 266 1  7.580552e-03
267 267 1 -7.741127e-03
268 268 1 -8.316270e-03
269 269 1  6.781295e-03
270 270 1  2.594530e-03
271 271 1 -2.264787e-03
272 272 1  6.661944e-03
273 273 1 -1.127134e-04
274 274 1 -2.375886e-03
275 275 1 -4.067951e-03
276 276 1 -3.042321e-03
277 277 1 -8.838274e-03
278 278 1  1.299492e-02
279 279 1 -1.849217e-03
280 280 1 -2.521473e-03
281 281 1 -7.228483e-03
282 282 1  2.037844e-03
283 283 1  3.073682e-03
284 284 1  8.270327e-03
285 285 1  6.316869e-03
286 286 1 -4.115636e-03
287 287 1  2.355461e-03
288 288 1 -5.576889e-03
289 289 1 -5.814529e-03
290 290 1 -6.443134e-03
291 291 1 -2.491909e-06
292 292 1 -2.920332e-03
293 293 1  5.153074e-03
294 294 1 -4.996047e-03
295 295 1 -3.668400e-04
296 296 1 -5.139914e-03
297 297 1  1.971093e-03
298 298 1 -2.731482e-03
299 299 1 -1.172690e-03
300 300 1 -1.081527e-04
301 301 1 -2.122043e-04
302 302 1  2.732440e-03
303 303 1  1.210865e-03
304 304 1 -4.588708e-03
305 305 1  3.974340e-04
306 306 1 -1.090837e-02
307 307 1  4.430670e-03
308 308 1  7.205148e-04
309 309 1  3.494751e-03
310 310 1 -6.338446e-04
311 311 1  1.030540e-02
312 312 1 -2.802985e-03
313 313 1 -4.472562e-03
314 314 1  1.601771e-03
315 315 1  9.106985e-03
316 316 1  4.484708e-03
317 317 1  8.073787e-03
318 318 1 -8.794442e-03
319 319 1  6.896121e-03
320 320 1 -9.527618e-03
321 321 1  6.815900e-05
322 322 1  5.519384e-03
323 323 1  2.713887e-03
324 324 1  7.332251e-03
325 325 1 -1.189264e-03
326 326 1  5.152776e-03
327 327 1 -2.764322e-03
328 328 1 -4.381371e-03
329 329 1  8.984933e-04
330 330 1  5.141153e-03
331 331 1 -2.697766e-03
332 332 1  4.564233e-03
333 333 1  8.296926e-04
334 334 1  5.209721e-03
335 335 1 -5.786606e-03
336 336 1  1.929039e-03
337 337 1 -3.663703e-03
338 338 1 -2.920104e-03
339 339 1  5.112390e-03
340 340 1  9.540380e-04
341 341 1 -5.277400e-03
342 342 1  5.638801e-03
343 343 1 -1.752860e-03
344 344 1 -1.037760e-03
345 345 1  3.793329e-03
346 346 1 -3.229546e-03
347 347 1  1.779262e-03
348 348 1 -3.712508e-03
349 349 1  5.605030e-03
350 350 1  4.682767e-03
351 351 1  5.012276e-03
352 352 1  7.517489e-03
353 353 1  3.970509e-03
354 354 1  4.319402e-03
355 355 1 -2.985801e-03
356 356 1 -2.675527e-03
357 357 1  1.633016e-03
358 358 1  4.983904e-03
359 359 1  2.272784e-03
360 360 1 -4.166665e-03
361 361 1  6.078812e-03
362 362 1 -5.655947e-03
363 363 1  5.969985e-03
364 364 1 -6.047992e-03
365 365 1  6.013206e-03
366 366 1  1.708502e-03
367 367 1  2.315502e-03
368 368 1 -3.477405e-03
369 369 1 -3.448137e-03
370 370 1 -1.267524e-03
371 371 1  4.728874e-03
372 372 1  1.644635e-03
373 373 1 -5.759358e-03
374 374 1 -2.781631e-03
375 375 1  2.308570e-03
376 376 1  7.748706e-03
377 377 1 -6.385896e-03
378 378 1  5.153537e-03
379 379 1  6.211136e-03
380 380 1  4.227793e-03
381 381 1  4.816518e-04
382 382 1 -1.539526e-03
383 383 1 -6.472664e-03
384 384 1 -6.255969e-03
385 385 1 -5.055861e-03
386 386 1 -2.642936e-03
387 387 1  2.263372e-03
388 388 1  7.599551e-04
389 389 1 -1.108238e-02
390 390 1 -1.943153e-03
391 391 1 -3.611814e-03
392 392 1  8.644939e-04
393 393 1  8.337669e-04
394 394 1 -4.512355e-03
395 395 1  6.205540e-03
396 396 1  2.053988e-03
397 397 1  5.452377e-03
398 398 1 -8.243326e-03
399 399 1 -3.339610e-03
400 400 1 -2.920425e-03
401 401 1  1.183527e-02
402 402 1  5.215340e-03
403 403 1 -1.009155e-03
404 404 1 -1.451998e-03
405 405 1  1.142506e-02
406 406 1 -7.333939e-03
407 407 1  4.177702e-04
408 408 1 -9.721280e-03
409 409 1 -1.231549e-03
410 410 1  2.492806e-03
411 411 1 -6.119629e-03
412 412 1 -3.515516e-03
413 413 1 -3.596183e-03
414 414 1  5.736394e-03
415 415 1 -2.349973e-03
416 416 1  9.429431e-03
417 417 1  9.132512e-03
418 418 1  5.708970e-03
419 419 1 -5.820411e-03
420 420 1 -9.041124e-03
421 421 1 -1.070361e-02
422 422 1  9.437785e-04
423 423 1 -1.398746e-03
424 424 1 -5.928614e-03
425 425 1 -8.970734e-03
426 426 1 -6.051239e-03
427 427 1  3.031232e-03
428 428 1  1.450519e-03
429 429 1  4.610585e-03
430 430 1  3.045866e-03
431 431 1  5.346196e-03
432 432 1 -1.614350e-03
433 433 1  9.346788e-03
434 434 1 -4.119257e-03
435 435 1  5.707966e-03
436 436 1  6.985731e-03
437 437 1 -6.134296e-03
438 438 1  3.379678e-03
439 439 1 -3.656488e-03
440 440 1 -7.333520e-03
441 441 1 -7.188280e-03
442 442 1  3.801964e-03
443 443 1 -3.520185e-03
444 444 1 -1.194841e-03
445 445 1 -9.937793e-03
446 446 1 -6.315110e-03
447 447 1 -9.311176e-03
448 448 1  5.175795e-03
449 449 1  3.432057e-03
450 450 1 -2.110297e-03
451 451 1 -2.686962e-03
452 452 1 -1.032232e-02
453 453 1 -1.114946e-02
454 454 1  2.451092e-03
455 455 1  4.365215e-03
456 456 1 -7.171983e-03
457 457 1  3.833804e-03
458 458 1  4.441320e-03
459 459 1  7.916086e-03
460 460 1  6.117981e-03
461 461 1  5.532847e-03
462 462 1  4.477788e-03
463 463 1 -5.706532e-04
464 464 1  2.764609e-03
465 465 1 -4.611028e-04
466 466 1  7.873330e-03
467 467 1  6.381709e-03
468 468 1  3.337498e-03
469 469 1 -1.913549e-03
470 470 1 -3.006609e-03
471 471 1 -9.014251e-04
472 472 1  4.222849e-03
473 473 1 -5.274096e-03
474 474 1  9.030968e-03
475 475 1  9.064897e-03
476 476 1  1.143490e-02
477 477 1  8.853548e-03
478 478 1  7.712550e-03
479 479 1  1.235461e-03
480 480 1  4.035600e-03
481 481 1 -3.247770e-03
482 482 1  1.069396e-03
483 483 1 -2.467288e-03
484 484 1  2.387484e-03
485 485 1 -1.026152e-02
486 486 1 -1.565222e-03
487 487 1 -3.973713e-03
488 488 1  8.514115e-03
489 489 1 -4.569701e-03
490 490 1  7.356157e-03
491 491 1 -5.480259e-03
492 492 1 -8.793487e-03
493 493 1  5.262687e-03
494 494 1 -9.087084e-03
495 495 1 -2.022139e-03
496 496 1 -2.735161e-03
497 497 1  3.461726e-03
498 498 1 -3.368337e-03
499 499 1 -5.075186e-05
500 500 1 -5.667755e-03
501 501 1  3.612740e-03

When scaling an independent variable significantly, such as transforming a measurement of income from dollars to thousands of dollars, it can have substantial effects on the coefficients determined by Lasso and Ridge regression. This impact arises because the regularization penalties applied in these models are sensitive to the scale of the variables.

Note scaling up x1 by a factor of a thousand in the Lasso regression model led to its coefficient being effectively shrunk to zero. This result is a clear demonstration of how the L1 penalty, which targets the absolute size of coefficients, is influenced by the scale of the features. The larger scale increases the penalty for that specific variable, potentially reducing its coefficient to zero unless the regularization parameter lambda is adjusted accordingly.

For Ridge regression, however, the coefficients are shrunk towards zero but generally remain non-zero.

In summary, while Lasso can zero out coefficients of significantly scaled variables, potentially eliminating them from the model, Ridge tends to retain all variables with smaller, adjusted coefficients.