Data Summary and Example Data

• Dataset: RegData U.S. 5.0 from QuantGov API (QuantGov API)
• Example Variables: document_id, year, document_reference, title, part, agency_parent_name, agency_name, restrictions, wordcount, shall, must, may_not, prohibited, required, restrictions_v2, sentence_length, conditionals_per_100_sentences, last_updated, long_word_score, acronyms_per_100_sentences, unique_acronyms
• Structured Data: We loaded the raw dataset from the API, handled missing values, corrected zero values, and engineered features like Score to quantify compliance burden. Based on this, a Compliance column was created to classify documents as “Compliant” or “Non-Compliant.” The processed data, which served as input to the ML model, looks like below:

Analytics Plan (Part 1)

• Objective: This project aims to investigate the estimated compliance costs associated with U.S. federal regulations.
• Goal: Categorize compliance burden into low, moderate, and high tiers. By developing a predictive model to categorize these compliance costs, the project seeks to provide a data-driven framework for understanding the impact of federal regulations on businesses.

Analytics Plan (Part 2)

• Feature Engineering: Derived predictors like:
  • “Score”
  • “Compliance”
• ML Framework: Used H2O for classification modeling.
• Evaluation:
  • Accuracy
  • F1-score
  • Feature importance to interpret results

Peer Comments Summary

• Key Comments from Proposal Presentation:
  • Consider addressing potential class imbalance and assigning weights for better model performance.
  • Use SHAP values to explain factors contributing to high or low-cost categorizations.
  • Recommendation to explore class imbalance mitigation techniques.
• Adjustments Made:
  • Oversampling technique implemented to handle class imbalance.
  • Assigned weights to handle class importance.
  • SHAP values and feature importance analysis added for model interpretability.

Data Exploration (EDA - Part 1)

• Visualizations: The bar chart displays the distribution of compliance categories in the dataset, comparing the frequency of “Compliant” and “Non-Compliant” records. It reveals that “Non-Compliant” documents significantly outnumber “Compliant” ones, indicating an imbalance in the dataset. This imbalance suggests the need for preprocessing techniques, such as oversampling or assigning class weights, to ensure fair and effective machine learning model training.

Data Exploration (EDA - Part 2)

• Correlation Analysis: The correlation heatmap visualizes the relationships between numerical features in the dataset. Strong positive correlations (red) indicate that features like shall, required, and restrictions_v2 are closely related, highlighting their joint influence on compliance. Conversely, weak or negative correlations (blue) suggest minimal or inverse relationships, such as between unique_acronyms and other features. This analysis helps identify feature dependencies, reduce multicollinearity, and select relevant predictors for model training.

Machine Learning Approach (Part 1)

• Model Details:
  • Three models were trained using H2O:
    • Gradient Boosting Machine (GBM)
    • Deep Learning
    • Random Forest
  • Binary classification to categorize compliance into “Compliant” and “Non-Compliant.”
• Preprocessing Steps:
  • Balanced dataset using oversampling to address class imbalance.
  • Data split into training (70%), validation (15%), and test (15%) sets.

Machine Learning Approach (Part 2)

• Hyperparameter Tuning:
  • GBM: Tuned the number of trees, maximum depth, and learning rate.
  • Deep Learning: Optimized hidden layers, epochs, and learning rates.
  • Random Forest: Adjusted the number of trees and maximum depth.
• Evaluation Metrics:
  • Area Under the Curve (AUC) used for cross-validation.
  • Comparison of AUC across all models to select the best one.

Model Results

• Performance Metrics:
  • GBM achieved the highest cross-validation AUC.
  • Deep Learning and Random Forest had slightly lower AUC values.
• Best Model:-
  • GBM was selected as the best-performing model based on AUC.
• Model Saving:-
  • The GBM model was retrained on the training and validation sets.
  • The model was saved for future predictions and analysis.

##   |                                                                              |                                                                      |   0%  |                                                                              |======================================================================| 100%

## [1] "AUC: 0.999641414173515"

## [1] "MSE: 0.01138214445232"

XAI Method 1: Feature Importance

The feature importance plot highlights the relative contributions of various features to the best-performing model’s predictions. Key insights include:

1. Top Features:
   • shall: The most influential feature, indicating its strong predictive power in determining compliance.
   • wordcount: A critical feature, suggesting that longer documents are more likely to comply with regulations.
   • required: A significant feature, highlighting its importance in compliance classification.
2. Secondary Features:
   • Features like restrictions, may_not, and must have moderate influence, reflecting their partial relevance to compliance prediction.
3. Least Influential Features:
   • Features such as acronyms_per_100_sentences, part, and long_word_score show minimal contributions, suggesting they play a negligible role in the model’s decisions.

This visualization provides transparency into how the model prioritizes different features, helping validate the model’s alignment with domain knowledge and guiding future feature engineering efforts.

SHAP Explanation for New Observation

The SHAP explanation plot provides insights into the contribution of individual features to the model’s prediction for a specific observation. Below is a summary of the key insights from the analysis:

Key Insights:

• required = 0:
  • The absence of the required keyword has the most significant negative impact on compliance likelihood.
• wordcount = 3:
  • A very low word count strongly contributes negatively, suggesting that documents with insufficient detail are less likely to comply.
• shall = 0:
  • The absence of the shall keyword negatively influences the prediction, highlighting its importance in compliance determination.

Other Features:

• restrictions, restrictions_v2, and must:
  • These features show moderate influence on the prediction, indicating some relevance to compliance classification.

Neutral or Minimal Impact:

• prohibited, long_word_score, and conditionals_per_100_sentences:
  • These features have minimal contributions, suggesting they play a minor role in determining compliance for this observation.

Interpretation:

This SHAP analysis provides transparency into how the model arrives at its decision for this specific observation. Features with larger bars (positive or negative) have a stronger influence, while smaller bars indicate negligible impact. This ensures interpretability and aligns the model’s predictions with domain knowledge.

Insight and Explanation of the Partial Dependence Profile for sentence_length

The Partial Dependence Profile (PDP) for the feature sentence_length illustrates its relationship with the average prediction made by the best-performing model. This graph helps to understand how varying the values of sentence_length influences the likelihood of the target outcome (e.g., compliance vs. non-compliance) predicted by the model.

Conclusion

The H2O-based predictive model successfully categorized compliance costs with strong performance metrics, achieving a high AUC and low MSE on the test data. Key predictors, such as the frequency of terms like shall and required, along with document length, played a significant role in determining compliance.

Insights from Explainable AI

• Explainable AI techniques provided valuable insights into feature importance and interactions.
• These techniques emphasized the importance of clear, concise, and structured documentation to improve compliance likelihood.

Practical Applicability

The findings underscore the practical applicability of our model in regulatory environments: - Enhanced Compliance Strategies: Organizations can leverage the model to focus on impactful document features, thereby reducing risks and associated compliance costs. - Feature-Driven Insights: The focus on key predictors enables targeted improvements in documentation, aligning with regulatory expectations.