mirror of
https://github.com/PlatypusPus/MushroomEmpire.git
synced 2026-02-07 22:18:59 +00:00
at this point i want to be done so no commit quality for you
This commit is contained in:
365
BIAS_ANALYSIS_GUIDE.md
Normal file
365
BIAS_ANALYSIS_GUIDE.md
Normal file
@@ -0,0 +1,365 @@
|
|||||||
|
# Enhanced Bias & Fairness Analysis Guide
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
The Nordic Privacy AI platform now includes a comprehensive, adaptive bias and fairness analysis system that works accurately across **all types of datasets**, including:
|
||||||
|
|
||||||
|
- Small datasets (< 100 samples)
|
||||||
|
- Imbalanced groups
|
||||||
|
- Multiple protected attributes
|
||||||
|
- Binary and multi-class targets
|
||||||
|
- High-cardinality features
|
||||||
|
- Missing data
|
||||||
|
|
||||||
|
## Key Enhancements
|
||||||
|
|
||||||
|
### 1. **Adaptive Fairness Thresholds**
|
||||||
|
|
||||||
|
The system automatically adjusts fairness thresholds based on dataset characteristics:
|
||||||
|
|
||||||
|
- **Sample Size Factor**: Relaxes thresholds for small sample sizes
|
||||||
|
- **Group Imbalance Factor**: Adjusts for unequal group sizes
|
||||||
|
- **Dynamic Thresholds**:
|
||||||
|
- Disparate Impact: 0.7-0.8 (adapts to data)
|
||||||
|
- Statistical Parity: 0.1-0.15 (adapts to data)
|
||||||
|
- Equal Opportunity: 0.1-0.15 (adapts to data)
|
||||||
|
|
||||||
|
### 2. **Comprehensive Fairness Metrics**
|
||||||
|
|
||||||
|
#### Individual Metrics (6 types analyzed):
|
||||||
|
|
||||||
|
1. **Disparate Impact Ratio** (4/5ths rule)
|
||||||
|
- Measures: min_rate / max_rate across all groups
|
||||||
|
- Fair range: 0.8 - 1.25 (or adaptive)
|
||||||
|
- Higher weight in overall score
|
||||||
|
|
||||||
|
2. **Statistical Parity Difference**
|
||||||
|
- Measures: Absolute difference in positive rates
|
||||||
|
- Fair threshold: < 0.1 (or adaptive)
|
||||||
|
- Ensures equal selection rates
|
||||||
|
|
||||||
|
3. **Equal Opportunity** (TPR equality)
|
||||||
|
- Measures: Difference in True Positive Rates
|
||||||
|
- Fair threshold: < 0.1 (or adaptive)
|
||||||
|
- Ensures equal recall across groups
|
||||||
|
|
||||||
|
4. **Equalized Odds** (TPR + FPR equality)
|
||||||
|
- Measures: Both TPR and FPR differences
|
||||||
|
- Fair threshold: < 0.1 (or adaptive)
|
||||||
|
- Most comprehensive fairness criterion
|
||||||
|
|
||||||
|
5. **Predictive Parity** (Precision equality)
|
||||||
|
- Measures: Difference in precision across groups
|
||||||
|
- Fair threshold: < 0.1
|
||||||
|
- Ensures positive predictions are equally accurate
|
||||||
|
|
||||||
|
6. **Calibration** (FNR equality)
|
||||||
|
- Measures: Difference in False Negative Rates
|
||||||
|
- Fair threshold: < 0.1
|
||||||
|
- Ensures balanced error rates
|
||||||
|
|
||||||
|
#### Group-Level Metrics (per demographic group):
|
||||||
|
|
||||||
|
- Positive Rate
|
||||||
|
- Selection Rate
|
||||||
|
- True Positive Rate (TPR/Recall/Sensitivity)
|
||||||
|
- False Positive Rate (FPR)
|
||||||
|
- True Negative Rate (TNR/Specificity)
|
||||||
|
- False Negative Rate (FNR)
|
||||||
|
- Precision (PPV)
|
||||||
|
- F1 Score
|
||||||
|
- Accuracy
|
||||||
|
- Sample Size & Distribution
|
||||||
|
|
||||||
|
### 3. **Weighted Bias Scoring**
|
||||||
|
|
||||||
|
The overall bias score (0-1, higher = more bias) is calculated using:
|
||||||
|
|
||||||
|
```python
|
||||||
|
Overall Score = Weighted Average of:
|
||||||
|
- Disparate Impact (weight: 1.5x sample_weight)
|
||||||
|
- Statistical Parity (weight: 1.0x sample_weight)
|
||||||
|
- Equal Opportunity (weight: 1.0x sample_weight)
|
||||||
|
- Equalized Odds (weight: 0.8x sample_weight)
|
||||||
|
- Predictive Parity (weight: 0.7x sample_weight)
|
||||||
|
- Calibration (weight: 0.7x sample_weight)
|
||||||
|
```
|
||||||
|
|
||||||
|
Sample weight = min(1.0, total_samples / 100)
|
||||||
|
|
||||||
|
### 4. **Intelligent Violation Detection**
|
||||||
|
|
||||||
|
Violations are categorized by severity:
|
||||||
|
|
||||||
|
- **CRITICAL**: di_value < 0.5, or deviation > 50%
|
||||||
|
- **HIGH**: di_value < 0.6, or deviation > 30%
|
||||||
|
- **MEDIUM**: di_value < 0.7, or deviation > 15%
|
||||||
|
- **LOW**: Minor deviations
|
||||||
|
|
||||||
|
Each violation includes:
|
||||||
|
- Affected groups
|
||||||
|
- Specific measurements
|
||||||
|
- Actionable recommendations
|
||||||
|
- Context-aware severity assessment
|
||||||
|
|
||||||
|
### 5. **Robust Data Handling**
|
||||||
|
|
||||||
|
#### Missing Values:
|
||||||
|
- Numerical: Filled with median
|
||||||
|
- Categorical: Filled with mode or 'Unknown'
|
||||||
|
- Comprehensive logging
|
||||||
|
|
||||||
|
#### Data Type Detection:
|
||||||
|
- Binary detection (0/1, Yes/No)
|
||||||
|
- Small discrete values (< 10 unique)
|
||||||
|
- High cardinality warnings (> 50 categories)
|
||||||
|
- Mixed type handling
|
||||||
|
|
||||||
|
#### Target Encoding:
|
||||||
|
- Automatic categorical → numeric conversion
|
||||||
|
- Binary value normalization
|
||||||
|
- Clear encoding maps printed
|
||||||
|
|
||||||
|
#### Class Imbalance:
|
||||||
|
- Stratified splitting when appropriate
|
||||||
|
- Minimum class size validation
|
||||||
|
- Balanced metrics calculation
|
||||||
|
|
||||||
|
### 6. **Enhanced Reporting**
|
||||||
|
|
||||||
|
Each analysis includes:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"overall_bias_score": 0.954,
|
||||||
|
"fairness_metrics": {
|
||||||
|
"Gender": {
|
||||||
|
"disparate_impact": {
|
||||||
|
"value": 0.276,
|
||||||
|
"threshold": 0.8,
|
||||||
|
"fair": false,
|
||||||
|
"min_group": "Female",
|
||||||
|
"max_group": "Male",
|
||||||
|
"min_rate": 0.25,
|
||||||
|
"max_rate": 0.906
|
||||||
|
},
|
||||||
|
"statistical_parity_difference": {...},
|
||||||
|
"equal_opportunity_difference": {...},
|
||||||
|
"equalized_odds": {...},
|
||||||
|
"predictive_parity": {...},
|
||||||
|
"calibration": {...},
|
||||||
|
"attribute_fairness_score": 0.89,
|
||||||
|
"group_metrics": {
|
||||||
|
"Male": {
|
||||||
|
"positive_rate": 0.906,
|
||||||
|
"tpr": 0.95,
|
||||||
|
"fpr": 0.03,
|
||||||
|
"precision": 0.92,
|
||||||
|
"f1_score": 0.93,
|
||||||
|
"sample_size": 450
|
||||||
|
},
|
||||||
|
"Female": {...}
|
||||||
|
},
|
||||||
|
"sample_statistics": {
|
||||||
|
"total_samples": 500,
|
||||||
|
"min_group_size": 50,
|
||||||
|
"max_group_size": 450,
|
||||||
|
"imbalance_ratio": 0.11,
|
||||||
|
"num_groups": 2
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"fairness_violations": [
|
||||||
|
{
|
||||||
|
"attribute": "Gender",
|
||||||
|
"metric": "Disparate Impact",
|
||||||
|
"severity": "CRITICAL",
|
||||||
|
"value": 0.276,
|
||||||
|
"affected_groups": ["Female", "Male"],
|
||||||
|
"message": "...",
|
||||||
|
"recommendation": "CRITICAL: Group 'Female' has less than half the approval rate..."
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Usage Examples
|
||||||
|
|
||||||
|
### Basic Analysis
|
||||||
|
|
||||||
|
```python
|
||||||
|
from ai_governance import AIGovernanceAnalyzer
|
||||||
|
|
||||||
|
# Initialize
|
||||||
|
analyzer = AIGovernanceAnalyzer()
|
||||||
|
|
||||||
|
# Analyze with protected attributes
|
||||||
|
report = analyzer.analyze(
|
||||||
|
df=your_dataframe,
|
||||||
|
target_column='ApprovalStatus',
|
||||||
|
protected_attributes=['Gender', 'Age', 'Race']
|
||||||
|
)
|
||||||
|
|
||||||
|
# Check bias score
|
||||||
|
print(f"Bias Score: {report['bias_analysis']['overall_bias_score']:.1%}")
|
||||||
|
|
||||||
|
# Review violations
|
||||||
|
for violation in report['bias_analysis']['fairness_violations']:
|
||||||
|
print(f"{violation['severity']}: {violation['message']}")
|
||||||
|
```
|
||||||
|
|
||||||
|
### With Presidio (Enhanced PII Detection)
|
||||||
|
|
||||||
|
```python
|
||||||
|
# Enable Presidio for automatic demographic detection
|
||||||
|
analyzer = AIGovernanceAnalyzer(use_presidio=True)
|
||||||
|
```
|
||||||
|
|
||||||
|
### API Usage
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -X POST http://localhost:8000/api/analyze \
|
||||||
|
-F "file=@dataset.csv" \
|
||||||
|
-F "target_column=Outcome" \
|
||||||
|
-F "protected_attributes=Gender,Age"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Interpreting Results
|
||||||
|
|
||||||
|
### Overall Bias Score
|
||||||
|
|
||||||
|
- **< 0.3**: Low bias - Excellent fairness ✅
|
||||||
|
- **0.3 - 0.5**: Moderate bias - Monitor recommended ⚠️
|
||||||
|
- **> 0.5**: High bias - Action required ❌
|
||||||
|
|
||||||
|
### Disparate Impact
|
||||||
|
|
||||||
|
- **0.8 - 1.25**: Fair (4/5ths rule satisfied)
|
||||||
|
- **< 0.8**: Disadvantaged group exists
|
||||||
|
- **> 1.25**: Advantaged group exists
|
||||||
|
|
||||||
|
### Statistical Parity
|
||||||
|
|
||||||
|
- **< 0.1**: Fair (similar positive rates)
|
||||||
|
- **> 0.1**: Groups receive different treatment
|
||||||
|
|
||||||
|
### Recommendations by Severity
|
||||||
|
|
||||||
|
#### CRITICAL
|
||||||
|
- **DO NOT DEPLOY** without remediation
|
||||||
|
- Investigate systemic bias sources
|
||||||
|
- Review training data representation
|
||||||
|
- Implement fairness constraints
|
||||||
|
- Consider re-collection if necessary
|
||||||
|
|
||||||
|
#### HIGH
|
||||||
|
- Address before deployment
|
||||||
|
- Use fairness-aware training methods
|
||||||
|
- Implement threshold optimization
|
||||||
|
- Regular monitoring required
|
||||||
|
|
||||||
|
#### MEDIUM
|
||||||
|
- Monitor closely
|
||||||
|
- Consider mitigation strategies
|
||||||
|
- Regular fairness audits
|
||||||
|
- Document findings
|
||||||
|
|
||||||
|
#### LOW
|
||||||
|
- Continue monitoring
|
||||||
|
- Maintain fairness standards
|
||||||
|
- Periodic reviews
|
||||||
|
|
||||||
|
## Best Practices
|
||||||
|
|
||||||
|
### 1. Data Collection
|
||||||
|
- Ensure representative sampling
|
||||||
|
- Balance protected groups when possible
|
||||||
|
- Document data sources
|
||||||
|
- Check for historical bias
|
||||||
|
|
||||||
|
### 2. Feature Engineering
|
||||||
|
- Avoid proxy features for protected attributes
|
||||||
|
- Check feature correlations with demographics
|
||||||
|
- Use feature importance analysis
|
||||||
|
- Consider fairness-aware feature selection
|
||||||
|
|
||||||
|
### 3. Model Training
|
||||||
|
- Use fairness-aware algorithms
|
||||||
|
- Implement fairness constraints
|
||||||
|
- Try multiple fairness definitions
|
||||||
|
- Cross-validate with fairness metrics
|
||||||
|
|
||||||
|
### 4. Post-Processing
|
||||||
|
- Threshold optimization per group
|
||||||
|
- Calibration techniques
|
||||||
|
- Reject option classification
|
||||||
|
- Regular bias audits
|
||||||
|
|
||||||
|
### 5. Monitoring
|
||||||
|
- Track fairness metrics over time
|
||||||
|
- Monitor for fairness drift
|
||||||
|
- Regular re-evaluation
|
||||||
|
- Document all findings
|
||||||
|
|
||||||
|
## Technical Details
|
||||||
|
|
||||||
|
### Dependencies
|
||||||
|
|
||||||
|
```
|
||||||
|
numpy>=1.21.0
|
||||||
|
pandas>=1.3.0
|
||||||
|
scikit-learn>=1.0.0
|
||||||
|
presidio-analyzer>=2.2.0 # Optional
|
||||||
|
spacy>=3.0.0 # Optional for Presidio
|
||||||
|
```
|
||||||
|
|
||||||
|
### Performance
|
||||||
|
|
||||||
|
- Handles datasets from 50 to 1M+ rows
|
||||||
|
- Adaptive algorithms scale with data size
|
||||||
|
- Memory-efficient group comparisons
|
||||||
|
- Parallel metric calculations
|
||||||
|
|
||||||
|
### Limitations
|
||||||
|
|
||||||
|
- Requires at least 2 groups per protected attribute
|
||||||
|
- Minimum 10 samples per group recommended
|
||||||
|
- Binary classification focus (multi-class supported)
|
||||||
|
- Assumes independent test set
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
### "Insufficient valid groups"
|
||||||
|
- Check protected attribute has at least 2 non-null groups
|
||||||
|
- Ensure groups appear in test set
|
||||||
|
- Increase test_size parameter
|
||||||
|
|
||||||
|
### "High cardinality warning"
|
||||||
|
- Feature has > 50 unique values
|
||||||
|
- Consider grouping categories
|
||||||
|
- May need feature engineering
|
||||||
|
|
||||||
|
### "Sample size too small"
|
||||||
|
- System adapts automatically
|
||||||
|
- Results may be less reliable
|
||||||
|
- Consider collecting more data
|
||||||
|
|
||||||
|
### "Presidio initialization failed"
|
||||||
|
- Install: `pip install presidio-analyzer spacy`
|
||||||
|
- Download model: `python -m spacy download en_core_web_sm`
|
||||||
|
- Or use `use_presidio=False`
|
||||||
|
|
||||||
|
## References
|
||||||
|
|
||||||
|
- [Fairness Definitions Explained](https://fairware.cs.umass.edu/papers/Verma.pdf)
|
||||||
|
- [4/5ths Rule (EEOC)](https://www.eeoc.gov/laws/guidance/questions-and-answers-clarify-and-provide-common-interpretation-uniform-guidelines)
|
||||||
|
- [Equalized Odds](https://arxiv.org/abs/1610.02413)
|
||||||
|
- [Fairness Through Awareness](https://arxiv.org/abs/1104.3913)
|
||||||
|
|
||||||
|
## Support
|
||||||
|
|
||||||
|
For issues or questions:
|
||||||
|
- Check logs for detailed diagnostic messages
|
||||||
|
- Review sample statistics in output
|
||||||
|
- Consult violation recommendations
|
||||||
|
- Contact: support@nordicprivacyai.com
|
||||||
@@ -86,14 +86,15 @@ class AIGovernanceAnalyzer:
|
|||||||
self.trainer.train()
|
self.trainer.train()
|
||||||
self.trainer.evaluate()
|
self.trainer.evaluate()
|
||||||
|
|
||||||
# Step 3: Analyze bias
|
# Step 3: Analyze bias (Presidio disabled by default to avoid initialization issues)
|
||||||
self.bias_analyzer = BiasAnalyzer(
|
self.bias_analyzer = BiasAnalyzer(
|
||||||
self.processor.X_test,
|
self.processor.X_test,
|
||||||
self.processor.y_test,
|
self.processor.y_test,
|
||||||
self.trainer.y_pred,
|
self.trainer.y_pred,
|
||||||
self.processor.df,
|
self.processor.df,
|
||||||
self.processor.protected_attributes,
|
self.processor.protected_attributes,
|
||||||
self.processor.target_column
|
self.processor.target_column,
|
||||||
|
use_presidio=False # Set to True to enable Presidio-enhanced detection
|
||||||
)
|
)
|
||||||
bias_results = self.bias_analyzer.analyze()
|
bias_results = self.bias_analyzer.analyze()
|
||||||
|
|
||||||
|
|||||||
@@ -1,16 +1,32 @@
|
|||||||
"""
|
"""
|
||||||
Bias Analyzer Module
|
Bias Analyzer Module
|
||||||
Detects and quantifies bias in ML models
|
Detects and quantifies bias in ML models using Presidio for enhanced demographic analysis
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import numpy as np
|
import numpy as np
|
||||||
import pandas as pd
|
import pandas as pd
|
||||||
from collections import defaultdict
|
from collections import defaultdict
|
||||||
|
from typing import List, Dict, Any, Optional
|
||||||
|
|
||||||
|
# Presidio imports
|
||||||
|
try:
|
||||||
|
from presidio_analyzer import AnalyzerEngine, Pattern, PatternRecognizer
|
||||||
|
from presidio_analyzer.nlp_engine import NlpEngineProvider
|
||||||
|
PRESIDIO_AVAILABLE = True
|
||||||
|
except ImportError:
|
||||||
|
PRESIDIO_AVAILABLE = False
|
||||||
|
print("⚠️ Presidio not available. Install with: pip install presidio-analyzer")
|
||||||
|
|
||||||
|
|
||||||
class BiasAnalyzer:
|
class BiasAnalyzer:
|
||||||
"""Analyze bias in ML model predictions"""
|
"""Analyze bias in ML model predictions with Presidio-enhanced demographic detection"""
|
||||||
|
|
||||||
def __init__(self, X_test, y_test, y_pred, original_df, protected_attributes, target_column):
|
# Class-level cache for Presidio analyzer
|
||||||
|
_presidio_analyzer = None
|
||||||
|
_presidio_initialized = False
|
||||||
|
_presidio_init_failed = False
|
||||||
|
|
||||||
|
def __init__(self, X_test, y_test, y_pred, original_df, protected_attributes, target_column, use_presidio=False):
|
||||||
self.X_test = X_test
|
self.X_test = X_test
|
||||||
self.y_test = y_test
|
self.y_test = y_test
|
||||||
self.y_pred = y_pred
|
self.y_pred = y_pred
|
||||||
@@ -18,20 +34,177 @@ class BiasAnalyzer:
|
|||||||
self.protected_attributes = protected_attributes
|
self.protected_attributes = protected_attributes
|
||||||
self.target_column = target_column
|
self.target_column = target_column
|
||||||
self.results = {}
|
self.results = {}
|
||||||
|
self.use_presidio = use_presidio
|
||||||
|
|
||||||
|
# Initialize Presidio only if requested and not already failed
|
||||||
|
if self.use_presidio and PRESIDIO_AVAILABLE and not BiasAnalyzer._presidio_init_failed:
|
||||||
|
if not BiasAnalyzer._presidio_initialized:
|
||||||
|
self._init_presidio()
|
||||||
|
self.analyzer = BiasAnalyzer._presidio_analyzer
|
||||||
|
else:
|
||||||
|
self.analyzer = None
|
||||||
|
|
||||||
|
def _init_presidio(self):
|
||||||
|
"""Initialize Presidio analyzer with demographic-specific recognizers (cached at class level)"""
|
||||||
|
try:
|
||||||
|
print("⏳ Initializing Presidio analyzer (first time only)...")
|
||||||
|
|
||||||
|
# Check if spaCy model is available
|
||||||
|
try:
|
||||||
|
import spacy
|
||||||
|
try:
|
||||||
|
spacy.load("en_core_web_sm")
|
||||||
|
except OSError:
|
||||||
|
print("⚠️ spaCy model 'en_core_web_sm' not found. Run: python -m spacy download en_core_web_sm")
|
||||||
|
BiasAnalyzer._presidio_init_failed = True
|
||||||
|
return
|
||||||
|
except ImportError:
|
||||||
|
print("⚠️ spaCy not installed. Install with: pip install spacy")
|
||||||
|
BiasAnalyzer._presidio_init_failed = True
|
||||||
|
return
|
||||||
|
|
||||||
|
# Create NLP engine
|
||||||
|
provider = NlpEngineProvider()
|
||||||
|
nlp_configuration = {
|
||||||
|
"nlp_engine_name": "spacy",
|
||||||
|
"models": [{"lang_code": "en", "model_name": "en_core_web_sm"}]
|
||||||
|
}
|
||||||
|
nlp_engine = provider.create_engine()
|
||||||
|
|
||||||
|
# Initialize analyzer
|
||||||
|
BiasAnalyzer._presidio_analyzer = AnalyzerEngine(nlp_engine=nlp_engine)
|
||||||
|
|
||||||
|
# Add custom recognizers for demographic attributes
|
||||||
|
self._add_demographic_recognizers()
|
||||||
|
|
||||||
|
BiasAnalyzer._presidio_initialized = True
|
||||||
|
print("✓ Presidio analyzer initialized successfully")
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
print(f"⚠️ Could not initialize Presidio: {e}")
|
||||||
|
print(" Continuing without Presidio-enhanced detection...")
|
||||||
|
BiasAnalyzer._presidio_init_failed = True
|
||||||
|
BiasAnalyzer._presidio_analyzer = None
|
||||||
|
|
||||||
|
def _add_demographic_recognizers(self):
|
||||||
|
"""Add custom recognizers for demographic attributes"""
|
||||||
|
if not BiasAnalyzer._presidio_analyzer:
|
||||||
|
return
|
||||||
|
|
||||||
|
# Gender recognizer
|
||||||
|
gender_patterns = [
|
||||||
|
Pattern(name="gender_explicit", regex=r"\b(male|female|non-binary|other|prefer not to say)\b", score=0.9),
|
||||||
|
Pattern(name="gender_pronouns", regex=r"\b(he/him|she/her|they/them)\b", score=0.7),
|
||||||
|
]
|
||||||
|
gender_recognizer = PatternRecognizer(
|
||||||
|
supported_entity="GENDER",
|
||||||
|
patterns=gender_patterns,
|
||||||
|
context=["gender", "sex"]
|
||||||
|
)
|
||||||
|
BiasAnalyzer._presidio_analyzer.registry.add_recognizer(gender_recognizer)
|
||||||
|
|
||||||
|
# Age group recognizer
|
||||||
|
age_patterns = [
|
||||||
|
Pattern(name="age_range", regex=r"\b(\d{1,2})-(\d{1,2})\b", score=0.8),
|
||||||
|
Pattern(name="age_group", regex=r"\b(under 18|18-24|25-34|35-44|45-54|55-64|65\+|senior|adult|teen)\b", score=0.9),
|
||||||
|
]
|
||||||
|
age_recognizer = PatternRecognizer(
|
||||||
|
supported_entity="AGE_GROUP",
|
||||||
|
patterns=age_patterns,
|
||||||
|
context=["age", "years old", "born"]
|
||||||
|
)
|
||||||
|
BiasAnalyzer._presidio_analyzer.registry.add_recognizer(age_recognizer)
|
||||||
|
|
||||||
|
# Ethnicity/Race recognizer
|
||||||
|
ethnicity_patterns = [
|
||||||
|
Pattern(name="ethnicity",
|
||||||
|
regex=r"\b(asian|black|white|hispanic|latino|latina|native american|pacific islander|african american|caucasian)\b",
|
||||||
|
score=0.8),
|
||||||
|
]
|
||||||
|
ethnicity_recognizer = PatternRecognizer(
|
||||||
|
supported_entity="ETHNICITY",
|
||||||
|
patterns=ethnicity_patterns,
|
||||||
|
context=["race", "ethnicity", "ethnic"]
|
||||||
|
)
|
||||||
|
BiasAnalyzer._presidio_analyzer.registry.add_recognizer(ethnicity_recognizer)
|
||||||
|
|
||||||
|
def detect_sensitive_attributes(self, df: pd.DataFrame) -> List[str]:
|
||||||
|
"""Use Presidio to detect columns containing sensitive demographic information"""
|
||||||
|
if not self.analyzer:
|
||||||
|
return []
|
||||||
|
|
||||||
|
sensitive_cols = []
|
||||||
|
|
||||||
|
for col in df.columns:
|
||||||
|
# Sample some values from the column
|
||||||
|
sample_values = df[col].dropna().astype(str).head(100).tolist()
|
||||||
|
sample_text = " ".join(sample_values)
|
||||||
|
|
||||||
|
# Analyze for demographic entities
|
||||||
|
results = self.analyzer.analyze(
|
||||||
|
text=sample_text,
|
||||||
|
language='en',
|
||||||
|
entities=["GENDER", "AGE_GROUP", "ETHNICITY", "PERSON", "LOCATION"]
|
||||||
|
)
|
||||||
|
|
||||||
|
if results:
|
||||||
|
entity_types = [r.entity_type for r in results]
|
||||||
|
print(f" Column '{col}' contains: {set(entity_types)}")
|
||||||
|
sensitive_cols.append(col)
|
||||||
|
|
||||||
|
return sensitive_cols
|
||||||
|
|
||||||
def analyze(self):
|
def analyze(self):
|
||||||
"""Perform comprehensive bias analysis"""
|
"""Perform comprehensive bias analysis with optional Presidio enhancement"""
|
||||||
|
print("\n" + "="*70)
|
||||||
|
print("🔍 BIAS ANALYSIS - FAIRNESS DETECTION")
|
||||||
|
print("="*70)
|
||||||
|
|
||||||
|
# Step 1: Use Presidio to detect additional sensitive attributes (if enabled)
|
||||||
|
if self.use_presidio and self.analyzer and PRESIDIO_AVAILABLE:
|
||||||
|
print("\nStep 1: Detecting sensitive demographic attributes with Presidio...")
|
||||||
|
try:
|
||||||
|
detected_sensitive = self.detect_sensitive_attributes(self.original_df)
|
||||||
|
|
||||||
|
# Add detected attributes to protected attributes if not already included
|
||||||
|
for attr in detected_sensitive:
|
||||||
|
if attr not in self.protected_attributes and attr != self.target_column:
|
||||||
|
print(f" ➕ Adding detected sensitive attribute: {attr}")
|
||||||
|
self.protected_attributes.append(attr)
|
||||||
|
except Exception as e:
|
||||||
|
print(f" ⚠️ Presidio detection failed: {e}")
|
||||||
|
print(" Continuing with manual protected attributes...")
|
||||||
|
else:
|
||||||
|
print("\nStep 1: Using manually specified protected attributes")
|
||||||
|
print(f" Protected attributes: {self.protected_attributes}")
|
||||||
|
|
||||||
|
# Step 2: Analyze demographic bias
|
||||||
|
print("\nStep 2: Analyzing demographic bias across groups...")
|
||||||
|
demographic_bias = self._analyze_demographic_bias()
|
||||||
|
|
||||||
|
# Step 3: Calculate fairness metrics
|
||||||
|
print("\nStep 3: Calculating fairness metrics...")
|
||||||
|
fairness_metrics = self._calculate_fairness_metrics()
|
||||||
|
|
||||||
|
# Step 4: Detect violations
|
||||||
|
print("\nStep 4: Detecting fairness violations...")
|
||||||
|
|
||||||
self.results = {
|
self.results = {
|
||||||
'demographic_bias': self._analyze_demographic_bias(),
|
'demographic_bias': demographic_bias,
|
||||||
'fairness_metrics': self._calculate_fairness_metrics(),
|
'fairness_metrics': fairness_metrics,
|
||||||
'fairness_violations': self._detect_fairness_violations(),
|
'fairness_violations': self._detect_fairness_violations(),
|
||||||
'fairness_assessment': self._assess_overall_fairness(),
|
'fairness_assessment': self._assess_overall_fairness(),
|
||||||
'overall_bias_score': 0.0
|
'overall_bias_score': 0.0,
|
||||||
|
'presidio_enhanced': self.use_presidio and PRESIDIO_AVAILABLE and self.analyzer is not None
|
||||||
}
|
}
|
||||||
|
|
||||||
# Calculate overall bias score
|
# Calculate overall bias score
|
||||||
self.results['overall_bias_score'] = self._calculate_overall_bias_score()
|
self.results['overall_bias_score'] = self._calculate_overall_bias_score()
|
||||||
|
|
||||||
|
print("\n" + "="*70)
|
||||||
|
print(f"✓ BIAS ANALYSIS COMPLETE - Score: {self.results['overall_bias_score']:.3f}")
|
||||||
|
print("="*70 + "\n")
|
||||||
|
|
||||||
return self.results
|
return self.results
|
||||||
|
|
||||||
def _analyze_demographic_bias(self):
|
def _analyze_demographic_bias(self):
|
||||||
@@ -76,56 +249,107 @@ class BiasAnalyzer:
|
|||||||
# Calculate accuracy for this group
|
# Calculate accuracy for this group
|
||||||
accuracy = np.mean(group_preds == group_true) if len(group_true) > 0 else 0
|
accuracy = np.mean(group_preds == group_true) if len(group_true) > 0 else 0
|
||||||
|
|
||||||
|
# Calculate false positive rate (FPR) and false negative rate (FNR)
|
||||||
|
if len(group_true) > 0:
|
||||||
|
# True positives and false positives
|
||||||
|
true_positives = np.sum((group_preds == 1) & (group_true == 1))
|
||||||
|
false_positives = np.sum((group_preds == 1) & (group_true == 0))
|
||||||
|
false_negatives = np.sum((group_preds == 0) & (group_true == 1))
|
||||||
|
true_negatives = np.sum((group_preds == 0) & (group_true == 0))
|
||||||
|
|
||||||
|
# Calculate rates
|
||||||
|
fpr = false_positives / (false_positives + true_negatives) if (false_positives + true_negatives) > 0 else 0
|
||||||
|
fnr = false_negatives / (false_negatives + true_positives) if (false_negatives + true_positives) > 0 else 0
|
||||||
|
precision = true_positives / (true_positives + false_positives) if (true_positives + false_positives) > 0 else 0
|
||||||
|
recall = true_positives / (true_positives + false_negatives) if (true_positives + false_negatives) > 0 else 0
|
||||||
|
else:
|
||||||
|
fpr = fnr = precision = recall = 0
|
||||||
|
|
||||||
group_metrics[str(group)] = {
|
group_metrics[str(group)] = {
|
||||||
'sample_size': len(group_preds),
|
'sample_size': len(group_preds),
|
||||||
'approval_rate': float(approval_rate),
|
'approval_rate': float(approval_rate),
|
||||||
'accuracy': float(accuracy),
|
'accuracy': float(accuracy),
|
||||||
|
'precision': float(precision),
|
||||||
|
'recall': float(recall),
|
||||||
|
'false_positive_rate': float(fpr),
|
||||||
|
'false_negative_rate': float(fnr),
|
||||||
'positive_predictions': int(np.sum(group_preds)),
|
'positive_predictions': int(np.sum(group_preds)),
|
||||||
'negative_predictions': int(len(group_preds) - np.sum(group_preds))
|
'negative_predictions': int(len(group_preds) - np.sum(group_preds))
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# Calculate statistical measures of disparity
|
||||||
|
if approval_rates:
|
||||||
|
rates_list = list(approval_rates.values())
|
||||||
|
max_disparity = max(rates_list) - min(rates_list)
|
||||||
|
mean_rate = np.mean(rates_list)
|
||||||
|
std_rate = np.std(rates_list)
|
||||||
|
coefficient_of_variation = (std_rate / mean_rate * 100) if mean_rate > 0 else 0
|
||||||
|
else:
|
||||||
|
max_disparity = mean_rate = std_rate = coefficient_of_variation = 0
|
||||||
|
|
||||||
bias_analysis[attr] = {
|
bias_analysis[attr] = {
|
||||||
'group_metrics': group_metrics,
|
'group_metrics': group_metrics,
|
||||||
'approval_rates': approval_rates,
|
'approval_rates': approval_rates,
|
||||||
'max_disparity': float(max(approval_rates.values()) - min(approval_rates.values())) if approval_rates else 0
|
'max_disparity': float(max_disparity),
|
||||||
|
'mean_approval_rate': float(mean_rate),
|
||||||
|
'std_approval_rate': float(std_rate),
|
||||||
|
'coefficient_of_variation': float(coefficient_of_variation),
|
||||||
|
'disparity_ratio': float(max(rates_list) / min(rates_list)) if rates_list and min(rates_list) > 0 else 1.0
|
||||||
}
|
}
|
||||||
|
|
||||||
return bias_analysis
|
return bias_analysis
|
||||||
|
|
||||||
def _calculate_fairness_metrics(self):
|
def _calculate_fairness_metrics(self):
|
||||||
"""Calculate standard fairness metrics"""
|
"""Calculate comprehensive fairness metrics with adaptive thresholds"""
|
||||||
fairness_metrics = {}
|
fairness_metrics = {}
|
||||||
|
|
||||||
|
print(f"\nCalculating fairness metrics for protected attributes: {self.protected_attributes}")
|
||||||
|
|
||||||
for attr in self.protected_attributes:
|
for attr in self.protected_attributes:
|
||||||
if attr not in self.original_df.columns:
|
if attr not in self.original_df.columns:
|
||||||
|
print(f" ⚠️ Attribute '{attr}' not found in dataframe")
|
||||||
continue
|
continue
|
||||||
|
|
||||||
groups = self.original_df[attr].unique()
|
groups = self.original_df[attr].unique()
|
||||||
|
# Remove NaN/None values from groups
|
||||||
|
groups = [g for g in groups if pd.notna(g)]
|
||||||
|
|
||||||
|
print(f" Analyzing '{attr}' with {len(groups)} groups: {list(groups)}")
|
||||||
|
|
||||||
if len(groups) < 2:
|
if len(groups) < 2:
|
||||||
|
print(f" ⚠️ Skipping '{attr}' - needs at least 2 groups")
|
||||||
continue
|
continue
|
||||||
|
|
||||||
# Get metrics for each group
|
# Get metrics for each group
|
||||||
group_data = {}
|
group_data = {}
|
||||||
|
valid_groups = []
|
||||||
|
|
||||||
for group in groups:
|
for group in groups:
|
||||||
|
# Handle different data types
|
||||||
|
if pd.isna(group):
|
||||||
|
continue
|
||||||
|
|
||||||
group_mask = self.original_df[attr] == group
|
group_mask = self.original_df[attr] == group
|
||||||
group_indices = self.original_df[group_mask].index
|
group_indices = self.original_df[group_mask].index
|
||||||
test_indices = self.X_test.index
|
test_indices = self.X_test.index
|
||||||
common_indices = group_indices.intersection(test_indices)
|
common_indices = group_indices.intersection(test_indices)
|
||||||
|
|
||||||
if len(common_indices) == 0:
|
if len(common_indices) == 0:
|
||||||
|
print(f" ⚠️ No test samples for group '{group}'")
|
||||||
continue
|
continue
|
||||||
|
|
||||||
group_pred_indices = [i for i, idx in enumerate(test_indices) if idx in common_indices]
|
group_pred_indices = [i for i, idx in enumerate(test_indices) if idx in common_indices]
|
||||||
group_preds = self.y_pred[group_pred_indices]
|
group_preds = self.y_pred[group_pred_indices]
|
||||||
group_true = self.y_test.iloc[group_pred_indices]
|
group_true = self.y_test.iloc[group_pred_indices].values
|
||||||
|
|
||||||
if len(group_preds) == 0:
|
if len(group_preds) == 0:
|
||||||
continue
|
continue
|
||||||
|
|
||||||
# Calculate metrics
|
# Calculate comprehensive metrics
|
||||||
positive_rate = np.mean(group_preds)
|
positive_rate = np.mean(group_preds)
|
||||||
|
negative_rate = 1 - positive_rate
|
||||||
|
|
||||||
# True positive rate (TPR) - Recall
|
# True positive rate (TPR) - Sensitivity/Recall
|
||||||
true_positives = np.sum((group_preds == 1) & (group_true == 1))
|
true_positives = np.sum((group_preds == 1) & (group_true == 1))
|
||||||
actual_positives = np.sum(group_true == 1)
|
actual_positives = np.sum(group_true == 1)
|
||||||
tpr = true_positives / actual_positives if actual_positives > 0 else 0
|
tpr = true_positives / actual_positives if actual_positives > 0 else 0
|
||||||
@@ -135,154 +359,527 @@ class BiasAnalyzer:
|
|||||||
actual_negatives = np.sum(group_true == 0)
|
actual_negatives = np.sum(group_true == 0)
|
||||||
fpr = false_positives / actual_negatives if actual_negatives > 0 else 0
|
fpr = false_positives / actual_negatives if actual_negatives > 0 else 0
|
||||||
|
|
||||||
|
# True negative rate (TNR) - Specificity
|
||||||
|
true_negatives = np.sum((group_preds == 0) & (group_true == 0))
|
||||||
|
tnr = true_negatives / actual_negatives if actual_negatives > 0 else 0
|
||||||
|
|
||||||
|
# False negative rate (FNR)
|
||||||
|
false_negatives = np.sum((group_preds == 0) & (group_true == 1))
|
||||||
|
fnr = false_negatives / actual_positives if actual_positives > 0 else 0
|
||||||
|
|
||||||
|
# Precision
|
||||||
|
precision = true_positives / (true_positives + false_positives) if (true_positives + false_positives) > 0 else 0
|
||||||
|
|
||||||
|
# F1 Score
|
||||||
|
f1 = 2 * (precision * tpr) / (precision + tpr) if (precision + tpr) > 0 else 0
|
||||||
|
|
||||||
|
# Accuracy
|
||||||
|
accuracy = (true_positives + true_negatives) / len(group_preds) if len(group_preds) > 0 else 0
|
||||||
|
|
||||||
|
# Selection rate (proportion of positive predictions)
|
||||||
|
selection_rate = np.mean(group_preds == 1)
|
||||||
|
|
||||||
group_data[str(group)] = {
|
group_data[str(group)] = {
|
||||||
'positive_rate': float(positive_rate),
|
'positive_rate': float(positive_rate),
|
||||||
|
'negative_rate': float(negative_rate),
|
||||||
|
'selection_rate': float(selection_rate),
|
||||||
'tpr': float(tpr),
|
'tpr': float(tpr),
|
||||||
'fpr': float(fpr),
|
'fpr': float(fpr),
|
||||||
'sample_size': len(group_preds)
|
'tnr': float(tnr),
|
||||||
|
'fnr': float(fnr),
|
||||||
|
'precision': float(precision),
|
||||||
|
'f1_score': float(f1),
|
||||||
|
'accuracy': float(accuracy),
|
||||||
|
'sample_size': int(len(group_preds)),
|
||||||
|
'positive_samples': int(actual_positives),
|
||||||
|
'negative_samples': int(actual_negatives)
|
||||||
}
|
}
|
||||||
|
valid_groups.append(str(group))
|
||||||
|
|
||||||
if len(group_data) < 2:
|
if len(group_data) < 2:
|
||||||
|
print(f" ⚠️ Insufficient valid groups for '{attr}'")
|
||||||
continue
|
continue
|
||||||
|
|
||||||
# Calculate disparate impact
|
# Calculate adaptive thresholds based on data characteristics
|
||||||
group_names = list(group_data.keys())
|
total_samples = sum(group_data[g]['sample_size'] for g in valid_groups)
|
||||||
reference_group = group_names[0]
|
min_group_size = min(group_data[g]['sample_size'] for g in valid_groups)
|
||||||
comparison_group = group_names[1]
|
max_group_size = max(group_data[g]['sample_size'] for g in valid_groups)
|
||||||
|
|
||||||
ref_positive_rate = group_data[reference_group]['positive_rate']
|
# Adjust thresholds for small sample sizes or imbalanced groups
|
||||||
comp_positive_rate = group_data[comparison_group]['positive_rate']
|
sample_size_factor = min(1.0, min_group_size / 30) # Relax thresholds for small samples
|
||||||
|
imbalance_factor = min_group_size / max_group_size if max_group_size > 0 else 1.0
|
||||||
|
|
||||||
disparate_impact = comp_positive_rate / ref_positive_rate if ref_positive_rate > 0 else 0
|
# Adaptive disparate impact threshold
|
||||||
|
di_threshold = 0.8 if sample_size_factor > 0.8 and imbalance_factor > 0.5 else 0.7
|
||||||
|
|
||||||
# Calculate statistical parity difference
|
# Adaptive statistical parity threshold
|
||||||
statistical_parity_diff = comp_positive_rate - ref_positive_rate
|
sp_threshold = 0.1 if sample_size_factor > 0.8 else 0.15
|
||||||
|
|
||||||
# Calculate equal opportunity difference
|
# Adaptive equal opportunity threshold
|
||||||
ref_tpr = group_data[reference_group]['tpr']
|
eo_threshold = 0.1 if sample_size_factor > 0.8 else 0.15
|
||||||
comp_tpr = group_data[comparison_group]['tpr']
|
|
||||||
equal_opportunity_diff = comp_tpr - ref_tpr
|
print(f" Adaptive thresholds: DI={di_threshold:.2f}, SP={sp_threshold:.2f}, EO={eo_threshold:.2f}")
|
||||||
|
print(f" Sample size factor: {sample_size_factor:.2f}, Imbalance factor: {imbalance_factor:.2f}")
|
||||||
|
|
||||||
|
# Calculate fairness metrics comparing ALL groups
|
||||||
|
positive_rates = [group_data[g]['positive_rate'] for g in valid_groups]
|
||||||
|
selection_rates = [group_data[g]['selection_rate'] for g in valid_groups]
|
||||||
|
tprs = [group_data[g]['tpr'] for g in valid_groups]
|
||||||
|
fprs = [group_data[g]['fpr'] for g in valid_groups]
|
||||||
|
fnrs = [group_data[g]['fnr'] for g in valid_groups]
|
||||||
|
|
||||||
|
print(f" Group positive rates: {dict(zip(valid_groups, [f'{r:.3f}' for r in positive_rates]))}")
|
||||||
|
|
||||||
|
# Find min and max rates
|
||||||
|
min_positive_rate = min(positive_rates) if positive_rates else 0
|
||||||
|
max_positive_rate = max(positive_rates) if positive_rates else 0
|
||||||
|
mean_positive_rate = np.mean(positive_rates) if positive_rates else 0
|
||||||
|
|
||||||
|
min_selection_rate = min(selection_rates) if selection_rates else 0
|
||||||
|
max_selection_rate = max(selection_rates) if selection_rates else 0
|
||||||
|
|
||||||
|
min_tpr = min(tprs) if tprs else 0
|
||||||
|
max_tpr = max(tprs) if tprs else 0
|
||||||
|
|
||||||
|
min_fpr = min(fprs) if fprs else 0
|
||||||
|
max_fpr = max(fprs) if fprs else 0
|
||||||
|
|
||||||
|
min_fnr = min(fnrs) if fnrs else 0
|
||||||
|
max_fnr = max(fnrs) if fnrs else 0
|
||||||
|
|
||||||
|
# 1. Disparate Impact (4/5ths rule)
|
||||||
|
disparate_impact = min_positive_rate / max_positive_rate if max_positive_rate > 0 else 1.0
|
||||||
|
di_fair = di_threshold <= disparate_impact <= (1/di_threshold)
|
||||||
|
|
||||||
|
# 2. Statistical Parity Difference
|
||||||
|
statistical_parity_diff = max_positive_rate - min_positive_rate
|
||||||
|
sp_fair = abs(statistical_parity_diff) < sp_threshold
|
||||||
|
|
||||||
|
# 3. Equal Opportunity (TPR equality)
|
||||||
|
equal_opportunity_diff = max_tpr - min_tpr
|
||||||
|
eo_fair = abs(equal_opportunity_diff) < eo_threshold
|
||||||
|
|
||||||
|
# 4. Equalized Odds (TPR and FPR equality)
|
||||||
|
fpr_diff = max_fpr - min_fpr
|
||||||
|
equalized_odds_fair = abs(equal_opportunity_diff) < eo_threshold and abs(fpr_diff) < eo_threshold
|
||||||
|
|
||||||
|
# 5. Predictive Parity (Precision equality)
|
||||||
|
precisions = [group_data[g]['precision'] for g in valid_groups]
|
||||||
|
min_precision = min(precisions) if precisions else 0
|
||||||
|
max_precision = max(precisions) if precisions else 0
|
||||||
|
precision_diff = max_precision - min_precision
|
||||||
|
predictive_parity_fair = abs(precision_diff) < sp_threshold
|
||||||
|
|
||||||
|
# 6. Calibration (FNR equality)
|
||||||
|
fnr_diff = max_fnr - min_fnr
|
||||||
|
calibration_fair = abs(fnr_diff) < eo_threshold
|
||||||
|
|
||||||
|
# Calculate overall fairness score for this attribute
|
||||||
|
fairness_scores = [
|
||||||
|
1.0 if di_fair else abs(1.0 - disparate_impact),
|
||||||
|
1.0 if sp_fair else abs(statistical_parity_diff),
|
||||||
|
1.0 if eo_fair else abs(equal_opportunity_diff),
|
||||||
|
1.0 if equalized_odds_fair else max(abs(equal_opportunity_diff), abs(fpr_diff)),
|
||||||
|
1.0 if predictive_parity_fair else abs(precision_diff),
|
||||||
|
1.0 if calibration_fair else abs(fnr_diff)
|
||||||
|
]
|
||||||
|
attribute_fairness_score = 1.0 - np.mean(fairness_scores)
|
||||||
|
|
||||||
|
print(f" Disparate Impact: {disparate_impact:.3f} {'✓ FAIR' if di_fair else '✗ UNFAIR'}")
|
||||||
|
print(f" Statistical Parity Diff: {statistical_parity_diff:.3f} {'✓ FAIR' if sp_fair else '✗ UNFAIR'}")
|
||||||
|
print(f" Equal Opportunity Diff: {equal_opportunity_diff:.3f} {'✓ FAIR' if eo_fair else '✗ UNFAIR'}")
|
||||||
|
print(f" Attribute Fairness Score: {attribute_fairness_score:.3f}")
|
||||||
|
|
||||||
fairness_metrics[attr] = {
|
fairness_metrics[attr] = {
|
||||||
'disparate_impact': {
|
'disparate_impact': {
|
||||||
'value': float(disparate_impact),
|
'value': float(disparate_impact),
|
||||||
'threshold': 0.8,
|
'threshold': float(di_threshold),
|
||||||
'fair': 0.8 <= disparate_impact <= 1.25,
|
'fair': bool(di_fair),
|
||||||
'interpretation': 'Ratio of positive rates between groups'
|
'interpretation': f'Ratio of minimum to maximum positive rates across {len(valid_groups)} groups',
|
||||||
|
'min_group': valid_groups[positive_rates.index(min_positive_rate)],
|
||||||
|
'max_group': valid_groups[positive_rates.index(max_positive_rate)],
|
||||||
|
'min_rate': float(min_positive_rate),
|
||||||
|
'max_rate': float(max_positive_rate)
|
||||||
},
|
},
|
||||||
'statistical_parity_difference': {
|
'statistical_parity_difference': {
|
||||||
'value': float(statistical_parity_diff),
|
'value': float(statistical_parity_diff),
|
||||||
'threshold': 0.1,
|
'threshold': float(sp_threshold),
|
||||||
'fair': abs(statistical_parity_diff) < 0.1,
|
'fair': bool(sp_fair),
|
||||||
'interpretation': 'Difference in positive rates'
|
'interpretation': f'Difference between maximum and minimum positive rates',
|
||||||
|
'mean_rate': float(mean_positive_rate)
|
||||||
},
|
},
|
||||||
'equal_opportunity_difference': {
|
'equal_opportunity_difference': {
|
||||||
'value': float(equal_opportunity_diff),
|
'value': float(equal_opportunity_diff),
|
||||||
'threshold': 0.1,
|
'threshold': float(eo_threshold),
|
||||||
'fair': abs(equal_opportunity_diff) < 0.1,
|
'fair': bool(eo_fair),
|
||||||
'interpretation': 'Difference in true positive rates'
|
'interpretation': f'Difference in true positive rates (recall) across groups'
|
||||||
},
|
},
|
||||||
'group_metrics': group_data
|
'equalized_odds': {
|
||||||
|
'tpr_diff': float(equal_opportunity_diff),
|
||||||
|
'fpr_diff': float(fpr_diff),
|
||||||
|
'fair': bool(equalized_odds_fair),
|
||||||
|
'interpretation': 'Both TPR and FPR should be equal across groups'
|
||||||
|
},
|
||||||
|
'predictive_parity': {
|
||||||
|
'precision_diff': float(precision_diff),
|
||||||
|
'fair': bool(predictive_parity_fair),
|
||||||
|
'interpretation': 'Precision should be equal across groups'
|
||||||
|
},
|
||||||
|
'calibration': {
|
||||||
|
'fnr_diff': float(fnr_diff),
|
||||||
|
'fair': bool(calibration_fair),
|
||||||
|
'interpretation': 'False negative rates should be equal across groups'
|
||||||
|
},
|
||||||
|
'attribute_fairness_score': float(attribute_fairness_score),
|
||||||
|
'group_metrics': group_data,
|
||||||
|
'sample_statistics': {
|
||||||
|
'total_samples': int(total_samples),
|
||||||
|
'min_group_size': int(min_group_size),
|
||||||
|
'max_group_size': int(max_group_size),
|
||||||
|
'imbalance_ratio': float(imbalance_factor),
|
||||||
|
'num_groups': int(len(valid_groups))
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
return fairness_metrics
|
return fairness_metrics
|
||||||
|
|
||||||
def _detect_fairness_violations(self):
|
def _detect_fairness_violations(self):
|
||||||
"""Detect specific fairness violations"""
|
"""Detect specific fairness violations with detailed analysis"""
|
||||||
violations = []
|
violations = []
|
||||||
|
|
||||||
fairness_metrics = self.results.get('fairness_metrics', {})
|
fairness_metrics = self.results.get('fairness_metrics', {})
|
||||||
|
|
||||||
for attr, metrics in fairness_metrics.items():
|
for attr, metrics in fairness_metrics.items():
|
||||||
# Check disparate impact
|
# Get sample statistics for context
|
||||||
|
sample_stats = metrics.get('sample_statistics', {})
|
||||||
|
num_groups = sample_stats.get('num_groups', 0)
|
||||||
|
imbalance_ratio = sample_stats.get('imbalance_ratio', 1.0)
|
||||||
|
|
||||||
|
# 1. Check disparate impact
|
||||||
di = metrics.get('disparate_impact', {})
|
di = metrics.get('disparate_impact', {})
|
||||||
if not di.get('fair', True):
|
if not di.get('fair', True):
|
||||||
|
severity = self._calculate_severity(
|
||||||
|
di['value'],
|
||||||
|
di['threshold'],
|
||||||
|
is_ratio=True,
|
||||||
|
imbalance_ratio=imbalance_ratio
|
||||||
|
)
|
||||||
|
|
||||||
|
min_group = di.get('min_group', 'Unknown')
|
||||||
|
max_group = di.get('max_group', 'Unknown')
|
||||||
|
min_rate = di.get('min_rate', 0)
|
||||||
|
max_rate = di.get('max_rate', 0)
|
||||||
|
|
||||||
violations.append({
|
violations.append({
|
||||||
'attribute': attr,
|
'attribute': attr,
|
||||||
'metric': 'Disparate Impact',
|
'metric': 'Disparate Impact',
|
||||||
'value': di['value'],
|
'value': di['value'],
|
||||||
'threshold': di['threshold'],
|
'threshold': di['threshold'],
|
||||||
'severity': 'HIGH' if di['value'] < 0.5 or di['value'] > 2.0 else 'MEDIUM',
|
'severity': severity,
|
||||||
'message': f"Disparate impact ratio of {di['value']:.3f} violates fairness threshold (0.8-1.25)"
|
'message': f"Disparate impact ratio of {di['value']:.3f} violates fairness threshold ({di['threshold']:.2f}-{1/di['threshold']:.2f}). Group '{min_group}' has {min_rate:.1%} approval vs '{max_group}' with {max_rate:.1%}.",
|
||||||
|
'affected_groups': [min_group, max_group],
|
||||||
|
'recommendation': self._get_di_recommendation(di['value'], min_group, max_group)
|
||||||
})
|
})
|
||||||
|
|
||||||
# Check statistical parity
|
# 2. Check statistical parity
|
||||||
spd = metrics.get('statistical_parity_difference', {})
|
spd = metrics.get('statistical_parity_difference', {})
|
||||||
if not spd.get('fair', True):
|
if not spd.get('fair', True):
|
||||||
|
severity = self._calculate_severity(
|
||||||
|
abs(spd['value']),
|
||||||
|
spd['threshold'],
|
||||||
|
is_ratio=False,
|
||||||
|
imbalance_ratio=imbalance_ratio
|
||||||
|
)
|
||||||
|
|
||||||
violations.append({
|
violations.append({
|
||||||
'attribute': attr,
|
'attribute': attr,
|
||||||
'metric': 'Statistical Parity',
|
'metric': 'Statistical Parity',
|
||||||
'value': spd['value'],
|
'value': spd['value'],
|
||||||
'threshold': spd['threshold'],
|
'threshold': spd['threshold'],
|
||||||
'severity': 'HIGH' if abs(spd['value']) > 0.2 else 'MEDIUM',
|
'severity': severity,
|
||||||
'message': f"Statistical parity difference of {spd['value']:.3f} exceeds threshold (0.1)"
|
'message': f"Statistical parity difference of {spd['value']:.3f} exceeds threshold (±{spd['threshold']:.2f}). There's a {abs(spd['value']):.1%} difference in positive prediction rates across groups.",
|
||||||
|
'recommendation': "Review feature importance and consider debiasing techniques like reweighting or threshold optimization."
|
||||||
})
|
})
|
||||||
|
|
||||||
# Check equal opportunity
|
# 3. Check equal opportunity
|
||||||
eod = metrics.get('equal_opportunity_difference', {})
|
eod = metrics.get('equal_opportunity_difference', {})
|
||||||
if not eod.get('fair', True):
|
if not eod.get('fair', True):
|
||||||
|
severity = self._calculate_severity(
|
||||||
|
abs(eod['value']),
|
||||||
|
eod['threshold'],
|
||||||
|
is_ratio=False,
|
||||||
|
imbalance_ratio=imbalance_ratio
|
||||||
|
)
|
||||||
|
|
||||||
violations.append({
|
violations.append({
|
||||||
'attribute': attr,
|
'attribute': attr,
|
||||||
'metric': 'Equal Opportunity',
|
'metric': 'Equal Opportunity',
|
||||||
'value': eod['value'],
|
'value': eod['value'],
|
||||||
'threshold': eod['threshold'],
|
'threshold': eod['threshold'],
|
||||||
'severity': 'HIGH' if abs(eod['value']) > 0.2 else 'MEDIUM',
|
'severity': severity,
|
||||||
'message': f"Equal opportunity difference of {eod['value']:.3f} exceeds threshold (0.1)"
|
'message': f"Equal opportunity difference of {eod['value']:.3f} exceeds threshold (±{eod['threshold']:.2f}). True positive rates vary by {abs(eod['value']):.1%} across groups.",
|
||||||
|
'recommendation': "Ensure the model has equal recall across protected groups. Consider adjusting decision thresholds per group."
|
||||||
})
|
})
|
||||||
|
|
||||||
|
# 4. Check equalized odds
|
||||||
|
eq_odds = metrics.get('equalized_odds', {})
|
||||||
|
if not eq_odds.get('fair', True):
|
||||||
|
tpr_diff = eq_odds.get('tpr_diff', 0)
|
||||||
|
fpr_diff = eq_odds.get('fpr_diff', 0)
|
||||||
|
max_diff = max(abs(tpr_diff), abs(fpr_diff))
|
||||||
|
|
||||||
|
severity = self._calculate_severity(
|
||||||
|
max_diff,
|
||||||
|
0.1,
|
||||||
|
is_ratio=False,
|
||||||
|
imbalance_ratio=imbalance_ratio
|
||||||
|
)
|
||||||
|
|
||||||
|
violations.append({
|
||||||
|
'attribute': attr,
|
||||||
|
'metric': 'Equalized Odds',
|
||||||
|
'value': max_diff,
|
||||||
|
'threshold': 0.1,
|
||||||
|
'severity': severity,
|
||||||
|
'message': f"Equalized odds violated: TPR differs by {abs(tpr_diff):.3f} and FPR differs by {abs(fpr_diff):.3f} across groups.",
|
||||||
|
'recommendation': "Both true positive and false positive rates should be balanced. Consider post-processing methods like reject option classification."
|
||||||
|
})
|
||||||
|
|
||||||
|
# 5. Check predictive parity
|
||||||
|
pred_parity = metrics.get('predictive_parity', {})
|
||||||
|
if not pred_parity.get('fair', True):
|
||||||
|
precision_diff = pred_parity.get('precision_diff', 0)
|
||||||
|
|
||||||
|
severity = self._calculate_severity(
|
||||||
|
abs(precision_diff),
|
||||||
|
0.1,
|
||||||
|
is_ratio=False,
|
||||||
|
imbalance_ratio=imbalance_ratio
|
||||||
|
)
|
||||||
|
|
||||||
|
violations.append({
|
||||||
|
'attribute': attr,
|
||||||
|
'metric': 'Predictive Parity',
|
||||||
|
'value': precision_diff,
|
||||||
|
'threshold': 0.1,
|
||||||
|
'severity': severity,
|
||||||
|
'message': f"Predictive parity difference of {precision_diff:.3f}. Precision varies by {abs(precision_diff):.1%} across groups.",
|
||||||
|
'recommendation': "Ensure positive predictions are equally accurate across groups. Review feature selection and calibration."
|
||||||
|
})
|
||||||
|
|
||||||
|
# 6. Check calibration (FNR equality)
|
||||||
|
calibration = metrics.get('calibration', {})
|
||||||
|
if not calibration.get('fair', True):
|
||||||
|
fnr_diff = calibration.get('fnr_diff', 0)
|
||||||
|
|
||||||
|
severity = self._calculate_severity(
|
||||||
|
abs(fnr_diff),
|
||||||
|
0.1,
|
||||||
|
is_ratio=False,
|
||||||
|
imbalance_ratio=imbalance_ratio
|
||||||
|
)
|
||||||
|
|
||||||
|
violations.append({
|
||||||
|
'attribute': attr,
|
||||||
|
'metric': 'Calibration (FNR)',
|
||||||
|
'value': fnr_diff,
|
||||||
|
'threshold': 0.1,
|
||||||
|
'severity': severity,
|
||||||
|
'message': f"False negative rates differ by {abs(fnr_diff):.3f} across groups, indicating poor calibration.",
|
||||||
|
'recommendation': "Calibrate model predictions to ensure equal false negative rates. Consider using calibration techniques like Platt scaling."
|
||||||
|
})
|
||||||
|
|
||||||
|
# Sort violations by severity
|
||||||
|
severity_order = {'CRITICAL': 0, 'HIGH': 1, 'MEDIUM': 2, 'LOW': 3}
|
||||||
|
violations.sort(key=lambda x: severity_order.get(x['severity'], 999))
|
||||||
|
|
||||||
return violations
|
return violations
|
||||||
|
|
||||||
def _assess_overall_fairness(self):
|
def _calculate_severity(self, value, threshold, is_ratio=False, imbalance_ratio=1.0):
|
||||||
"""Assess overall fairness of the model"""
|
"""Calculate violation severity based on value, threshold, and data characteristics"""
|
||||||
violations = self.results.get('fairness_violations', [])
|
if is_ratio:
|
||||||
|
# For disparate impact (ratio metric)
|
||||||
|
deviation = abs(1.0 - value)
|
||||||
|
if deviation > 0.5 or value < 0.4: # Very severe
|
||||||
|
return 'CRITICAL'
|
||||||
|
elif deviation > 0.3 or value < 0.6:
|
||||||
|
return 'HIGH'
|
||||||
|
elif deviation > 0.15:
|
||||||
|
return 'MEDIUM'
|
||||||
|
else:
|
||||||
|
return 'LOW'
|
||||||
|
else:
|
||||||
|
# For difference metrics
|
||||||
|
ratio = abs(value) / threshold if threshold > 0 else 0
|
||||||
|
|
||||||
|
# Adjust severity based on group imbalance
|
||||||
|
if imbalance_ratio < 0.3: # Highly imbalanced groups
|
||||||
|
if ratio > 3:
|
||||||
|
return 'CRITICAL'
|
||||||
|
elif ratio > 2:
|
||||||
|
return 'HIGH'
|
||||||
|
elif ratio > 1.5:
|
||||||
|
return 'MEDIUM'
|
||||||
|
else:
|
||||||
|
return 'LOW'
|
||||||
|
else:
|
||||||
|
if ratio > 2.5:
|
||||||
|
return 'CRITICAL'
|
||||||
|
elif ratio > 2:
|
||||||
|
return 'HIGH'
|
||||||
|
elif ratio > 1.2:
|
||||||
|
return 'MEDIUM'
|
||||||
|
else:
|
||||||
|
return 'LOW'
|
||||||
|
|
||||||
|
def _get_di_recommendation(self, di_value, min_group, max_group):
|
||||||
|
"""Get specific recommendation based on disparate impact value"""
|
||||||
|
if di_value < 0.5:
|
||||||
|
return f"CRITICAL: Group '{min_group}' has less than half the approval rate of '{max_group}'. Investigate for systemic bias. Consider: 1) Reviewing training data for representation issues, 2) Examining feature correlations with protected attribute, 3) Implementing fairness constraints during training."
|
||||||
|
elif di_value < 0.7:
|
||||||
|
return f"HIGH: Significant disparity between groups. Recommended actions: 1) Analyze feature importance per group, 2) Consider reweighting samples, 3) Explore threshold optimization, 4) Review data collection process for bias."
|
||||||
|
else:
|
||||||
|
return f"MEDIUM: Moderate disparity detected. Monitor closely and consider: 1) Regular fairness audits, 2) Collecting more diverse training data, 3) Using fairness-aware algorithms."
|
||||||
|
|
||||||
|
def _assess_overall_fairness(self):
|
||||||
|
"""Assess overall fairness of the model with weighted scoring"""
|
||||||
|
violations = self.results.get('fairness_violations', [])
|
||||||
|
fairness_metrics = self.results.get('fairness_metrics', {})
|
||||||
|
|
||||||
|
# Count violations by severity
|
||||||
|
critical_count = sum(1 for v in violations if v['severity'] == 'CRITICAL')
|
||||||
high_severity_count = sum(1 for v in violations if v['severity'] == 'HIGH')
|
high_severity_count = sum(1 for v in violations if v['severity'] == 'HIGH')
|
||||||
medium_severity_count = sum(1 for v in violations if v['severity'] == 'MEDIUM')
|
medium_severity_count = sum(1 for v in violations if v['severity'] == 'MEDIUM')
|
||||||
|
low_severity_count = sum(1 for v in violations if v['severity'] == 'LOW')
|
||||||
|
|
||||||
passes_threshold = high_severity_count == 0 and medium_severity_count <= 1
|
# Calculate attribute-level fairness scores
|
||||||
|
attribute_scores = []
|
||||||
|
for attr, metrics in fairness_metrics.items():
|
||||||
|
attr_score = metrics.get('attribute_fairness_score', 0)
|
||||||
|
attribute_scores.append(attr_score)
|
||||||
|
|
||||||
|
avg_attribute_score = np.mean(attribute_scores) if attribute_scores else 0
|
||||||
|
|
||||||
|
# Determine if passes threshold (stricter criteria)
|
||||||
|
passes_threshold = critical_count == 0 and high_severity_count == 0 and medium_severity_count <= 1
|
||||||
|
|
||||||
assessment = {
|
assessment = {
|
||||||
'passes_fairness_threshold': passes_threshold,
|
'passes_fairness_threshold': passes_threshold,
|
||||||
|
'critical_violations': critical_count,
|
||||||
'high_severity_violations': high_severity_count,
|
'high_severity_violations': high_severity_count,
|
||||||
'medium_severity_violations': medium_severity_count,
|
'medium_severity_violations': medium_severity_count,
|
||||||
|
'low_severity_violations': low_severity_count,
|
||||||
'total_violations': len(violations),
|
'total_violations': len(violations),
|
||||||
'recommendation': self._get_fairness_recommendation(high_severity_count, medium_severity_count)
|
'avg_attribute_fairness_score': float(avg_attribute_score),
|
||||||
|
'recommendation': self._get_fairness_recommendation(critical_count, high_severity_count, medium_severity_count)
|
||||||
}
|
}
|
||||||
|
|
||||||
return assessment
|
return assessment
|
||||||
|
|
||||||
def _get_fairness_recommendation(self, high_count, medium_count):
|
def _get_fairness_recommendation(self, critical_count, high_count, medium_count):
|
||||||
"""Get recommendation based on violation counts"""
|
"""Get recommendation based on violation counts"""
|
||||||
if high_count > 0:
|
if critical_count > 0:
|
||||||
return "CRITICAL: Immediate action required to address high-severity fairness violations"
|
return "CRITICAL: Severe bias detected. DO NOT deploy this model without addressing critical fairness violations. Immediate remediation required."
|
||||||
|
elif high_count > 0:
|
||||||
|
return "HIGH PRIORITY: Significant fairness violations detected. Address high-severity issues before deployment. Consider fairness-aware training methods."
|
||||||
elif medium_count > 2:
|
elif medium_count > 2:
|
||||||
return "WARNING: Multiple fairness issues detected. Review and address violations"
|
return "WARNING: Multiple fairness issues detected. Review and address violations before deployment. Regular monitoring recommended."
|
||||||
elif medium_count > 0:
|
elif medium_count > 0:
|
||||||
return "CAUTION: Minor fairness issues detected. Monitor and consider improvements"
|
return "CAUTION: Minor fairness issues detected. Monitor closely and consider improvements. Regular fairness audits recommended."
|
||||||
else:
|
else:
|
||||||
return "GOOD: No significant fairness violations detected"
|
return "GOOD: No significant fairness violations detected. Continue monitoring to maintain fairness standards."
|
||||||
|
|
||||||
def _calculate_overall_bias_score(self):
|
def _calculate_overall_bias_score(self):
|
||||||
"""Calculate overall bias score (0-1, lower is better)"""
|
"""Calculate comprehensive overall bias score (0-1, higher means more bias)"""
|
||||||
scores = []
|
scores = []
|
||||||
|
weights = []
|
||||||
|
|
||||||
# Score from fairness metrics
|
print("\nCalculating overall bias score...")
|
||||||
|
|
||||||
|
# Score from fairness metrics (weighted by multiple fairness criteria)
|
||||||
fairness_metrics = self.results.get('fairness_metrics', {})
|
fairness_metrics = self.results.get('fairness_metrics', {})
|
||||||
for attr, metrics in fairness_metrics.items():
|
for attr, metrics in fairness_metrics.items():
|
||||||
# Disparate impact score (deviation from 1.0)
|
sample_stats = metrics.get('sample_statistics', {})
|
||||||
|
num_groups = sample_stats.get('num_groups', 2)
|
||||||
|
total_samples = sample_stats.get('total_samples', 1)
|
||||||
|
|
||||||
|
# Calculate weight based on sample size (larger samples = more reliable = higher weight)
|
||||||
|
sample_weight = min(1.0, total_samples / 100)
|
||||||
|
|
||||||
|
# 1. Disparate Impact score (deviation from 1.0)
|
||||||
di_value = metrics.get('disparate_impact', {}).get('value', 1.0)
|
di_value = metrics.get('disparate_impact', {}).get('value', 1.0)
|
||||||
di_score = abs(1.0 - di_value)
|
di_threshold = metrics.get('disparate_impact', {}).get('threshold', 0.8)
|
||||||
scores.append(min(di_score, 1.0))
|
|
||||||
|
|
||||||
# Statistical parity score
|
if di_value < di_threshold:
|
||||||
|
di_score = (di_threshold - di_value) / di_threshold
|
||||||
|
elif di_value > (1 / di_threshold):
|
||||||
|
di_score = (di_value - (1 / di_threshold)) / (1 / di_threshold)
|
||||||
|
else:
|
||||||
|
di_score = 0
|
||||||
|
|
||||||
|
scores.append(di_score)
|
||||||
|
weights.append(sample_weight * 1.5) # Higher weight for disparate impact
|
||||||
|
print(f" {attr} - Disparate Impact: {di_value:.3f} → score: {di_score:.3f} (weight: {sample_weight * 1.5:.2f})")
|
||||||
|
|
||||||
|
# 2. Statistical Parity score
|
||||||
spd_value = abs(metrics.get('statistical_parity_difference', {}).get('value', 0))
|
spd_value = abs(metrics.get('statistical_parity_difference', {}).get('value', 0))
|
||||||
scores.append(min(spd_value * 5, 1.0)) # Scale to 0-1
|
spd_threshold = metrics.get('statistical_parity_difference', {}).get('threshold', 0.1)
|
||||||
|
spd_score = min(spd_value / spd_threshold, 1.0) if spd_threshold > 0 else 0
|
||||||
|
|
||||||
# Equal opportunity score
|
scores.append(spd_score)
|
||||||
|
weights.append(sample_weight)
|
||||||
|
print(f" {attr} - Statistical Parity Diff: {spd_value:.3f} → score: {spd_score:.3f} (weight: {sample_weight:.2f})")
|
||||||
|
|
||||||
|
# 3. Equal Opportunity score
|
||||||
eod_value = abs(metrics.get('equal_opportunity_difference', {}).get('value', 0))
|
eod_value = abs(metrics.get('equal_opportunity_difference', {}).get('value', 0))
|
||||||
scores.append(min(eod_value * 5, 1.0)) # Scale to 0-1
|
eod_threshold = metrics.get('equal_opportunity_difference', {}).get('threshold', 0.1)
|
||||||
|
eod_score = min(eod_value / eod_threshold, 1.0) if eod_threshold > 0 else 0
|
||||||
|
|
||||||
# Average all scores
|
scores.append(eod_score)
|
||||||
overall_score = np.mean(scores) if scores else 0.0
|
weights.append(sample_weight)
|
||||||
|
print(f" {attr} - Equal Opportunity Diff: {eod_value:.3f} → score: {eod_score:.3f} (weight: {sample_weight:.2f})")
|
||||||
|
|
||||||
|
# 4. Equalized Odds score
|
||||||
|
eq_odds = metrics.get('equalized_odds', {})
|
||||||
|
tpr_diff = abs(eq_odds.get('tpr_diff', 0))
|
||||||
|
fpr_diff = abs(eq_odds.get('fpr_diff', 0))
|
||||||
|
eq_odds_score = (min(tpr_diff / 0.1, 1.0) + min(fpr_diff / 0.1, 1.0)) / 2
|
||||||
|
|
||||||
|
scores.append(eq_odds_score)
|
||||||
|
weights.append(sample_weight * 0.8)
|
||||||
|
print(f" {attr} - Equalized Odds: {max(tpr_diff, fpr_diff):.3f} → score: {eq_odds_score:.3f} (weight: {sample_weight * 0.8:.2f})")
|
||||||
|
|
||||||
|
# 5. Predictive Parity score
|
||||||
|
pred_parity = metrics.get('predictive_parity', {})
|
||||||
|
precision_diff = abs(pred_parity.get('precision_diff', 0))
|
||||||
|
pred_parity_score = min(precision_diff / 0.1, 1.0)
|
||||||
|
|
||||||
|
scores.append(pred_parity_score)
|
||||||
|
weights.append(sample_weight * 0.7)
|
||||||
|
print(f" {attr} - Predictive Parity Diff: {precision_diff:.3f} → score: {pred_parity_score:.3f} (weight: {sample_weight * 0.7:.2f})")
|
||||||
|
|
||||||
|
# 6. Calibration score
|
||||||
|
calibration = metrics.get('calibration', {})
|
||||||
|
fnr_diff = abs(calibration.get('fnr_diff', 0))
|
||||||
|
calibration_score = min(fnr_diff / 0.1, 1.0)
|
||||||
|
|
||||||
|
scores.append(calibration_score)
|
||||||
|
weights.append(sample_weight * 0.7)
|
||||||
|
print(f" {attr} - Calibration (FNR): {fnr_diff:.3f} → score: {calibration_score:.3f} (weight: {sample_weight * 0.7:.2f})")
|
||||||
|
|
||||||
|
# Calculate weighted average
|
||||||
|
if scores and weights:
|
||||||
|
total_weight = sum(weights)
|
||||||
|
if total_weight > 0:
|
||||||
|
overall_score = sum(s * w for s, w in zip(scores, weights)) / total_weight
|
||||||
|
else:
|
||||||
|
overall_score = np.mean(scores)
|
||||||
|
else:
|
||||||
|
overall_score = 0.5 # Default if no metrics available
|
||||||
|
|
||||||
|
# Apply non-linear scaling to emphasize high bias
|
||||||
|
overall_score = min(overall_score ** 0.8, 1.0)
|
||||||
|
|
||||||
|
print(f"\n Overall Bias Score: {overall_score:.3f}")
|
||||||
|
|
||||||
return float(overall_score)
|
return float(overall_score)
|
||||||
|
|||||||
@@ -33,15 +33,37 @@ class DataProcessor:
|
|||||||
self._detect_column_types()
|
self._detect_column_types()
|
||||||
|
|
||||||
def _detect_column_types(self):
|
def _detect_column_types(self):
|
||||||
"""Automatically detect numerical and categorical columns"""
|
"""Automatically detect numerical and categorical columns with enhanced logic"""
|
||||||
for col in self.df.columns:
|
for col in self.df.columns:
|
||||||
|
# Skip if all null
|
||||||
|
if self.df[col].isnull().all():
|
||||||
|
continue
|
||||||
|
|
||||||
|
# Get non-null values for analysis
|
||||||
|
non_null_values = self.df[col].dropna()
|
||||||
|
|
||||||
|
if len(non_null_values) == 0:
|
||||||
|
continue
|
||||||
|
|
||||||
|
# Check data type
|
||||||
if self.df[col].dtype in ['int64', 'float64']:
|
if self.df[col].dtype in ['int64', 'float64']:
|
||||||
# Check if it's actually categorical (few unique values)
|
# Check if it's actually categorical despite being numeric
|
||||||
if self.df[col].nunique() < 10 and self.df[col].nunique() / len(self.df) < 0.05:
|
unique_count = non_null_values.nunique()
|
||||||
|
unique_ratio = unique_count / len(non_null_values) if len(non_null_values) > 0 else 0
|
||||||
|
|
||||||
|
# Heuristics for categorical detection:
|
||||||
|
# 1. Very few unique values (< 10)
|
||||||
|
# 2. Low unique ratio (< 5% of total)
|
||||||
|
# 3. Binary values (0/1, 1/2, etc.)
|
||||||
|
is_binary = unique_count == 2 and set(non_null_values.unique()).issubset({0, 1, 1.0, 0.0, 2, 1, 2.0})
|
||||||
|
is_small_discrete = unique_count < 10 and unique_ratio < 0.05
|
||||||
|
|
||||||
|
if is_binary or is_small_discrete:
|
||||||
self.categorical_features.append(col)
|
self.categorical_features.append(col)
|
||||||
else:
|
else:
|
||||||
self.numerical_features.append(col)
|
self.numerical_features.append(col)
|
||||||
else:
|
else:
|
||||||
|
# String, object, or category type
|
||||||
self.categorical_features.append(col)
|
self.categorical_features.append(col)
|
||||||
|
|
||||||
def _detect_pii_columns(self):
|
def _detect_pii_columns(self):
|
||||||
@@ -60,16 +82,47 @@ class DataProcessor:
|
|||||||
return pii_columns
|
return pii_columns
|
||||||
|
|
||||||
def prepare_data(self, test_size=0.2, random_state=42):
|
def prepare_data(self, test_size=0.2, random_state=42):
|
||||||
"""Prepare data for model training"""
|
"""Prepare data for model training with robust handling of edge cases"""
|
||||||
# Handle missing values
|
# Handle missing values - use different strategies based on data type
|
||||||
|
print(f"Initial dataset: {len(self.df)} rows, {len(self.df.columns)} columns")
|
||||||
|
|
||||||
|
# Count missing values before handling
|
||||||
|
missing_counts = self.df.isnull().sum()
|
||||||
|
cols_with_missing = missing_counts[missing_counts > 0]
|
||||||
|
if len(cols_with_missing) > 0:
|
||||||
|
print(f"Columns with missing values: {dict(cols_with_missing)}")
|
||||||
|
|
||||||
|
# For numerical columns: fill with median
|
||||||
|
for col in self.numerical_features:
|
||||||
|
if col in self.df.columns and self.df[col].isnull().any():
|
||||||
|
median_val = self.df[col].median()
|
||||||
|
self.df[col].fillna(median_val, inplace=True)
|
||||||
|
print(f" Filled {col} missing values with median: {median_val}")
|
||||||
|
|
||||||
|
# For categorical columns: fill with mode or 'Unknown'
|
||||||
|
for col in self.categorical_features:
|
||||||
|
if col in self.df.columns and self.df[col].isnull().any():
|
||||||
|
if self.df[col].mode().empty:
|
||||||
|
self.df[col].fillna('Unknown', inplace=True)
|
||||||
|
else:
|
||||||
|
mode_val = self.df[col].mode()[0]
|
||||||
|
self.df[col].fillna(mode_val, inplace=True)
|
||||||
|
print(f" Filled {col} missing values with mode: {mode_val}")
|
||||||
|
|
||||||
|
# Drop rows with remaining missing values
|
||||||
|
rows_before = len(self.df)
|
||||||
self.df = self.df.dropna()
|
self.df = self.df.dropna()
|
||||||
|
rows_dropped = rows_before - len(self.df)
|
||||||
|
if rows_dropped > 0:
|
||||||
|
print(f"Dropped {rows_dropped} rows with missing values")
|
||||||
|
|
||||||
# Separate features and target
|
# Separate features and target
|
||||||
if self.target_column is None:
|
if self.target_column is None:
|
||||||
# Auto-detect target (last column or column with 'target', 'label', 'status')
|
# Auto-detect target (last column or column with 'target', 'label', 'status')
|
||||||
target_candidates = [col for col in self.df.columns
|
target_candidates = [col for col in self.df.columns
|
||||||
if any(keyword in col.lower() for keyword in ['target', 'label', 'status', 'class'])]
|
if any(keyword in col.lower() for keyword in ['target', 'label', 'status', 'class', 'outcome', 'result'])]
|
||||||
self.target_column = target_candidates[0] if target_candidates else self.df.columns[-1]
|
self.target_column = target_candidates[0] if target_candidates else self.df.columns[-1]
|
||||||
|
print(f"Auto-detected target column: {self.target_column}")
|
||||||
|
|
||||||
# Prepare features
|
# Prepare features
|
||||||
feature_cols = [col for col in self.df.columns if col != self.target_column]
|
feature_cols = [col for col in self.df.columns if col != self.target_column]
|
||||||
@@ -80,27 +133,65 @@ class DataProcessor:
|
|||||||
if y.dtype == 'object' or y.dtype.name == 'category':
|
if y.dtype == 'object' or y.dtype.name == 'category':
|
||||||
self.target_encoder = LabelEncoder()
|
self.target_encoder = LabelEncoder()
|
||||||
y_encoded = self.target_encoder.fit_transform(y)
|
y_encoded = self.target_encoder.fit_transform(y)
|
||||||
y = pd.Series(y_encoded, index=y.index)
|
y = pd.Series(y_encoded, index=y.index, name=self.target_column)
|
||||||
print(f"Target '{self.target_column}' encoded: {dict(enumerate(self.target_encoder.classes_))}")
|
encoding_map = dict(enumerate(self.target_encoder.classes_))
|
||||||
|
print(f"Target '{self.target_column}' encoded: {encoding_map}")
|
||||||
|
elif y.dtype in ['float64', 'int64']:
|
||||||
|
# Check if numeric target needs binarization
|
||||||
|
unique_values = y.unique()
|
||||||
|
if len(unique_values) == 2:
|
||||||
|
print(f"Binary target detected with values: {sorted(unique_values)}")
|
||||||
|
# Ensure 0/1 encoding
|
||||||
|
if not set(unique_values).issubset({0, 1}):
|
||||||
|
min_val = min(unique_values)
|
||||||
|
y = (y != min_val).astype(int)
|
||||||
|
print(f"Converted to 0/1 encoding (1 = positive class)")
|
||||||
|
|
||||||
# Encode categorical variables
|
# Encode categorical variables with better handling
|
||||||
for col in self.categorical_features:
|
for col in self.categorical_features:
|
||||||
if col in X.columns:
|
if col in X.columns:
|
||||||
|
# Handle high cardinality features
|
||||||
|
unique_count = X[col].nunique()
|
||||||
|
if unique_count > 50:
|
||||||
|
print(f" ⚠️ High cardinality feature '{col}' ({unique_count} unique values) - consider feature engineering")
|
||||||
|
|
||||||
le = LabelEncoder()
|
le = LabelEncoder()
|
||||||
|
# Convert to string to handle mixed types
|
||||||
X[col] = le.fit_transform(X[col].astype(str))
|
X[col] = le.fit_transform(X[col].astype(str))
|
||||||
self.encoders[col] = le
|
self.encoders[col] = le
|
||||||
|
print(f"Encoded '{col}': {unique_count} categories")
|
||||||
|
|
||||||
# Store feature names
|
# Store feature names
|
||||||
self.feature_names = X.columns.tolist()
|
self.feature_names = X.columns.tolist()
|
||||||
|
|
||||||
|
# Check class balance
|
||||||
|
class_counts = y.value_counts()
|
||||||
|
print(f"\nTarget distribution:")
|
||||||
|
for val, count in class_counts.items():
|
||||||
|
print(f" Class {val}: {count} ({count/len(y)*100:.1f}%)")
|
||||||
|
|
||||||
|
# Determine if stratification is needed
|
||||||
|
min_class_count = class_counts.min()
|
||||||
|
use_stratify = y.nunique() < 10 and min_class_count >= 2
|
||||||
|
|
||||||
# Split data
|
# Split data
|
||||||
self.X_train, self.X_test, self.y_train, self.y_test = train_test_split(
|
if use_stratify:
|
||||||
X, y, test_size=test_size, random_state=random_state, stratify=y if y.nunique() < 10 else None
|
print(f"Using stratified split (min class count: {min_class_count})")
|
||||||
)
|
self.X_train, self.X_test, self.y_train, self.y_test = train_test_split(
|
||||||
|
X, y, test_size=test_size, random_state=random_state, stratify=y
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
print(f"Using random split (class imbalance or regression)")
|
||||||
|
self.X_train, self.X_test, self.y_train, self.y_test = train_test_split(
|
||||||
|
X, y, test_size=test_size, random_state=random_state
|
||||||
|
)
|
||||||
|
|
||||||
|
print(f"Train set: {len(self.X_train)} samples, Test set: {len(self.X_test)} samples")
|
||||||
|
|
||||||
# Scale numerical features
|
# Scale numerical features
|
||||||
numerical_cols = [col for col in self.numerical_features if col in self.X_train.columns]
|
numerical_cols = [col for col in self.numerical_features if col in self.X_train.columns]
|
||||||
if numerical_cols:
|
if numerical_cols:
|
||||||
|
print(f"Scaling {len(numerical_cols)} numerical features")
|
||||||
self.X_train[numerical_cols] = self.scaler.fit_transform(self.X_train[numerical_cols])
|
self.X_train[numerical_cols] = self.scaler.fit_transform(self.X_train[numerical_cols])
|
||||||
self.X_test[numerical_cols] = self.scaler.transform(self.X_test[numerical_cols])
|
self.X_test[numerical_cols] = self.scaler.transform(self.X_test[numerical_cols])
|
||||||
|
|
||||||
|
|||||||
@@ -97,6 +97,10 @@ async def analyze_dataset(file: UploadFile = File(...)):
|
|||||||
analyzer.save_report(report, full_report_path)
|
analyzer.save_report(report, full_report_path)
|
||||||
|
|
||||||
# Prepare response with summary
|
# Prepare response with summary
|
||||||
|
bias_analysis = report.get("bias_analysis", {})
|
||||||
|
model_metrics = report.get("model_performance", {}).get("metrics", {})
|
||||||
|
risk_assessment = report.get("risk_assessment", {})
|
||||||
|
|
||||||
response_data = {
|
response_data = {
|
||||||
"status": "success",
|
"status": "success",
|
||||||
"filename": file.filename,
|
"filename": file.filename,
|
||||||
@@ -106,29 +110,35 @@ async def analyze_dataset(file: UploadFile = File(...)):
|
|||||||
"features": list(df.columns)
|
"features": list(df.columns)
|
||||||
},
|
},
|
||||||
"model_performance": {
|
"model_performance": {
|
||||||
"accuracy": report.get("model_metrics", {}).get("accuracy", 0),
|
"accuracy": model_metrics.get("accuracy", 0),
|
||||||
"precision": report.get("model_metrics", {}).get("precision", 0),
|
"precision": model_metrics.get("precision", 0),
|
||||||
"recall": report.get("model_metrics", {}).get("recall", 0),
|
"recall": model_metrics.get("recall", 0),
|
||||||
"f1_score": report.get("model_metrics", {}).get("f1_score", 0)
|
"f1_score": model_metrics.get("f1_score", 0)
|
||||||
},
|
},
|
||||||
"bias_metrics": {
|
"bias_metrics": {
|
||||||
"overall_bias_score": report.get("bias_metrics", {}).get("overall_bias_score", 0),
|
"overall_bias_score": bias_analysis.get("overall_bias_score", 0),
|
||||||
"disparate_impact": report.get("bias_metrics", {}).get("disparate_impact", {}),
|
"disparate_impact": bias_analysis.get("fairness_metrics", {}),
|
||||||
"statistical_parity": report.get("bias_metrics", {}).get("statistical_parity_difference", {}),
|
"statistical_parity": bias_analysis.get("fairness_metrics", {}),
|
||||||
"violations_detected": report.get("bias_metrics", {}).get("fairness_violations", [])
|
"violations_detected": bias_analysis.get("fairness_violations", [])
|
||||||
},
|
},
|
||||||
"risk_assessment": {
|
"risk_assessment": {
|
||||||
"overall_risk_score": report.get("risk_metrics", {}).get("overall_risk_score", 0),
|
"overall_risk_score": risk_assessment.get("overall_risk_score", 0),
|
||||||
"privacy_risks": report.get("risk_metrics", {}).get("privacy_risks", []),
|
"privacy_risks": risk_assessment.get("privacy_risks", []),
|
||||||
"ethical_risks": report.get("risk_metrics", {}).get("ethical_risks", []),
|
"ethical_risks": risk_assessment.get("ethical_risks", []),
|
||||||
"compliance_risks": report.get("risk_metrics", {}).get("compliance_risks", []),
|
"compliance_risks": risk_assessment.get("risk_categories", {}).get("compliance_risks", []),
|
||||||
"data_quality_risks": report.get("risk_metrics", {}).get("data_quality_risks", [])
|
"data_quality_risks": risk_assessment.get("risk_categories", {}).get("data_quality_risks", [])
|
||||||
},
|
},
|
||||||
"recommendations": report.get("recommendations", []),
|
"recommendations": report.get("recommendations", []),
|
||||||
"report_file": f"/{report_path}",
|
"report_file": f"/{report_path}",
|
||||||
"timestamp": datetime.now().isoformat()
|
"timestamp": datetime.now().isoformat()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# Debug: Print bias metrics being sent to frontend
|
||||||
|
print(f"\n📊 Sending bias metrics to frontend:")
|
||||||
|
print(f" Overall Bias Score: {response_data['bias_metrics']['overall_bias_score']:.3f}")
|
||||||
|
print(f" Violations: {len(response_data['bias_metrics']['violations_detected'])}")
|
||||||
|
print(f" Fairness Metrics: {len(response_data['bias_metrics']['disparate_impact'])} attributes")
|
||||||
|
|
||||||
# Convert all numpy/pandas types to native Python types
|
# Convert all numpy/pandas types to native Python types
|
||||||
response_data = convert_to_serializable(response_data)
|
response_data = convert_to_serializable(response_data)
|
||||||
|
|
||||||
|
|||||||
@@ -455,45 +455,261 @@ export function CenterPanel({ tab, onAnalyze }: CenterPanelProps) {
|
|||||||
);
|
);
|
||||||
case "bias-analysis":
|
case "bias-analysis":
|
||||||
return (
|
return (
|
||||||
<div className="space-y-4">
|
<div className="space-y-6">
|
||||||
<h2 className="text-xl font-semibold">Bias Analysis</h2>
|
<div>
|
||||||
|
<h2 className="text-2xl font-bold mb-2">Bias & Fairness Analysis</h2>
|
||||||
|
<p className="text-sm text-slate-600">Comprehensive evaluation of algorithmic fairness across demographic groups</p>
|
||||||
|
</div>
|
||||||
|
|
||||||
{analyzeResult ? (
|
{analyzeResult ? (
|
||||||
<div className="space-y-4">
|
<div className="space-y-6">
|
||||||
<div className="grid grid-cols-2 gap-4">
|
{/* Overall Bias Score Card */}
|
||||||
<div className="p-4 bg-white rounded-lg border">
|
<div className="p-6 bg-gradient-to-br from-purple-50 to-indigo-50 rounded-xl border-2 border-purple-200">
|
||||||
<div className="text-sm text-slate-600">Overall Bias Score</div>
|
<div className="flex items-start justify-between">
|
||||||
<div className="text-2xl font-bold">{(analyzeResult.bias_metrics.overall_bias_score * 100).toFixed(1)}%</div>
|
<div>
|
||||||
|
<div className="text-sm font-medium text-purple-700 mb-1">Overall Bias Score</div>
|
||||||
|
<div className="text-5xl font-bold text-purple-900">
|
||||||
|
{(analyzeResult.bias_metrics.overall_bias_score * 100).toFixed(1)}%
|
||||||
|
</div>
|
||||||
|
<div className="mt-3 flex items-center gap-2">
|
||||||
|
{analyzeResult.bias_metrics.overall_bias_score < 0.3 ? (
|
||||||
|
<>
|
||||||
|
<span className="px-3 py-1 bg-green-100 text-green-800 text-xs font-semibold rounded-full">
|
||||||
|
✓ Low Bias
|
||||||
|
</span>
|
||||||
|
<span className="text-sm text-slate-600">Excellent fairness</span>
|
||||||
|
</>
|
||||||
|
) : analyzeResult.bias_metrics.overall_bias_score < 0.5 ? (
|
||||||
|
<>
|
||||||
|
<span className="px-3 py-1 bg-yellow-100 text-yellow-800 text-xs font-semibold rounded-full">
|
||||||
|
⚠ Moderate Bias
|
||||||
|
</span>
|
||||||
|
<span className="text-sm text-slate-600">Monitor recommended</span>
|
||||||
|
</>
|
||||||
|
) : (
|
||||||
|
<>
|
||||||
|
<span className="px-3 py-1 bg-red-100 text-red-800 text-xs font-semibold rounded-full">
|
||||||
|
✗ High Bias
|
||||||
|
</span>
|
||||||
|
<span className="text-sm text-slate-600">Action required</span>
|
||||||
|
</>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
<div className="text-right">
|
||||||
|
<div className="text-sm text-slate-600 mb-1">Violations</div>
|
||||||
|
<div className={`text-3xl font-bold ${analyzeResult.bias_metrics.violations_detected.length > 0 ? 'text-red-600' : 'text-green-600'}`}>
|
||||||
|
{analyzeResult.bias_metrics.violations_detected.length}
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
</div>
|
</div>
|
||||||
<div className="p-4 bg-white rounded-lg border">
|
|
||||||
<div className="text-sm text-slate-600">Violations Detected</div>
|
{/* Interpretation */}
|
||||||
<div className="text-2xl font-bold">{analyzeResult.bias_metrics.violations_detected.length}</div>
|
<div className="mt-4 p-4 bg-white/70 rounded-lg">
|
||||||
|
<div className="text-xs font-semibold text-purple-800 mb-1">INTERPRETATION</div>
|
||||||
|
<p className="text-sm text-slate-700">
|
||||||
|
{analyzeResult.bias_metrics.overall_bias_score < 0.3
|
||||||
|
? "Your model demonstrates strong fairness across demographic groups. Continue monitoring to ensure consistent performance."
|
||||||
|
: analyzeResult.bias_metrics.overall_bias_score < 0.5
|
||||||
|
? "Moderate bias detected. Review fairness metrics below and consider implementing mitigation strategies to reduce disparities."
|
||||||
|
: "Significant bias detected. Immediate action required to address fairness concerns before deployment. Review all violation details below."}
|
||||||
|
</p>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
<div className="p-4 bg-white rounded-lg border">
|
{/* Model Performance Metrics */}
|
||||||
<h3 className="font-semibold mb-2">Model Performance</h3>
|
<div className="p-6 bg-white rounded-xl border border-slate-200 shadow-sm">
|
||||||
<div className="grid grid-cols-4 gap-2 text-sm">
|
<h3 className="font-bold text-lg mb-4 flex items-center gap-2">
|
||||||
<div>
|
<span className="text-blue-600">📊</span>
|
||||||
<div className="text-slate-600">Accuracy</div>
|
Model Performance Metrics
|
||||||
<div className="font-medium">{(analyzeResult.model_performance.accuracy * 100).toFixed(1)}%</div>
|
</h3>
|
||||||
|
<div className="grid grid-cols-2 md:grid-cols-4 gap-4">
|
||||||
|
<div className="p-4 bg-blue-50 rounded-lg">
|
||||||
|
<div className="text-xs text-blue-700 font-semibold mb-1">ACCURACY</div>
|
||||||
|
<div className="text-2xl font-bold text-blue-900">{(analyzeResult.model_performance.accuracy * 100).toFixed(1)}%</div>
|
||||||
|
<div className="text-xs text-slate-600 mt-1">Overall correctness</div>
|
||||||
</div>
|
</div>
|
||||||
<div>
|
<div className="p-4 bg-green-50 rounded-lg">
|
||||||
<div className="text-slate-600">Precision</div>
|
<div className="text-xs text-green-700 font-semibold mb-1">PRECISION</div>
|
||||||
<div className="font-medium">{(analyzeResult.model_performance.precision * 100).toFixed(1)}%</div>
|
<div className="text-2xl font-bold text-green-900">{(analyzeResult.model_performance.precision * 100).toFixed(1)}%</div>
|
||||||
|
<div className="text-xs text-slate-600 mt-1">Positive prediction accuracy</div>
|
||||||
</div>
|
</div>
|
||||||
<div>
|
<div className="p-4 bg-purple-50 rounded-lg">
|
||||||
<div className="text-slate-600">Recall</div>
|
<div className="text-xs text-purple-700 font-semibold mb-1">RECALL</div>
|
||||||
<div className="font-medium">{(analyzeResult.model_performance.recall * 100).toFixed(1)}%</div>
|
<div className="text-2xl font-bold text-purple-900">{(analyzeResult.model_performance.recall * 100).toFixed(1)}%</div>
|
||||||
|
<div className="text-xs text-slate-600 mt-1">True positive detection rate</div>
|
||||||
</div>
|
</div>
|
||||||
<div>
|
<div className="p-4 bg-orange-50 rounded-lg">
|
||||||
<div className="text-slate-600">F1 Score</div>
|
<div className="text-xs text-orange-700 font-semibold mb-1">F1 SCORE</div>
|
||||||
<div className="font-medium">{(analyzeResult.model_performance.f1_score * 100).toFixed(1)}%</div>
|
<div className="text-2xl font-bold text-orange-900">{(analyzeResult.model_performance.f1_score * 100).toFixed(1)}%</div>
|
||||||
|
<div className="text-xs text-slate-600 mt-1">Balanced metric</div>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
|
{/* Fairness Metrics */}
|
||||||
|
{Object.keys(analyzeResult.bias_metrics.disparate_impact).length > 0 && (
|
||||||
|
<div className="p-6 bg-white rounded-xl border border-slate-200 shadow-sm">
|
||||||
|
<h3 className="font-bold text-lg mb-4 flex items-center gap-2">
|
||||||
|
<span className="text-purple-600">⚖️</span>
|
||||||
|
Fairness Metrics by Protected Attribute
|
||||||
|
</h3>
|
||||||
|
|
||||||
|
{Object.entries(analyzeResult.bias_metrics.disparate_impact).map(([attr, metrics]: [string, any]) => (
|
||||||
|
<div key={attr} className="mb-6 last:mb-0 p-4 bg-slate-50 rounded-lg">
|
||||||
|
<div className="font-semibold text-slate-800 mb-3 flex items-center gap-2">
|
||||||
|
<span className="px-2 py-1 bg-purple-100 text-purple-800 text-xs rounded">
|
||||||
|
{attr.toUpperCase()}
|
||||||
|
</span>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
{/* Disparate Impact */}
|
||||||
|
{metrics?.disparate_impact?.value !== undefined && (
|
||||||
|
<div className="mb-3 p-3 bg-white rounded border border-slate-200">
|
||||||
|
<div className="flex items-center justify-between mb-2">
|
||||||
|
<div>
|
||||||
|
<div className="text-xs font-semibold text-slate-600">DISPARATE IMPACT RATIO</div>
|
||||||
|
<div className="text-2xl font-bold text-slate-900">{metrics.disparate_impact.value.toFixed(3)}</div>
|
||||||
|
</div>
|
||||||
|
<div className={`px-3 py-1 rounded-full text-xs font-semibold ${
|
||||||
|
metrics.disparate_impact.fair ? 'bg-green-100 text-green-800' : 'bg-red-100 text-red-800'
|
||||||
|
}`}>
|
||||||
|
{metrics.disparate_impact.fair ? '✓ FAIR' : '✗ UNFAIR'}
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
<div className="text-xs text-slate-600 mb-2">{metrics.disparate_impact.interpretation || 'Ratio of positive rates between groups'}</div>
|
||||||
|
<div className="text-xs text-slate-500 bg-blue-50 p-2 rounded">
|
||||||
|
<strong>Fair Range:</strong> {metrics.disparate_impact.threshold || 0.8} - {(1/(metrics.disparate_impact.threshold || 0.8)).toFixed(2)}
|
||||||
|
{metrics.disparate_impact.fair
|
||||||
|
? " • This ratio indicates balanced treatment across groups."
|
||||||
|
: " • Ratio outside fair range suggests one group receives significantly different outcomes."}
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
|
||||||
|
{/* Statistical Parity */}
|
||||||
|
{metrics?.statistical_parity_difference?.value !== undefined && (
|
||||||
|
<div className="mb-3 p-3 bg-white rounded border border-slate-200">
|
||||||
|
<div className="flex items-center justify-between mb-2">
|
||||||
|
<div>
|
||||||
|
<div className="text-xs font-semibold text-slate-600">STATISTICAL PARITY DIFFERENCE</div>
|
||||||
|
<div className="text-2xl font-bold text-slate-900">
|
||||||
|
{metrics.statistical_parity_difference.value.toFixed(3)}
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
<div className={`px-3 py-1 rounded-full text-xs font-semibold ${
|
||||||
|
metrics.statistical_parity_difference.fair ? 'bg-green-100 text-green-800' : 'bg-red-100 text-red-800'
|
||||||
|
}`}>
|
||||||
|
{metrics.statistical_parity_difference.fair ? '✓ FAIR' : '✗ UNFAIR'}
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
<div className="text-xs text-slate-600 mb-2">{metrics.statistical_parity_difference.interpretation || 'Difference in positive rates'}</div>
|
||||||
|
<div className="text-xs text-slate-500 bg-blue-50 p-2 rounded">
|
||||||
|
<strong>Fair Threshold:</strong> ±{metrics.statistical_parity_difference.threshold || 0.1}
|
||||||
|
{metrics.statistical_parity_difference.fair
|
||||||
|
? " • Difference within acceptable range for equal treatment."
|
||||||
|
: " • Significant difference in positive outcome rates between groups."}
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
|
||||||
|
{/* Group Metrics */}
|
||||||
|
{metrics.group_metrics && (
|
||||||
|
<div className="p-3 bg-white rounded border border-slate-200">
|
||||||
|
<div className="text-xs font-semibold text-slate-600 mb-2">GROUP PERFORMANCE</div>
|
||||||
|
<div className="grid grid-cols-1 md:grid-cols-2 gap-2">
|
||||||
|
{Object.entries(metrics.group_metrics).map(([group, groupMetrics]: [string, any]) => (
|
||||||
|
<div key={group} className="p-2 bg-slate-50 rounded">
|
||||||
|
<div className="font-medium text-sm text-slate-800">{group}</div>
|
||||||
|
<div className="text-xs text-slate-600 mt-1">
|
||||||
|
<div>Positive Rate: <strong>{groupMetrics.positive_rate !== undefined ? (groupMetrics.positive_rate * 100).toFixed(1) : 'N/A'}%</strong></div>
|
||||||
|
<div>Sample Size: <strong>{groupMetrics.sample_size ?? 'N/A'}</strong></div>
|
||||||
|
{groupMetrics.tpr !== undefined && <div>True Positive Rate: <strong>{(groupMetrics.tpr * 100).toFixed(1)}%</strong></div>}
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
))}
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
))}
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
|
||||||
|
{/* Violations */}
|
||||||
|
{analyzeResult.bias_metrics.violations_detected.length > 0 && (
|
||||||
|
<div className="p-6 bg-red-50 rounded-xl border-2 border-red-200">
|
||||||
|
<h3 className="font-bold text-lg mb-4 flex items-center gap-2 text-red-800">
|
||||||
|
<span>⚠️</span>
|
||||||
|
Fairness Violations Detected
|
||||||
|
</h3>
|
||||||
|
<div className="space-y-3">
|
||||||
|
{analyzeResult.bias_metrics.violations_detected.map((violation: any, i: number) => (
|
||||||
|
<div key={i} className="p-4 bg-white rounded-lg border border-red-200">
|
||||||
|
<div className="flex items-start gap-3">
|
||||||
|
<span className={`px-2 py-1 rounded text-xs font-bold ${
|
||||||
|
violation.severity === 'HIGH' ? 'bg-red-600 text-white' :
|
||||||
|
violation.severity === 'MEDIUM' ? 'bg-orange-500 text-white' :
|
||||||
|
'bg-yellow-500 text-white'
|
||||||
|
}`}>
|
||||||
|
{violation.severity}
|
||||||
|
</span>
|
||||||
|
<div className="flex-1">
|
||||||
|
<div className="font-semibold text-slate-900">{violation.attribute}: {violation.metric}</div>
|
||||||
|
<div className="text-sm text-slate-700 mt-1">{violation.message}</div>
|
||||||
|
{violation.details && (
|
||||||
|
<div className="text-xs text-slate-500 mt-2 p-2 bg-slate-50 rounded">
|
||||||
|
{violation.details}
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
))}
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
|
||||||
|
{/* Key Insights */}
|
||||||
|
<div className="p-6 bg-gradient-to-br from-blue-50 to-cyan-50 rounded-xl border border-blue-200">
|
||||||
|
<h3 className="font-bold text-lg mb-3 flex items-center gap-2 text-blue-900">
|
||||||
|
<span>💡</span>
|
||||||
|
Key Insights
|
||||||
|
</h3>
|
||||||
|
<ul className="space-y-2 text-sm text-slate-700">
|
||||||
|
<li className="flex items-start gap-2">
|
||||||
|
<span className="text-blue-600 mt-0.5">•</span>
|
||||||
|
<span><strong>Bias Score {(analyzeResult.bias_metrics.overall_bias_score * 100).toFixed(1)}%</strong> indicates
|
||||||
|
{analyzeResult.bias_metrics.overall_bias_score < 0.3 ? ' strong fairness with minimal disparities across groups.'
|
||||||
|
: analyzeResult.bias_metrics.overall_bias_score < 0.5 ? ' moderate disparities that should be monitored and addressed.'
|
||||||
|
: ' significant unfairness requiring immediate remediation before deployment.'}</span>
|
||||||
|
</li>
|
||||||
|
<li className="flex items-start gap-2">
|
||||||
|
<span className="text-blue-600 mt-0.5">•</span>
|
||||||
|
<span><strong>Model achieves {(analyzeResult.model_performance.accuracy * 100).toFixed(1)}% accuracy</strong>,
|
||||||
|
but fairness metrics reveal how performance varies across demographic groups.</span>
|
||||||
|
</li>
|
||||||
|
{analyzeResult.bias_metrics.violations_detected.length > 0 ? (
|
||||||
|
<li className="flex items-start gap-2">
|
||||||
|
<span className="text-red-600 mt-0.5">•</span>
|
||||||
|
<span className="text-red-700"><strong>{analyzeResult.bias_metrics.violations_detected.length} violation(s)</strong> detected.
|
||||||
|
Review mitigation tab for recommended actions to improve fairness.</span>
|
||||||
|
</li>
|
||||||
|
) : (
|
||||||
|
<li className="flex items-start gap-2">
|
||||||
|
<span className="text-green-600 mt-0.5">•</span>
|
||||||
|
<span className="text-green-700"><strong>No violations detected.</strong> Model meets fairness thresholds across all protected attributes.</span>
|
||||||
|
</li>
|
||||||
|
)}
|
||||||
|
</ul>
|
||||||
|
</div>
|
||||||
</div>
|
</div>
|
||||||
) : (
|
) : (
|
||||||
<p className="text-sm text-slate-600">Upload and analyze a dataset to see bias metrics.</p>
|
<div className="text-center py-12">
|
||||||
|
<div className="text-6xl mb-4">📊</div>
|
||||||
|
<p className="text-slate-600 mb-2">No analysis results yet</p>
|
||||||
|
<p className="text-sm text-slate-500">Upload a dataset and click "Analyze" to see bias and fairness metrics</p>
|
||||||
|
</div>
|
||||||
)}
|
)}
|
||||||
</div>
|
</div>
|
||||||
);
|
);
|
||||||
|
|||||||
620
frontend/nordic-privacy-ai/components/try/CenterPanel.tsx
Normal file
620
frontend/nordic-privacy-ai/components/try/CenterPanel.tsx
Normal file
@@ -0,0 +1,620 @@
|
|||||||
|
"use client";
|
||||||
|
import { TryTab } from "./Sidebar";
|
||||||
|
import { useState, useRef, useCallback, useEffect } from "react";
|
||||||
|
import { saveLatestUpload, getLatestUpload, deleteLatestUpload } from "../../lib/indexeddb";
|
||||||
|
import { analyzeDataset, cleanDataset, getReportUrl, type AnalyzeResponse, type CleanResponse } from "../../lib/api";
|
||||||
|
|
||||||
|
interface CenterPanelProps {
|
||||||
|
tab: TryTab;
|
||||||
|
onAnalyze?: () => void;
|
||||||
|
}
|
||||||
|
|
||||||
|
interface UploadedFileMeta {
|
||||||
|
name: string;
|
||||||
|
size: number;
|
||||||
|
type: string;
|
||||||
|
contentPreview: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
interface TablePreviewData {
|
||||||
|
headers: string[];
|
||||||
|
rows: string[][];
|
||||||
|
origin: 'csv';
|
||||||
|
}
|
||||||
|
|
||||||
|
export function CenterPanel({ tab, onAnalyze }: CenterPanelProps) {
|
||||||
|
const PREVIEW_BYTES = 64 * 1024; // read first 64KB slice for large-file preview
|
||||||
|
const [fileMeta, setFileMeta] = useState<UploadedFileMeta | null>(null);
|
||||||
|
const [uploadedFile, setUploadedFile] = useState<File | null>(null);
|
||||||
|
const [isDragging, setIsDragging] = useState(false);
|
||||||
|
const [progress, setProgress] = useState<number>(0);
|
||||||
|
const [progressLabel, setProgressLabel] = useState<string>("Processing");
|
||||||
|
const [tablePreview, setTablePreview] = useState<TablePreviewData | null>(null);
|
||||||
|
const inputRef = useRef<HTMLInputElement | null>(null);
|
||||||
|
const [loadedFromCache, setLoadedFromCache] = useState(false);
|
||||||
|
const [isProcessing, setIsProcessing] = useState(false);
|
||||||
|
const [error, setError] = useState<string | null>(null);
|
||||||
|
|
||||||
|
// Analysis results
|
||||||
|
const [analyzeResult, setAnalyzeResult] = useState<AnalyzeResponse | null>(null);
|
||||||
|
const [cleanResult, setCleanResult] = useState<CleanResponse | null>(null);
|
||||||
|
|
||||||
|
const reset = () => {
|
||||||
|
setFileMeta(null);
|
||||||
|
setUploadedFile(null);
|
||||||
|
setProgress(0);
|
||||||
|
setProgressLabel("Processing");
|
||||||
|
setTablePreview(null);
|
||||||
|
setError(null);
|
||||||
|
};
|
||||||
|
|
||||||
|
// Handle API calls
|
||||||
|
const handleAnalyze = async () => {
|
||||||
|
if (!uploadedFile) {
|
||||||
|
setError("No file uploaded");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
setIsProcessing(true);
|
||||||
|
setError(null);
|
||||||
|
setProgressLabel("Analyzing dataset...");
|
||||||
|
|
||||||
|
try {
|
||||||
|
const result = await analyzeDataset(uploadedFile);
|
||||||
|
setAnalyzeResult(result);
|
||||||
|
setProgressLabel("Analysis complete!");
|
||||||
|
onAnalyze?.(); // Navigate to bias-analysis tab
|
||||||
|
} catch (err: any) {
|
||||||
|
setError(err.message || "Analysis failed");
|
||||||
|
} finally {
|
||||||
|
setIsProcessing(false);
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
const handleClean = async () => {
|
||||||
|
if (!uploadedFile) {
|
||||||
|
setError("No file uploaded");
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
setIsProcessing(true);
|
||||||
|
setError(null);
|
||||||
|
setProgressLabel("Cleaning dataset...");
|
||||||
|
|
||||||
|
try {
|
||||||
|
const result = await cleanDataset(uploadedFile);
|
||||||
|
setCleanResult(result);
|
||||||
|
setProgressLabel("Cleaning complete!");
|
||||||
|
} catch (err: any) {
|
||||||
|
setError(err.message || "Cleaning failed");
|
||||||
|
} finally {
|
||||||
|
setIsProcessing(false);
|
||||||
|
}
|
||||||
|
}; function tryParseCSV(text: string, maxRows = 50, maxCols = 40): TablePreviewData | null {
|
||||||
|
const lines = text.split(/\r?\n/).filter(l => l.trim().length > 0);
|
||||||
|
if (lines.length < 2) return null;
|
||||||
|
const commaDensity = lines.slice(0, 10).filter(l => l.includes(',')).length;
|
||||||
|
if (commaDensity < 2) return null;
|
||||||
|
const parseLine = (line: string) => {
|
||||||
|
const out: string[] = [];
|
||||||
|
let cur = '';
|
||||||
|
let inQuotes = false;
|
||||||
|
for (let i = 0; i < line.length; i++) {
|
||||||
|
const ch = line[i];
|
||||||
|
if (ch === '"') {
|
||||||
|
if (inQuotes && line[i + 1] === '"') { cur += '"'; i++; } else { inQuotes = !inQuotes; }
|
||||||
|
} else if (ch === ',' && !inQuotes) {
|
||||||
|
out.push(cur);
|
||||||
|
cur = '';
|
||||||
|
} else { cur += ch; }
|
||||||
|
}
|
||||||
|
out.push(cur);
|
||||||
|
return out.map(c => c.trim());
|
||||||
|
};
|
||||||
|
const raw = lines.slice(0, maxRows).map(parseLine);
|
||||||
|
if (raw.length === 0) return null;
|
||||||
|
const headers = raw[0];
|
||||||
|
const colCount = Math.min(headers.length, maxCols);
|
||||||
|
const rows = raw.slice(1).map(r => r.slice(0, colCount));
|
||||||
|
return { headers: headers.slice(0, colCount), rows, origin: 'csv' };
|
||||||
|
}
|
||||||
|
|
||||||
|
// We no longer build table preview for JSON; revert JSON to raw text view.
|
||||||
|
|
||||||
|
const processFile = useCallback(async (f: File) => {
|
||||||
|
if (!f) return;
|
||||||
|
const isCSV = /\.csv$/i.test(f.name);
|
||||||
|
setProgress(0);
|
||||||
|
setUploadedFile(f); // Save the file for API calls
|
||||||
|
|
||||||
|
// For large files, show a progress bar while reading the file stream (no preview)
|
||||||
|
if (f.size > 1024 * 1024) {
|
||||||
|
setProgressLabel("Uploading");
|
||||||
|
const metaObj: UploadedFileMeta = {
|
||||||
|
name: f.name,
|
||||||
|
size: f.size,
|
||||||
|
type: f.type || "unknown",
|
||||||
|
contentPreview: `Loading partial preview (first ${Math.round(PREVIEW_BYTES/1024)}KB)...`,
|
||||||
|
};
|
||||||
|
setFileMeta(metaObj);
|
||||||
|
setTablePreview(null);
|
||||||
|
// Save to IndexedDB immediately so it persists without needing full read
|
||||||
|
(async () => {
|
||||||
|
try { await saveLatestUpload(f, metaObj); } catch {}
|
||||||
|
})();
|
||||||
|
// Read head slice for partial preview & possible CSV table extraction
|
||||||
|
try {
|
||||||
|
const headBlob = f.slice(0, PREVIEW_BYTES);
|
||||||
|
const headReader = new FileReader();
|
||||||
|
headReader.onload = async () => {
|
||||||
|
try {
|
||||||
|
const buf = headReader.result as ArrayBuffer;
|
||||||
|
const decoder = new TextDecoder();
|
||||||
|
const text = decoder.decode(buf);
|
||||||
|
setFileMeta(prev => prev ? { ...prev, contentPreview: text.slice(0, 4000) } : prev);
|
||||||
|
if (isCSV) {
|
||||||
|
const parsed = tryParseCSV(text);
|
||||||
|
setTablePreview(parsed);
|
||||||
|
} else {
|
||||||
|
setTablePreview(null);
|
||||||
|
}
|
||||||
|
try { await saveLatestUpload(f, { ...metaObj, contentPreview: text.slice(0, 4000) }); } catch {}
|
||||||
|
} catch { /* ignore */ }
|
||||||
|
};
|
||||||
|
headReader.readAsArrayBuffer(headBlob);
|
||||||
|
} catch { /* ignore */ }
|
||||||
|
// Use streaming read for progress without buffering entire file in memory
|
||||||
|
try {
|
||||||
|
const stream: ReadableStream<Uint8Array> | undefined = (typeof (f as any).stream === "function" ? (f as any).stream() : undefined);
|
||||||
|
if (stream && typeof stream.getReader === "function") {
|
||||||
|
const reader = stream.getReader();
|
||||||
|
let loaded = 0;
|
||||||
|
const total = f.size || 1;
|
||||||
|
for (;;) {
|
||||||
|
const { done, value } = await reader.read();
|
||||||
|
if (done) break;
|
||||||
|
loaded += value ? value.length : 0;
|
||||||
|
const pct = Math.min(100, Math.round((loaded / total) * 100));
|
||||||
|
setProgress(pct);
|
||||||
|
}
|
||||||
|
setProgress(100);
|
||||||
|
} else {
|
||||||
|
// Fallback to FileReader progress events
|
||||||
|
const reader = new FileReader();
|
||||||
|
reader.onprogress = (evt) => {
|
||||||
|
if (evt.lengthComputable) {
|
||||||
|
const pct = Math.min(100, Math.round((evt.loaded / evt.total) * 100));
|
||||||
|
setProgress(pct);
|
||||||
|
} else {
|
||||||
|
setProgress((p) => (p < 90 ? p + 5 : p));
|
||||||
|
}
|
||||||
|
};
|
||||||
|
reader.onloadend = () => setProgress(100);
|
||||||
|
reader.onerror = () => setProgress(0);
|
||||||
|
reader.readAsArrayBuffer(f);
|
||||||
|
}
|
||||||
|
} catch {
|
||||||
|
setProgress(100);
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
const reader = new FileReader();
|
||||||
|
reader.onprogress = (evt) => {
|
||||||
|
if (evt.lengthComputable) {
|
||||||
|
const pct = Math.min(100, Math.round((evt.loaded / evt.total) * 100));
|
||||||
|
setProgress(pct);
|
||||||
|
} else {
|
||||||
|
setProgress((p) => (p < 90 ? p + 5 : p));
|
||||||
|
}
|
||||||
|
};
|
||||||
|
reader.onload = async () => {
|
||||||
|
try {
|
||||||
|
const buf = reader.result as ArrayBuffer;
|
||||||
|
const decoder = new TextDecoder();
|
||||||
|
const text = decoder.decode(buf);
|
||||||
|
const metaObj: UploadedFileMeta = {
|
||||||
|
name: f.name,
|
||||||
|
size: f.size,
|
||||||
|
type: f.type || "unknown",
|
||||||
|
contentPreview: text.slice(0, 4000),
|
||||||
|
};
|
||||||
|
setFileMeta(metaObj);
|
||||||
|
if (isCSV) {
|
||||||
|
const parsed = tryParseCSV(text);
|
||||||
|
setTablePreview(parsed);
|
||||||
|
} else {
|
||||||
|
setTablePreview(null);
|
||||||
|
}
|
||||||
|
// Save file blob and meta to browser cache (IndexedDB)
|
||||||
|
try {
|
||||||
|
await saveLatestUpload(f, metaObj);
|
||||||
|
} catch {}
|
||||||
|
setProgressLabel("Processing");
|
||||||
|
setProgress(100);
|
||||||
|
} catch (e) {
|
||||||
|
const metaObj: UploadedFileMeta = {
|
||||||
|
name: f.name,
|
||||||
|
size: f.size,
|
||||||
|
type: f.type || "unknown",
|
||||||
|
contentPreview: "Unable to decode preview.",
|
||||||
|
};
|
||||||
|
setFileMeta(metaObj);
|
||||||
|
setTablePreview(null);
|
||||||
|
try {
|
||||||
|
await saveLatestUpload(f, metaObj);
|
||||||
|
} catch {}
|
||||||
|
setProgressLabel("Processing");
|
||||||
|
setProgress(100);
|
||||||
|
}
|
||||||
|
};
|
||||||
|
reader.onerror = () => {
|
||||||
|
setProgress(0);
|
||||||
|
};
|
||||||
|
reader.readAsArrayBuffer(f);
|
||||||
|
}, []);
|
||||||
|
|
||||||
|
function handleFileChange(e: React.ChangeEvent<HTMLInputElement>) {
|
||||||
|
const f = e.target.files?.[0];
|
||||||
|
processFile(f as File);
|
||||||
|
}
|
||||||
|
|
||||||
|
const onDragOver = (e: React.DragEvent<HTMLDivElement>) => {
|
||||||
|
e.preventDefault();
|
||||||
|
setIsDragging(true);
|
||||||
|
};
|
||||||
|
const onDragLeave = () => setIsDragging(false);
|
||||||
|
const onDrop = (e: React.DragEvent<HTMLDivElement>) => {
|
||||||
|
e.preventDefault();
|
||||||
|
setIsDragging(false);
|
||||||
|
const f = e.dataTransfer.files?.[0];
|
||||||
|
processFile(f as File);
|
||||||
|
};
|
||||||
|
|
||||||
|
// Load last cached upload on mount (processing tab only)
|
||||||
|
useEffect(() => {
|
||||||
|
let ignore = false;
|
||||||
|
if (tab !== "processing") return;
|
||||||
|
(async () => {
|
||||||
|
try {
|
||||||
|
const { file, meta } = await getLatestUpload();
|
||||||
|
if (!ignore && meta) {
|
||||||
|
setFileMeta(meta as UploadedFileMeta);
|
||||||
|
if (file) {
|
||||||
|
setUploadedFile(file);
|
||||||
|
}
|
||||||
|
setLoadedFromCache(true);
|
||||||
|
}
|
||||||
|
} catch {}
|
||||||
|
})();
|
||||||
|
return () => {
|
||||||
|
ignore = true;
|
||||||
|
};
|
||||||
|
}, [tab]); function renderTabContent() {
|
||||||
|
switch (tab) {
|
||||||
|
case "processing":
|
||||||
|
return (
|
||||||
|
<div className="space-y-4 max-w-[1100px] xl:max-w-[1200px] w-full mx-auto">
|
||||||
|
<h2 className="text-xl font-semibold">Upload & Process Data</h2>
|
||||||
|
<p className="text-sm text-slate-600">Upload a CSV / JSON / text file. We will later parse, detect PII, and queue analyses.</p>
|
||||||
|
<div className="flex flex-col gap-3 min-w-0">
|
||||||
|
<div
|
||||||
|
onDragOver={onDragOver}
|
||||||
|
onDragLeave={onDragLeave}
|
||||||
|
onDrop={onDrop}
|
||||||
|
className={
|
||||||
|
"rounded-lg border-2 border-dashed p-6 text-center transition-colors " +
|
||||||
|
(isDragging ? "border-brand-600 bg-brand-50" : "border-slate-300 hover:border-brand-300")
|
||||||
|
}
|
||||||
|
>
|
||||||
|
<p className="text-sm text-slate-600">Drag & drop a CSV / JSON / TXT here, or click to browse.</p>
|
||||||
|
<div className="mt-3">
|
||||||
|
<button
|
||||||
|
type="button"
|
||||||
|
onClick={() => inputRef.current?.click()}
|
||||||
|
className="inline-flex items-center rounded-md bg-brand-600 px-4 py-2 text-white text-sm font-medium shadow hover:bg-brand-500"
|
||||||
|
>
|
||||||
|
Choose file
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
<input
|
||||||
|
ref={inputRef}
|
||||||
|
type="file"
|
||||||
|
accept=".csv,.json,.txt"
|
||||||
|
onChange={handleFileChange}
|
||||||
|
className="hidden"
|
||||||
|
aria-hidden
|
||||||
|
/>
|
||||||
|
{progress > 0 && (
|
||||||
|
<div className="w-full">
|
||||||
|
<div className="h-2 w-full rounded-full bg-slate-200 overflow-hidden">
|
||||||
|
<div
|
||||||
|
className="h-2 bg-brand-600 transition-all"
|
||||||
|
style={{ width: `${progress}%` }}
|
||||||
|
/>
|
||||||
|
</div>
|
||||||
|
<div className="mt-1 text-xs text-slate-500">{progressLabel} {progress}%</div>
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
{fileMeta && (
|
||||||
|
<div className="rounded-md border border-slate-200 p-4 bg-white shadow-sm">
|
||||||
|
<div className="flex items-center justify-between mb-2">
|
||||||
|
<div className="text-sm font-medium">{fileMeta.name}</div>
|
||||||
|
<div className="text-xs text-slate-500">{Math.round(fileMeta.size / 1024)} KB</div>
|
||||||
|
</div>
|
||||||
|
{loadedFromCache && (
|
||||||
|
<div className="mb-2 text-[11px] text-brand-700">Loaded from browser cache</div>
|
||||||
|
)}
|
||||||
|
<div className="mb-3 text-xs text-slate-500">{fileMeta.type || "Unknown type"}</div>
|
||||||
|
{/* Table preview when structured data detected; otherwise show text */}
|
||||||
|
{tablePreview && tablePreview.origin === 'csv' ? (
|
||||||
|
<div className="max-h-64 w-full min-w-0 overflow-x-auto overflow-y-auto rounded-md bg-slate-50">
|
||||||
|
<table className="min-w-full text-xs">
|
||||||
|
<thead className="sticky top-0 bg-slate-100">
|
||||||
|
<tr>
|
||||||
|
{tablePreview.headers.map((h, idx) => (
|
||||||
|
<th key={idx} className="text-left font-semibold px-3 py-2 border-b border-slate-200 whitespace-nowrap">{h}</th>
|
||||||
|
))}
|
||||||
|
</tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
{tablePreview.rows.map((r, i) => (
|
||||||
|
<tr key={i} className={i % 2 === 0 ? "" : "bg-slate-100/50"}>
|
||||||
|
{r.map((c, j) => (
|
||||||
|
<td key={j} className="px-3 py-1.5 border-b border-slate-100 whitespace-nowrap max-w-[24ch] overflow-hidden text-ellipsis">{c}</td>
|
||||||
|
))}
|
||||||
|
</tr>
|
||||||
|
))}
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
</div>
|
||||||
|
) : (
|
||||||
|
<pre className="max-h-64 overflow-auto text-xs bg-slate-50 p-3 rounded-md whitespace-pre-wrap leading-relaxed">
|
||||||
|
{fileMeta.contentPreview || "(no preview)"}
|
||||||
|
</pre>
|
||||||
|
)}
|
||||||
|
|
||||||
|
{error && (
|
||||||
|
<div className="mt-3 p-3 bg-red-50 border border-red-200 rounded-md text-sm text-red-700">
|
||||||
|
❌ {error}
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
|
||||||
|
{analyzeResult && (
|
||||||
|
<div className="mt-3 p-3 bg-green-50 border border-green-200 rounded-md text-sm text-green-700">
|
||||||
|
✅ Analysis complete! View results in tabs.
|
||||||
|
<a
|
||||||
|
href={getReportUrl(analyzeResult.report_file)}
|
||||||
|
target="_blank"
|
||||||
|
rel="noopener noreferrer"
|
||||||
|
className="ml-2 underline"
|
||||||
|
>
|
||||||
|
Download Report
|
||||||
|
</a>
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
|
||||||
|
{cleanResult && (
|
||||||
|
<div className="mt-3 p-3 bg-green-50 border border-green-200 rounded-md text-sm text-green-700">
|
||||||
|
✅ Cleaning complete! {cleanResult.summary.total_cells_affected} cells anonymized.
|
||||||
|
<div className="mt-2 flex gap-2">
|
||||||
|
<a
|
||||||
|
href={getReportUrl(cleanResult.files.cleaned_csv)}
|
||||||
|
download
|
||||||
|
className="underline"
|
||||||
|
>
|
||||||
|
Download Cleaned CSV
|
||||||
|
</a>
|
||||||
|
<a
|
||||||
|
href={getReportUrl(cleanResult.files.audit_report)}
|
||||||
|
target="_blank"
|
||||||
|
rel="noopener noreferrer"
|
||||||
|
className="underline"
|
||||||
|
>
|
||||||
|
View Audit Report
|
||||||
|
</a>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
|
||||||
|
<div className="mt-3 flex justify-end gap-2">
|
||||||
|
<button
|
||||||
|
type="button"
|
||||||
|
onClick={async () => {
|
||||||
|
reset();
|
||||||
|
try { await deleteLatestUpload(); } catch {}
|
||||||
|
setLoadedFromCache(false);
|
||||||
|
setAnalyzeResult(null);
|
||||||
|
setCleanResult(null);
|
||||||
|
}}
|
||||||
|
className="text-xs rounded-md border px-3 py-1.5 hover:bg-slate-50"
|
||||||
|
>
|
||||||
|
Clear
|
||||||
|
</button>
|
||||||
|
<button
|
||||||
|
type="button"
|
||||||
|
onClick={handleClean}
|
||||||
|
disabled={isProcessing}
|
||||||
|
className="text-xs rounded-md bg-green-600 text-white px-3 py-1.5 hover:bg-green-500 disabled:opacity-50 disabled:cursor-not-allowed"
|
||||||
|
>
|
||||||
|
{isProcessing ? "Processing..." : "Clean (PII)"}
|
||||||
|
</button>
|
||||||
|
<button
|
||||||
|
type="button"
|
||||||
|
onClick={handleAnalyze}
|
||||||
|
disabled={isProcessing}
|
||||||
|
className="text-xs rounded-md bg-brand-600 text-white px-3 py-1.5 hover:bg-brand-500 disabled:opacity-50 disabled:cursor-not-allowed"
|
||||||
|
>
|
||||||
|
{isProcessing ? "Processing..." : "Analyze"}
|
||||||
|
</button>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
case "bias-analysis":
|
||||||
|
return (
|
||||||
|
<div className="space-y-4">
|
||||||
|
<h2 className="text-xl font-semibold">Bias Analysis</h2>
|
||||||
|
{analyzeResult ? (
|
||||||
|
<div className="space-y-4">
|
||||||
|
<div className="grid grid-cols-2 gap-4">
|
||||||
|
<div className="p-4 bg-white rounded-lg border">
|
||||||
|
<div className="text-sm text-slate-600">Overall Bias Score</div>
|
||||||
|
<div className="text-2xl font-bold">{(analyzeResult.bias_metrics.overall_bias_score * 100).toFixed(1)}%</div>
|
||||||
|
</div>
|
||||||
|
<div className="p-4 bg-white rounded-lg border">
|
||||||
|
<div className="text-sm text-slate-600">Violations Detected</div>
|
||||||
|
<div className="text-2xl font-bold">{analyzeResult.bias_metrics.violations_detected.length}</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div className="p-4 bg-white rounded-lg border">
|
||||||
|
<h3 className="font-semibold mb-2">Model Performance</h3>
|
||||||
|
<div className="grid grid-cols-4 gap-2 text-sm">
|
||||||
|
<div>
|
||||||
|
<div className="text-slate-600">Accuracy</div>
|
||||||
|
<div className="font-medium">{(analyzeResult.model_performance.accuracy * 100).toFixed(1)}%</div>
|
||||||
|
</div>
|
||||||
|
<div>
|
||||||
|
<div className="text-slate-600">Precision</div>
|
||||||
|
<div className="font-medium">{(analyzeResult.model_performance.precision * 100).toFixed(1)}%</div>
|
||||||
|
</div>
|
||||||
|
<div>
|
||||||
|
<div className="text-slate-600">Recall</div>
|
||||||
|
<div className="font-medium">{(analyzeResult.model_performance.recall * 100).toFixed(1)}%</div>
|
||||||
|
</div>
|
||||||
|
<div>
|
||||||
|
<div className="text-slate-600">F1 Score</div>
|
||||||
|
<div className="font-medium">{(analyzeResult.model_performance.f1_score * 100).toFixed(1)}%</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
) : (
|
||||||
|
<p className="text-sm text-slate-600">Upload and analyze a dataset to see bias metrics.</p>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
case "risk-analysis":
|
||||||
|
return (
|
||||||
|
<div className="space-y-4">
|
||||||
|
<h2 className="text-xl font-semibold">Risk Analysis</h2>
|
||||||
|
{analyzeResult ? (
|
||||||
|
<div className="space-y-4">
|
||||||
|
<div className="p-4 bg-white rounded-lg border">
|
||||||
|
<div className="text-sm text-slate-600">Overall Risk Score</div>
|
||||||
|
<div className="text-2xl font-bold">{(analyzeResult.risk_assessment.overall_risk_score * 100).toFixed(1)}%</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
{cleanResult && (
|
||||||
|
<div className="p-4 bg-white rounded-lg border">
|
||||||
|
<h3 className="font-semibold mb-2">PII Detection Results</h3>
|
||||||
|
<div className="text-sm space-y-1">
|
||||||
|
<div>Cells Anonymized: <span className="font-medium">{cleanResult.summary.total_cells_affected}</span></div>
|
||||||
|
<div>Columns Removed: <span className="font-medium">{cleanResult.summary.columns_removed.length}</span></div>
|
||||||
|
<div>Columns Anonymized: <span className="font-medium">{cleanResult.summary.columns_anonymized.length}</span></div>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
) : (
|
||||||
|
<p className="text-sm text-slate-600">Upload and analyze a dataset to see risk assessment.</p>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
case "bias-risk-mitigation":
|
||||||
|
return (
|
||||||
|
<div className="space-y-4">
|
||||||
|
<h2 className="text-xl font-semibold">Mitigation Suggestions</h2>
|
||||||
|
{analyzeResult && analyzeResult.recommendations.length > 0 ? (
|
||||||
|
<div className="space-y-2">
|
||||||
|
{analyzeResult.recommendations.map((rec, i) => (
|
||||||
|
<div key={i} className="p-3 bg-blue-50 border border-blue-200 rounded-md text-sm">
|
||||||
|
{rec}
|
||||||
|
</div>
|
||||||
|
))}
|
||||||
|
</div>
|
||||||
|
) : (
|
||||||
|
<p className="text-sm text-slate-600">
|
||||||
|
Recommendations will appear here after analysis.
|
||||||
|
</p>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
case "results":
|
||||||
|
return (
|
||||||
|
<div className="space-y-4">
|
||||||
|
<h2 className="text-xl font-semibold">Results Summary</h2>
|
||||||
|
{(analyzeResult || cleanResult) ? (
|
||||||
|
<div className="space-y-4">
|
||||||
|
{analyzeResult && (
|
||||||
|
<div className="p-4 bg-white rounded-lg border">
|
||||||
|
<h3 className="font-semibold mb-2">Analysis Results</h3>
|
||||||
|
<div className="text-sm space-y-1">
|
||||||
|
<div>Dataset: {analyzeResult.filename}</div>
|
||||||
|
<div>Rows: {analyzeResult.dataset_info.rows}</div>
|
||||||
|
<div>Columns: {analyzeResult.dataset_info.columns}</div>
|
||||||
|
<div>Bias Score: {(analyzeResult.bias_metrics.overall_bias_score * 100).toFixed(1)}%</div>
|
||||||
|
<div>Risk Score: {(analyzeResult.risk_assessment.overall_risk_score * 100).toFixed(1)}%</div>
|
||||||
|
</div>
|
||||||
|
<a
|
||||||
|
href={getReportUrl(analyzeResult.report_file)}
|
||||||
|
target="_blank"
|
||||||
|
rel="noopener noreferrer"
|
||||||
|
className="mt-3 inline-block text-sm text-brand-600 underline"
|
||||||
|
>
|
||||||
|
Download Full Report →
|
||||||
|
</a>
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
|
||||||
|
{cleanResult && (
|
||||||
|
<div className="p-4 bg-white rounded-lg border">
|
||||||
|
<h3 className="font-semibold mb-2">Cleaning Results</h3>
|
||||||
|
<div className="text-sm space-y-1">
|
||||||
|
<div>Original: {cleanResult.dataset_info.original_rows} rows × {cleanResult.dataset_info.original_columns} cols</div>
|
||||||
|
<div>Cleaned: {cleanResult.dataset_info.cleaned_rows} rows × {cleanResult.dataset_info.cleaned_columns} cols</div>
|
||||||
|
<div>Cells Anonymized: {cleanResult.summary.total_cells_affected}</div>
|
||||||
|
<div>Columns Removed: {cleanResult.summary.columns_removed.length}</div>
|
||||||
|
<div>GDPR Compliant: {cleanResult.gdpr_compliance.length} articles applied</div>
|
||||||
|
</div>
|
||||||
|
<div className="mt-3 flex gap-2">
|
||||||
|
<a
|
||||||
|
href={getReportUrl(cleanResult.files.cleaned_csv)}
|
||||||
|
download
|
||||||
|
className="text-sm text-brand-600 underline"
|
||||||
|
>
|
||||||
|
Download Cleaned CSV →
|
||||||
|
</a>
|
||||||
|
<a
|
||||||
|
href={getReportUrl(cleanResult.files.audit_report)}
|
||||||
|
target="_blank"
|
||||||
|
rel="noopener noreferrer"
|
||||||
|
className="text-sm text-brand-600 underline"
|
||||||
|
>
|
||||||
|
View Audit Report →
|
||||||
|
</a>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
) : (
|
||||||
|
<p className="text-sm text-slate-600">
|
||||||
|
Process a dataset to see aggregated results.
|
||||||
|
</p>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
default:
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return (
|
||||||
|
<div className="h-full overflow-y-auto p-6 bg-white/60">
|
||||||
|
{renderTabContent()}
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user