# Magika ML Model Implementation

This document describes how Google's Magika content type detection model was integrated into Copy Haste.

## Overview

Copy Haste uses Google's [Magika](https://github.com/google/magika) model for content type detection. Magika is a deep learning model that identifies 214 different content types from file/text content with high accuracy.

The model was converted from ONNX to CoreML format for fast inference using Apple's Neural Engine.

## Model Architecture (MagikaV2)

```
Input: 2048 bytes (1024 from beginning + 1024 from ending)
    ↓
┌──────────────────────────────────────────────────────┐
│  1. EMBEDDING LOOKUP                                 │
│     - Input: 2048 int values (0-256, 256=padding)    │
│     - One-hot encode to (2048, 257)                  │
│     - MatMul with embedding (257, 64)                │
│     - Output: (2048, 64)                             │
├──────────────────────────────────────────────────────┤
│  2. DENSE0 (bias only)                               │
│     - Add bias (64,)                                 │
│     - Output: (2048, 64)                             │
├──────────────────────────────────────────────────────┤
│  3. GELU ACTIVATION                                  │
│     - x * 0.5 * (1 + tanh(√(2/π) * (x + 0.044715x³)))│
├──────────────────────────────────────────────────────┤
│  4. RESHAPE                                          │
│     - (2048, 64) → (512, 256)                        │
│     - Groups every 4 positions (4 × 64 = 256)        │
├──────────────────────────────────────────────────────┤
│  5. LAYERNORM0                                       │
│     - Normalize over SEQUENCE dim (512 positions)    │
│     - For each of 256 features, compute mean/var     │
│       across all 512 positions                       │
│     - Scale: (512,), Bias: (512,) - per-position     │
│     - These broadcast over the 256 features          │
├──────────────────────────────────────────────────────┤
│  6. CONV1D                                           │
│     - Kernel: (512, 256, 5, 1)                       │
│     - 512 output channels, kernel size 5             │
│     - Valid padding → output length: 508             │
│     - Add bias (512,)                                │
├──────────────────────────────────────────────────────┤
│  7. GELU ACTIVATION                                  │
├──────────────────────────────────────────────────────┤
│  8. GLOBAL MAX POOL                                  │
│     - Max over sequence dimension                    │
│     - Output: (512,)                                 │
├──────────────────────────────────────────────────────┤
│  9. LAYERNORM1                                       │
│     - Normalize over 512 features                    │
│     - Scale: (512,), Bias: (512,)                    │
├──────────────────────────────────────────────────────┤
│  10. DENSE1 (output layer)                           │
│      - Weights: (512, 214)                           │
│      - Bias: (214,)                                  │
│      - Output: (214,) logits                         │
├──────────────────────────────────────────────────────┤
│  11. SOFTMAX                                         │
│      - Output: (214,) probabilities                  │
└──────────────────────────────────────────────────────┘
```

## Preprocessing Details

### Input Format

Magika expects 2048 bytes of content:
- **First 1024 bytes**: From the beginning of the content
- **Last 1024 bytes**: From the end of the content

For content shorter than 2048 bytes, the remaining positions are filled with the **padding token (256)**.

```swift
// Preprocessing logic
if content.count <= 2048 {
    // Content fits - pad the rest with 256
    input[0..<content.count] = content
    input[content.count..<2048] = [256, 256, ...]
} else {
    // Content too long - take beginning and ending
    input[0..<1024] = content[0..<1024]           // First 1024 bytes
    input[1024..<2048] = content[end-1024..<end]  // Last 1024 bytes
}
```

### Byte Values

Input values are integers 0-256:
- **0-255**: Actual byte values from UTF-8 encoding
- **256**: Padding token for short content

## Files in This Directory

### magika/

**Current version: Magika v3.3 (standard_v3_3)**

- `model.onnx` - Original ONNX model from Google (3.16 MB, for reference only)
- `config.min.json` - Model configuration with labels, thresholds, and settings
- `magika_weights.bin` - Extracted weights in custom binary format (~3 MB, used during development)
- `magika_labels.txt` - 214 content type labels (one per line)
- `Magika.mlpackage/` - CoreML model converted from ONNX (~3 MB)
- `convert_manual_pytorch.py` - Script to recreate PyTorch model and convert to CoreML
- `venv311/` - Python 3.11 virtual environment for conversion (not committed)

### Weight File Format (magika_weights.bin)

```
Header:
  - Magic: "MGKA" (4 bytes)
  - Version: 1 (uint32, little-endian)

Tensors (in order):
  For each tensor:
    - Name length (uint32)
    - Name (UTF-8 bytes)
    - Number of dimensions (uint32)
    - Shape (uint32 × ndim)
    - Data (float32 × product(shape))

Tensor order:
  1. embedding      (257, 64)      - 65,792 bytes
  2. dense0_bias    (64,)          - 256 bytes
  3. conv_kernel    (512, 256, 5, 1) - 2,621,440 bytes
  4. conv_bias      (512,)         - 2,048 bytes
  5. layernorm0_scale (512,)       - 2,048 bytes
  6. layernorm0_bias  (512,)       - 2,048 bytes
  7. layernorm1_scale (512,)       - 2,048 bytes
  8. layernorm1_bias  (512,)       - 2,048 bytes
  9. dense1_weights (512, 214)     - 438,272 bytes
  10. dense1_bias   (214,)         - 856 bytes

Total: ~3.0 MB
```

## Swift Implementation

### Services/MagikaDetectionCoreML.swift (Primary)

CoreML-based implementation for fast inference:

```swift
// Usage
let result = await MagikaDetectionCoreML.shared.detect(text: "def hello(): print('world')")
print(result?.label)      // "python"
print(result?.confidence) // ~0.95
print(result?.isCode)     // true

// Get top N predictions
let topResults = await MagikaDetectionCoreML.shared.detectTopN(text: code, n: 5)
```

## Content Types (214 Labels)

### Code-Related Labels (51)

```
asm, batch, c, clojure, coffeescript, cpp, cs, css, dart, dockerfile,
elixir, erlang, fortran, go, groovy, haskell, html, java, javascript,
json, julia, kotlin, lisp, lua, markdown, matlab, objectivec, ocaml,
pascal, perl, php, powershell, prolog, python, r, ruby, rust, scala,
scss, shell, solidity, sql, swift, tcl, typescript, vba, verilog,
vhdl, xml, yaml, zig
```

### Other Labels

Includes: text files (txt, csv, tsv), documents (pdf, doc, docx), images (png, jpeg, gif), archives (zip, tar, gzip), and many more.

## Configuration (config.min.json)

Key settings from the model config:

```json
{
  "beg_size": 1024,           // Bytes from beginning
  "end_size": 1024,           // Bytes from ending
  "padding_token": 256,       // Value for padding
  "medium_confidence_threshold": 0.5,
  "min_file_size_for_dl": 8,  // Minimum bytes for detection

  "thresholds": {             // Per-label confidence overrides
    "markdown": 0.75,
    "sql": 0.9,
    "latex": 0.95
    // ... more
  }
}
```

## How to Update the Model (Future Versions)

When Google releases a new version of Magika, follow these steps to update the CoreML model.

### Prerequisites

- Python 3.11 (required for compatibility with all packages)
- The new ONNX model from Google

### Step 1: Download New Model

```bash
cd ML/magika

# Check https://github.com/google/magika/releases for the latest version
# Update the version in the URL (e.g., standard_v3_4)
curl -L -o model.onnx "https://github.com/google/magika/raw/main/python/magika/models/standard_v3_3/model.onnx"
curl -L -o config.min.json "https://github.com/google/magika/raw/main/python/magika/models/standard_v3_3/config.min.json"
```

### Step 2: Set Up Python Environment

```bash
cd ML/magika

# Create fresh venv with Python 3.11 (MUST be 3.11 for package compatibility)
rm -rf venv311
python3.11 -m venv venv311
source venv311/bin/activate

# Install dependencies (specific versions matter!)
pip install numpy==1.26.4 onnx onnxruntime torch coremltools==7.2
```

**Why these versions?**
- `numpy<2` - coremltools 7.2 uses deprecated numpy APIs removed in numpy 2.0
- `coremltools==7.2` - newer versions dropped ONNX support, older versions have bugs
- `torch` - any recent version works, but don't use with coremltools 8+

### Step 3: Extract Labels

```bash
# Extract labels from config
python3 -c "
import json
with open('config.min.json') as f:
    config = json.load(f)
with open('magika_labels.txt', 'w') as f:
    f.write('\n'.join(config['target_labels_space']))
print(f'Wrote {len(config[\"target_labels_space\"])} labels')
"
```

### Step 4: Verify Model Architecture Hasn't Changed

Before converting, check that the model architecture matches what we expect:

```bash
python3 -c "
import onnx
from onnx import numpy_helper

model = onnx.load('model.onnx')
weights = {init.name: numpy_helper.to_array(init) for init in model.graph.initializer}

# These are the weights we need - verify they exist and have expected shapes
expected = {
    'embedding': (257, 64),
    'dense0_bias': (1, 1, 64),
    'conv_kernel': (512, 256, 5, 1),
    'conv_bias': (1, 512, 1),
    'ln0_scale': (1, 512, 1),
    'ln0_bias': (1, 512, 1),
    'ln1_scale': (1, 512),
    'ln1_bias': (1, 512),
    'dense1_weights': (512, 214),
    'dense1_bias': (1, 214),
}

# Find weights by pattern matching (names may vary slightly between versions)
for name in weights:
    print(f'{name}: {weights[name].shape}')
"
```

**If shapes match**: Proceed with conversion.
**If shapes differ**: The model architecture changed - you'll need to update `convert_manual_pytorch.py` to match.

### Step 5: Run Conversion

```bash
source venv311/bin/activate
python convert_manual_pytorch.py
```

**Expected output:**
```
PyTorch matches ONNX: True
Converting PyTorch to CoreML...
Conversion successful!
CoreML matches ONNX: True
Saved to Magika.mlpackage
SUCCESS!
```

**If PyTorch doesn't match ONNX**: The model architecture changed. You need to:
1. Analyze the new ONNX graph structure
2. Update the `MagikaModel` class in `convert_manual_pytorch.py`
3. Pay special attention to LayerNorm axis (this was a tricky bug!)

### Step 6: Update Xcode Project

1. **Remove old model**: Delete existing `Magika.mlmodelc` from Xcode
2. **Add new model**: Drag `Magika.mlpackage` into Xcode project
3. **Update labels**: Copy `magika_labels.txt` to project if label count changed
4. **Update constants**: If label count changed, update `numClasses` in Swift code

### Step 7: Test

Build and run the app, then copy some code to clipboard:

```
[Timing] ML detection: 6-10ms  ← Should be fast
```

Test with various code snippets:
- Python: `def hello(): print("world")`
- JavaScript: `const x = () => console.log("hi")`
- Shell: `#!/bin/bash\necho hello`

### Troubleshooting

**"Cannot find 'MagikaDetectionCoreML' in scope"**
- Add `MagikaDetectionCoreML.swift` to Xcode project

**"Could not find Magika.mlmodelc in bundle"**
- Ensure `Magika.mlpackage` is in Build Phases → Copy Bundle Resources

**CoreML prediction returns nil**
- Check that input name matches: should be "bytes"
- Check that output name matches: should be "probabilities"
- Run `convert_manual_pytorch.py` again and check the printed input/output names

**Wrong predictions (e.g., detecting "wav" for code)**
- This was caused by LayerNorm normalizing over wrong axis
- LayerNorm0 MUST normalize over sequence dim (512), not feature dim (256)
- Check that `layer_norm_over_seq` computes mean/var over `dim=1`

## Performance Notes

### CoreML Implementation (Current)
- **Model loading**: ~75ms (one-time on first detection)
- **Inference**: ~6-10ms per text snippet
- **Memory**: More efficient via CoreML runtime
- **App size impact**: ~3MB for .mlmodelc

### Why CoreML is 10x Faster for Magika

The original pure Swift implementation using Accelerate took ~70ms per inference. The bottleneck was Magika's **Conv1D layer**:

```
Conv1D: 512 output channels × 256 input channels × 5 kernel size × 512 positions
      = ~335 million multiply-accumulate operations
```

Even with Accelerate, convolutions implemented as nested loops are expensive. CoreML can dispatch this to the **Neural Engine**, which is purpose-built for convolutions and achieves 10-100x speedup for this operation.

### Why Guesslang Doesn't Need CoreML

Guesslang uses a simpler architecture with **no convolutions**—just dense (fully-connected) layers:

| Model | Architecture | Accelerate Time |
|-------|--------------|-----------------|
| Magika | Conv1D + LayerNorm + Dense | ~70ms (before CoreML) |
| Guesslang | Dense → Dense → Dense | ~5ms |

For pure matrix multiplications, `cblas_sgemm` from Accelerate is already near-optimal. Converting Guesslang to CoreML would add overhead (~2-5ms for dispatch/scheduling) without meaningful improvement.

## Debugging

Enable debug output by checking Xcode console for:
```
MagikaDetection: Model loaded successfully
  - Labels: 214
  - Embedding: 16448 floats
  - Conv kernel: 655360 floats
  - Dense weights: 109568 floats
```

If model fails to load, check:
1. Files are in app bundle (Build Phases → Copy Bundle Resources)
2. File names match exactly: `magika_weights.bin`, `magika_labels.txt`
3. Binary file starts with "MGKA" magic bytes

## Copy Haste Preprocessing (ClipboardMonitor)

Before text is sent to Magika for detection, Copy Haste applies preprocessing in `ClipboardMonitor.swift`:

### 1. Line Ending Normalization

Different apps put different line endings on the clipboard:
- **Xcode**: Uses CRLF (`\r\n`) - Windows-style
- **Browsers/VS Code**: Use LF (`\n`) - Unix-style

We normalize all line endings to LF to ensure consistent detection results:

```swift
let normalizedString = string
    .replacingOccurrences(of: "\r\n", with: "\n")
    .replacingOccurrences(of: "\r", with: "\n")
```

### 2. Trailing Whitespace Removal for Detection

Clipboard content often includes trailing newlines, which significantly impact detection accuracy. For example, a single-line JavaScript function:
- Without trailing `\n`: **72% JavaScript**
- With trailing `\n`: **31% txt**

However, **leading whitespace (indentation) helps detection**:
- With 8 leading spaces: **59% Python**
- Without leading spaces: **45% txt**

This happens because Magika's preprocessing only applies lstrip to the beginning chunk and rstrip to the ending chunk. For short content that fits entirely in one chunk, trailing whitespace remains and affects the model's prediction. But indentation provides useful signal about code structure.

We strip only **trailing** whitespace for detection, preserving leading indentation:

```swift
var textForDetection = normalizedString
while textForDetection.hasSuffix(" ") || textForDetection.hasSuffix("\t") ||
      textForDetection.hasSuffix("\n") || textForDetection.hasSuffix("\r") {
    textForDetection.removeLast()
}
let result = await CodeDetectionService.shared.detect(text: textForDetection)
```

### 3. Flow Summary

```
Clipboard → normalize CRLF → store normalizedString (preserves all whitespace)
                          ↓
                    rstrip only → send to Magika (keeps indentation)
```

| Step | Text Used | Purpose |
|------|-----------|---------|
| Storage | `normalizedString` | Preserve all whitespace for pasting |
| Duplicate detection | `normalizedString` | Exact match comparison |
| ML detection | `rstripped` | Better detection (keeps indentation, removes trailing) |

## Key Differences from Guesslang/VS Code Model

| Aspect | Magika | Guesslang |
|--------|--------|-----------|
| Content types | 214 | 54 |
| Input | Raw bytes | FarmHash bigrams |
| Architecture | CNN + Dense | DNN + Linear |
| Preprocessing | Simple byte extraction | Complex hashing |
| Short text handling | Better (byte features) | Worse (needs repetition) |
| Model size | ~3MB | ~2.7MB |

## CoreML Conversion

The Magika ONNX model was converted to CoreML using the following process. This is documented for future reference when updating the model.

### Why Not Direct ONNX Conversion?

Direct ONNX-to-CoreML conversion failed due to:
1. **coremltools 8.0+** removed direct ONNX support
2. **coremltools 5.x-7.x** couldn't handle ops like `Shape`, `Expand`, `Equal`, and `Unsqueeze` with dynamic axes
3. **onnx2torch** doesn't support `GlobalMaxPool`

### Successful Conversion: PyTorch Recreation

The solution was to manually recreate the model architecture in PyTorch, load weights from ONNX, and convert to CoreML.

#### Step 1: Set Up Environment

```bash
cd ML/magika
python3.11 -m venv venv311
source venv311/bin/activate
pip install numpy==1.26.4 onnx onnxruntime torch coremltools==7.2
```

#### Step 2: PyTorch Model Recreation

Key considerations:

1. **Custom LayerNorm**: CoreML's PyTorch converter doesn't support `torch.var()`, so variance must be computed manually:
   ```python
   # Instead of: var = x.var(dim=1, keepdim=True)
   x_centered = x - mean
   var = (x_centered * x_centered).mean(dim=1, keepdim=True)
   ```

2. **LayerNorm0 axis**: Normalizes over **sequence dimension (512)** for each feature, not over features. The scale/bias are (512,) and broadcast over the 256 features.

3. **Embedding dimension**: The embedding outputs 64 features (not 256). The reshape combines 4 positions: `(2048, 64) → (512, 4, 64) → (512, 256)`.

#### Step 3: Run Conversion

```bash
python convert_manual_pytorch.py
```

This script:
1. Loads weights from `model.onnx`
2. Creates a PyTorch model matching the architecture
3. Verifies PyTorch output matches ONNX output
4. Traces and converts to CoreML
5. Verifies CoreML output matches ONNX output
6. Saves `Magika.mlpackage`

#### Step 4: Add to Xcode Project

1. Drag `Magika.mlpackage` into Xcode project
2. Xcode compiles it to `Magika.mlmodelc` in the bundle
3. Ensure `magika_labels.txt` is also in the bundle

### Conversion Script Location

- `ML/magika/convert_manual_pytorch.py` - Main conversion script
- `ML/magika/Magika.mlpackage/` - Generated CoreML model

### Verifying Conversion

The conversion script automatically verifies that:
- PyTorch output matches ONNX output (atol=1e-3)
- CoreML output matches ONNX output (atol=1e-3)

Test with:
```python
test_text = 'print("Hello, World!")'
# Should detect as: lua ~0.73, autoit ~0.09, julia ~0.05, python ~0.03
```

## References

- [Magika GitHub](https://github.com/google/magika)
- [Magika Blog Post](https://opensource.googleblog.com/2024/02/magika-ai-powered-fast-and-efficient-file-type-identification.html)
- [ONNX Format](https://onnx.ai/)
- [Apple Accelerate](https://developer.apple.com/documentation/accelerate)
- [CoreML Tools](https://coremltools.readme.io/)
