docs(autoresearch): add example research project and README

Add LoRA rank convergence study example demonstrating:
- Bootstrap phase with literature search
- Hypothesis formation and testing
- Inner/outer loop workflow
- Progress tracking and findings synthesis
This commit is contained in:
Howard Li 2026-04-11 12:28:50 -07:00
parent 762b582704
commit c77175b7f7
2 changed files with 108 additions and 0 deletions

View file

@ -0,0 +1,34 @@
# Autoresearch Examples
This directory contains example research projects using the autoresearch methodology.
## Available Examples
### `lora-rank-study.md`
**Question:** Does LoRA rank affect convergence speed on small datasets?
**Type:** Benchmark optimization, hyperparameter study
**Skills Used:**
- `arxiv` — Literature search
- `mlops` — Model training
- `tensorboard` — Experiment tracking
**Key Takeaway:** Higher rank improves convergence speed up to a point (r=16), then diminishing returns.
---
## Creating Your Own Research
1. Start with `/autoresearch "your question"`
2. Follow the two-loop architecture
3. Commit protocols before running
4. Generate progress reports with `/research-report`
## Tips from Examples
- **Start small:** First experiment should complete in <30 minutes
- **Define metrics upfront:** Know what you're measuring before you start
- **Document surprises:** Negative results are progress too
- **Show your work:** Progress reports help humans follow along

View file

@ -0,0 +1,74 @@
# LoRA Rank Convergence Study
**Research Question:** Does LoRA rank affect convergence speed on small datasets?
## Bootstrap
### Literature
Key papers:
- Hu et al. (2021) — LoRA: Low-Rank Adaptation of Large Language Models
- Valipour et al. (2023) — DyLoRA: Parameter-Efficient Tuning with Dynamic Search
Gap: Most papers focus on final performance, not convergence dynamics.
### Hypotheses
- **H1:** Higher rank (r=16) converges faster but may overfit on small data
- **H2:** Lower rank (r=4) converges slower but generalizes better
- **H3:** There's an optimal rank (r=8) that balances speed and generalization
## Experiments
### H001 — Baseline (r=8)
```bash
# Protocol: Train with rank 8, measure convergence steps to 90% of max accuracy
# Prediction: Baseline behavior, ~50 steps to converge
```
**Results:**
- Convergence steps: 47
- Final accuracy: 0.892
- Wall time: 12 min
### H002 — Low Rank (r=4)
**Results:**
- Convergence steps: 68 (+44% vs baseline)
- Final accuracy: 0.887 (-0.6%)
### H003 — High Rank (r=16)
**Results:**
- Convergence steps: 41 (-13% vs baseline)
- Final accuracy: 0.894 (+0.2%)
## Outer Loop #1
**Pattern:** Higher rank → faster convergence, minimal overfit on this dataset
**Decision:** DEEPEN — Test r=32 and r=64 to find saturation point
### H004 — Very High Rank (r=32)
**Results:**
- Convergence steps: 38 (-6% vs r=16)
- Final accuracy: 0.891 (-0.3%)
- **Diminishing returns observed**
### H005 — Optimal Search (r=6, r=10, r=12)
[Running...]
## Current Findings
1. Convergence speed improves with rank up to r=16, then plateaus
2. Final accuracy relatively stable across ranks (±0.5%)
3. For small datasets, r=8-12 appears optimal (speed vs compute tradeoff)
## Next Steps
- Complete H005-H007
- Test on different dataset sizes (generalization)
- Write up findings