mnemo_cards/mnemo_cards_backend/workflow_state.md

# Workflow State - mnemo_cards_backend

**Last Updated:** 2025-11-08

---

## PLAN - 🔥 STATISTICS SYSTEM UPGRADE (BACKEND)

**Phase:** Backend Statistics Implementation  
**Goal:** Extend backend models, API, and automatic tracking for comprehensive user statistics

**Plan Document:** `STATISTICS_TASKS.md` (35-47 hours total)

**Status:** 🟡 PLANNING COMPLETE - READY TO START

---

## CURRENT STATUS

### Project State:
- ✅ API v2 Auth working (Google OAuth, JWT, refresh tokens)
- ✅ Core API structure in place
- ✅ Isar database operational
- ✅ Dependency injection with GetIt/Injectable
- ⬜ Statistics system (old, needs upgrade)

### Recent Work:
- ✅ Replaced dynamic user handling with typed UserModel
- ✅ Added RefreshTokenModel for token management
- ✅ Improved JWT implementation with crypto
- ✅ Added access control policies

---

## NEXT_ACTIONS

### Phase 1: Models and DTOs ✅ COMPLETED (8 hours)
1. ✅ Create PackProgressDto in mnemo_cards_common
2. ✅ Create AchievementDto and AchievementType enum
3. ✅ Create DetailedWordStatisticsDto extending WordStatisticsDto
4. ✅ Create StudySessionDto for session tracking
5. ✅ Extend UserDataDto with new statistics fields
6. ✅ Create Isar models (PackProgressModel, AchievementModel, StudySessionModel)
7. ✅ Update UserDataModel with new embedded relations
8. ✅ Run codegen (manual .g.dart creation due to build_runner issues)
9. ⬜ Write unit tests for all models

### Phase 2: Statistics Calculator ✅ COMPLETED (5 hours)
10. ✅ Create StatisticsCalculator service with @lazySingleton
11. ✅ Implement calculatePackProgress method with DTO conversion
12. ✅ Implement calculateStreak (consecutive days logic with date normalization)
13. ✅ Implement findDifficultWords method with difficulty scoring
14. ✅ Implement calculateAccuracy method for word statistics
15. ✅ Implement calculateStudyTime methods (total and daily aggregation)
16. ✅ Implement getTimelineStatistics method with period filtering
17. ✅ Add StatisticsCalculator to DI (GetIt/Injectable auto-registration)
18. ✅ Write comprehensive unit tests (23 tests, all passing)

### Phase 3: API Endpoints ✅ COMPLETED (7 hours)
19. ✅ Add getDetailedStatistics endpoint to UsersApiV2 (/api/v2/users/me/statistics/detailed)
20. ✅ Add getPacksStatistics endpoint with packId filter (/api/v2/users/me/statistics/packs)
21. ✅ Add getWordsStatistics endpoint with pagination and filters (/api/v2/users/me/statistics/words)
22. ✅ Add getTimelineStatistics endpoint with period filtering (/api/v2/users/me/statistics/timeline)
23. ✅ Add recordStudySession endpoint for session tracking (/api/v2/users/me/sessions)
24. ✅ Add getAchievements endpoint (/api/v2/users/me/achievements)
25. ✅ Update UserManager with statistics methods (toDto extension)
26. ✅ Write integration tests (9 tests, all passing)
27. ⬜ Update OpenAPI specification

### Phase 4: Automatic Tracking (AFTER - 9-12 hours)
28. ✅ Create SessionTracker service (COMPLETED - 2 hours)
29. ✅ Add session tracking middleware (COMPLETED - 1 hour)
30. ✅ Add hooks in test completion flow (COMPLETED - 2 hours)
31. ✅ Create AchievementManager (COMPLETED - 3 hours)
32. ✅ Define all achievement types (COMPLETED - included)
33. ✅ Implement achievement checking logic (COMPLETED - included)
34. ✅ Add achievement hooks to user actions (COMPLETED - 1 hour)
35. ✅ Write comprehensive tests (COMPLETED - 2 hours)

### Phase 5: Testing and Docs (FINAL - 5-7 hours)
36. ⬜ Complete all unit tests
37. ⬜ Complete all integration tests
38. ⬜ Update OpenAPI specification
39. ⬜ Update PROGRESS.md
40. ⬜ Update TODO.md
41. ⬜ Create STATISTICS_API.md documentation

---

## ASSUMPTIONS

### Technical:
1. Isar database can handle new models without migration issues
2. Statistics calculations can be done synchronously (fast enough)
3. Streak calculation runs daily via cron job
4. Session tracking uses in-memory cache with persistence
5. Achievement checking is asynchronous (won't block requests)

### Business Logic:
6. Streak counts consecutive calendar days (user's timezone)
7. Difficulty score based on incorrect/correct ratio
8. Study session timeout is 30 minutes
9. Achievements are retroactive (can be unlocked for past data)
10. Statistics are cached for 5 minutes

### Architecture:
11. Use existing UserManager for user operations
12. Create separate StatisticsCalculator for stats logic
13. SessionTracker runs as middleware
14. AchievementManager is triggered by events
15. All stats endpoints require authentication

---

## PROGRESS_LOG

### 2025-11-08: Statistics Planning Complete ✅

**Analysis:**
- Reviewed current UserModel, UserDataModel structures
- Identified fields needed: streaks, pack progress, achievements, sessions
- Analyzed existing statistics collection (TestStatisticsDto, WordStatisticsDto)
- Examined UserManager methods for statistics updates

**Planning:**
- Created STATISTICS_TASKS.md with detailed breakdown
- 5 phases: Models (6-9h), Calculator (4-5h), API (6-8h), Tracking (9-12h), Testing (5-7h)
- Total: 35-47 hours estimated
- Clear dependencies: Models → Calculator → API → Tracking → Testing

**Key Components Planned:**

**New DTOs:**
- PackProgressDto (pack stats per user)
- AchievementDto (achievements with unlock dates)
- DetailedWordStatisticsDto (extended word stats)
- StudySessionDto (session tracking)

**New Services:**
- StatisticsCalculator (all calculation logic)
- SessionTracker (automatic session tracking)
- AchievementManager (achievement checking and awarding)

**New Endpoints:**
- GET /api/v2/users/me/statistics/detailed
- GET /api/v2/users/me/statistics/packs
- GET /api/v2/users/me/statistics/words (with pagination)
- GET /api/v2/users/me/statistics/timeline
- POST /api/v2/users/me/sessions
- GET /api/v2/users/me/achievements

**Next Step:** Create new DTOs in mnemo_cards_common

---

### Previous Work:

#### Recent Updates:
- ✅ Fixed access control to use typed UserModel
- ✅ Added refresh token tests
- ✅ Improved Packs API v2 test coverage
- ✅ Added optional authentication handling

#### API v2 Implementation:
- ✅ JWT Service with proper crypto
- ✅ Authentication API (Google OAuth, tokens)
- ✅ Packs API (list, details, cards, images)
- ✅ Tests API (details, results, history)
- ✅ Games API (list, assets)
- ✅ Purchases API (create, verify)

---

## OPEN_ISSUES

### Statistics Feature:
1. ⬜ Decide on difficulty scoring formula (incorrect/(correct+incorrect)?)
2. ⬜ Determine achievement unlock criteria precisely
3. ⬜ Plan database migration for new Isar models
4. ⬜ Decide on caching strategy (in-memory? Redis?)
5. ⬜ Handle timezone for streak calculations (use user's timezone from request?)
6. ⬜ Define session timeout behavior (auto-end after 30 min inactivity?)
7. ⬜ Plan for performance with large datasets (indexes needed?)
8. ⬜ Decide on pagination defaults (50 items per page?)

### General:
9. ⬜ Need to run all existing tests after model changes
10. ⬜ Consider adding rate limiting for statistics endpoints
11. ⬜ Plan for data export (GDPR compliance)
12. ⬜ Consider adding admin endpoints for statistics (analytics)

---

## DEPENDENCIES

### External:
- mnemo_cards_common (shared DTOs) - will be modified
- mnemo_cards_common_backend (shared models) - will be modified
- Isar database - will add new collections
- Shelf HTTP framework - existing
- GetIt/Injectable - existing

### Internal:
- Phase 1 (Models) must complete before Phase 2 (Calculator)
- Phase 2 (Calculator) must complete before Phase 3 (API)
- Phase 3 (API) must complete before Phase 4 (Tracking)
- Phase 5 (Testing) runs parallel to implementation

---

## TESTING STRATEGY

### Unit Tests:
- All DTO serialization/deserialization
- All model conversions (toDto/fromDto)
- All StatisticsCalculator methods
- Streak calculation logic
- Difficulty scoring logic
- Achievement checking logic

### Integration Tests:
- All new API endpoints
- End-to-end statistics flow
- Session tracking
- Achievement awarding

### Test Coverage Goal:
- 80%+ coverage for new code
- 100% coverage for business logic (streaks, achievements)

---

## ACCEPTANCE CRITERIA

### Phase 1 - Models:
- [ ] All new DTOs created and working
- [ ] Codegen runs without errors
- [ ] Models convert to/from DTOs correctly
- [ ] Unit tests pass

### Phase 2 - Calculator:
- [ ] StatisticsCalculator service created
- [ ] All calculation methods implemented
- [ ] Unit tests cover all methods
- [ ] Calculations are accurate

### Phase 3 - API:
- [ ] All 6 new endpoints working
- [ ] Request validation implemented
- [ ] Error handling proper
- [ ] Integration tests pass
- [ ] OpenAPI spec updated

### Phase 4 - Tracking:
- [ ] SessionTracker middleware working
- [ ] Test completion updates statistics
- [ ] AchievementManager awards achievements
- [ ] All tests pass

### Phase 5 - Final:
- [ ] All tests pass
- [ ] Documentation updated
- [ ] Code reviewed
- [ ] Ready for frontend integration

---

## WORK STRATEGY

### Development Approach:
1. **Start with DTOs:** Foundation for everything
2. **Test Models:** Ensure serialization works
3. **Build Calculator:** Pure logic, easy to test
4. **Add Endpoints:** Connect calculator to API
5. **Implement Tracking:** Make it automatic
6. **Test Everything:** Comprehensive testing

### Quality:
- Write tests alongside code
- Run `./codegen.sh` after model changes
- Run `flutter test` frequently
- Check `dart analyze` before committing
- Keep methods small and focused

### Documentation:
- Update workflow_state.md daily
- Update PROGRESS.md after each phase
- Update TODO.md as tasks complete
- Create STATISTICS_API.md for API docs

---

## FILES TO TRACK

**Models & DTOs:**
- `mnemo_cards_common/lib/src/dtos/user/data/pack_progress_dto.dart`
- `mnemo_cards_common/lib/src/dtos/user/achievement_dto.dart`
- `mnemo_cards_common/lib/src/dtos/user/data/detailed_word_statistics_dto.dart`
- `mnemo_cards_common/lib/src/dtos/user/study_session_dto.dart`
- `mnemo_cards_common/lib/src/dtos/user/data/user_data_dto.dart`
- `mnemo_cards_common_backend/lib/src/models/pack_progress_model.dart`
- `mnemo_cards_common_backend/lib/src/models/achievement_model.dart`
- `mnemo_cards_common_backend/lib/src/models/study_session_model.dart`

**Services:**
- `lib/statistics/statistics_calculator.dart`
- `lib/statistics/session_tracker.dart`
- `lib/statistics/achievement_manager.dart`

**API:**
- `lib/api/v2/users_api_v2.dart`
- `lib/user/user_manager.dart`
- `public/open_api.yaml`

**Tests:**
- `test/statistics/statistics_calculator_test.dart`
- `test/statistics/achievement_manager_test.dart`
- `test/api/v2/users_api_v2_statistics_test.dart`

**Documentation:**
- `PROGRESS.md`
- `TODO.md`
- `STATISTICS_API.md` (to be created)

---

## COMMANDS REFERENCE

```bash
# Code generation
./codegen.sh

# Run all tests
flutter test

# Run specific test file
flutter test test/statistics/statistics_calculator_test.dart

# Analyze code
dart analyze

# Format code
dart format lib/ test/

# Build (for verification)
flutter build

# Run dev server
./run_dev.sh

# Restart dev server
./restart_dev.sh
```

---

**Status:** Ready to begin implementation  
**Next Action:** Create PackProgressDto in mnemo_cards_common  
**Estimated Time for Next Phase:** 6-9 hours  
**Total Remaining:** 35-47 hours