# 💎 Data Quality, Deduplication & Timezone Architecture

### منظومة تدقيق وجودة البيانات وتوحيد السجلات والمطابقة الزمنية

The **Data Quality & Deduplication Hub** (`/admin/data-quality`) is an advanced data integrity and harmonization center built into the Al-Hilal Medical Visits platform. It provides automated anomaly detection, deduplication algorithms, title prefix standardization, and clinic workplace cleanup for both **Doctors** and **Pharmacies**, while maintaining synchronization with the **Asia/Damascus** local timezone across all field operations.

---

## 📑 Table of Contents

1. [Overview & Core Objectives](#1-overview--core-objectives)
2. [Doctor Honorific Prefix Scanner & Standardizer](#2-doctor-honorific-prefix-scanner--standardizer)
3. [Double Clinic Prefix Cleaner & Dal-Letter Protection](#3-double-clinic-prefix-cleaner--dal-letter-protection)
4. [Deduplication Algorithms & Conflict Resolution](#4-deduplication-algorithms--conflict-resolution)
5. [Mobile-First Responsive Interface](#5-mobile-first-responsive-interface)
6. [Asia/Damascus Timezone Standard (+2h Discrepancy Elimination)](#6-asiadamascus-timezone-standard-2h-discrepancy-elimination)
7. [Service & Controller Architecture](#7-service--controller-architecture)
8. [Testing & Verification](#8-testing--verification)

---

## 1. Overview & Core Objectives

When importing or managing hundreds of healthcare practitioners across Syria, real-world data frequently exhibits:
- Inconsistent honorific prefixes (`د.`, `د/`, `دكتور`, `الدكتور`, `Dr.`) stored inside doctor names, leading to missed searches, duplicate entries, and messy reporting.
- Redundant nested clinic labels (e.g., `عيادة د. د. أحمد`, `عيادة د. دكتور سامر`).
- Duplicate doctor and pharmacy records created across different visits or regional imports with slight variations in phone numbers or spelling.
- UTC timestamp drift causing scheduled field visits to display ±2 hours off from Damascus local time.

The Data Quality Hub provides a central dashboard for administrators to audit, preview, and resolve these issues in real time with atomic database transactions.

---

## 2. Doctor Honorific Prefix Scanner & Standardizer

### 2.1 Problem Statement
When a doctor's name is saved with `د.` or `دكتور`, standard queries like `WHERE name LIKE 'أحمد%'` fail to locate them, and the UI displays repetitive titles like `د. د. أحمد`. Furthermore, naive stripping rules can accidentally truncate legitimate Arabic names starting with `أد`/`اد` (such as `أديب`, `ادم`, `ادريس`, `ادهم`).

### 2.2 Safe Normalization Engine (`ArabicNormalizer::cleanDoctorName`)
The system employs a strict Arabic-aware regular expression in `App\Support\ArabicNormalizer`:

```php
public static function cleanDoctorName(?string $name): string
{
    if (empty($name)) return '';
    $clean = trim($name);

    // Matches honorific prefixes followed by punctuation, spaces, or slashes:
    // (دكتور|الدكتور|دكتورة|الدكتورة|بروفيسور|أخصائي|اخصائي|استشاري|طبيب|dr\.?|doctor)
    // or standalone 'د' followed by dot/space/slash
    $pattern = '/^(?:(?:دكتور[ة]?|الدكتور[ة]?|بروفيسور|أخصائي[ة]?|اخصائي[ة]?|استشاري[ة]?|طبيب[ة]?|dr\.?|doctor)[\s\.\:\-\/]+|د[\.\:\-\/\s]+)+/ui';
    
    $clean = preg_replace($pattern, '', $clean);
    return trim($clean);
}
```

**Name Safety Assurance**:
Genuine names like `أديب` (Adib), `ادم` (Adam), or `ادريس` (Idris) do not match because they do not have a separator after the `د`, ensuring zero false-positive truncations.

### 2.3 Collision-Aware Standardization (`standardizeDoctorNames`)
When removing a prefix, the cleaned name might collide with an existing doctor in the same area. The system handles this automatically:
1. **Target Inspection**: Determines if a doctor with the cleaned name already exists in the same specialty and area.
2. **Merge Collision**: If a duplicate exists, reassigns all visits, representative targets, workplaces, and activity logs to the canonical record, then soft-deletes the prefixed duplicate.
3. **Clinic Workplace Synchronization**: Automatically inspects the doctor's primary `doctor_workplaces` record and generates/cleans the clinic name to match `عيادة د. {Cleaned Name}`.

---

## 3. Double Clinic Prefix Cleaner & Dal-Letter Protection

### 3.1 Problem Statement
Repeated imports or manual entries often cause nested clinic prefixes:
`عيادة د. د. زياد` or `عيادة دكتور د. خالد`.

### 3.2 Arabic Dal-Letter (`د`) Protection
A critical algorithmic edge case exists in Arabic: several common Arabic given names begin with the letter Dal (`د`), such as:
- `دريد` (Duraid)
- `ديانا` (Diana)
- `ديما` (Dima)
- `دعاء` (Doaa)
- `دلال` (Dalal)

A naive query searching for `عيادة د. د%` would incorrectly flag `عيادة د. دريد` as a "double clinic" because `دريد` starts with `د`!

### 3.3 Protection Implementation
`ArabicNormalizer::isDoublePrefixClinic` strictly verifies that the second `د` is indeed a title and not the first letter of the doctor's actual name:

```php
public static function isDoublePrefixClinic(string $clinicName): bool
{
    // Match 'عيادة د. د.' or 'عيادة د. دكتور' with explicit delimiter after the second prefix
    $pattern = '/^عيادة\s*(?:د[\.\s\/]+|دكتور[ة]?[\.\s\/]+)+(?:د[\.\s\/]+|دكتور[ة]?[\.\s\/]+)+/ui';
    return (bool) preg_match($pattern, trim($clinicName));
}
```
If a clinic name is `عيادة د. دريد`, the second token is `دريد` (no dot or space following `د`), correctly recognizing it as a single prefix clinic and ignoring it during cleanups.

---

## 4. Deduplication Algorithms & Conflict Resolution

The Data Quality Hub categorizes duplicates into distinct audit streams:

| Stream | Criteria | Resolution Mechanism |
|---|---|---|
| **Doctor Phone Clones** | Same normalized phone number (`09xxxxxxxx`) | Direct atomic merge into chosen canonical master record |
| **Doctor Name & Area Clones** | Same normalized Arabic name within the same administrative zone | Detailed comparative modal displaying visit history, ratings, and clinics |
| **Pharmacy Exact Clones** | Identical phone or normalized license name within same area | Merges inventory surveys, visits, and soft-deletes clone |
| **Prefixed Names Audit** | Doctors with `د.`/`دكتور` in the `name` column | One-click bulk standardize or individual row standardizer |
| **Double Clinic Audit** | Clinics with repeated `عيادة د. د.` prefixes | Single-click mass cleanup to canonical `عيادة د. {Name}` |

### Merge Integrity:
- Transfers all `visits` foreign keys.
- Transfers all `doctor_workplaces` and updates unique primary flags.
- Re-associates all representative scheduling and targets.
- Preserves full audit logs in `activity_logs`.

---

## 5. Mobile-First Responsive Interface

The frontend (`resources/js/pages/Admin/DataQuality/Index.vue`) is engineered for seamless operation on mobile smartphones, tablets, and desktop workstations:

- **Balanced Main Tabs**: Doctors (`الأطباء`) and Pharmacies (`الصيدليات`) occupy equal 50%/50% width on mobile screens.
- **Responsive KPI Grid**: 5 metric badges dynamically adapt from `grid-cols-2` on mobile (with the 5th card spanning full width) to `lg:grid-cols-5` on desktop.
- **Touch-Scrollable Subtabs**: Horizontal pill navigation allows smooth finger-swipe browsing across audit streams without wrapping.
- **Dual Presentation Mode**:
  - **Mobile (< 768px)**: Compact, touch-friendly cards showing candidate comparison, badges, and quick-action buttons.
  - **Desktop (≥ 768px)**: Data-dense tables with sortable columns, phone tags, and inline controls.
- **Compact Mobile Pagination (`ClientPagination.vue`)**: Automatically renders a streamlined `[السابق] [ 1 / 15 ] [التالي]` interface on mobile screens to save screen estate, while offering the full numeric button strip on larger screens.

---

## 6. Asia/Damascus Timezone Standard (+2h Discrepancy Elimination)

### 6.1 Issue Diagnosis
Users experienced a +2-hour shift on visit logs and reports (e.g., a visit logged at `10:00` appeared as `12:00` or `08:00`). This was caused by:
1. `config/app.php` defaulting to `UTC`.
2. Browser JavaScript parsing ISO timestamps with a trailing `Z` or UTC offset and shifting them to local operating system settings.
3. Date pickers in `Visits/Edit.vue` expecting `YYYY-MM-DDTHH:mm`.

### 6.2 Three-Tiered Fix Applied

1. **Framework Timezone Configuration (`config/app.php`)**:
   ```php
   'timezone' => env('APP_TIMEZONE', 'Asia/Damascus'),
   ```
   Ensures all Carbon dates, scheduled tasks, and database timestamps are anchored to Damascus local time (`UTC+3`).

2. **Model Date Serialization (`Visit.php` & `ArchivedVisit.php`)**:
   ```php
   protected function casts(): array
   {
       return [
           'visited_at' => 'datetime:Y-m-d H:i:s',
           // ...
       ];
   }
   ```
   Prevents Laravel from serializing `visited_at` as an ISO-8601 UTC string (`...Z`), preventing unintended client-side timezone conversions.

3. **Client-Side Safe Date Parsing (`useDateFormat.ts`)**:
   ```typescript
   export function parseLocalDate(dtStr: string): Date {
       // Strips ISO 'Z' and UTC markers so dates are treated strictly as local time
       const cleaned = dtStr.replace('T', ' ').replace(/\.\d+Z$/, '').replace(/Z$/, '').trim();
       return new Date(cleaned.replace(/-/g, '/'));
   }
   ```

---

## 7. Service & Controller Architecture

### Key Classes:
- `App\Support\ArabicNormalizer`: Canonical regex routines for Arabic name cleaning, phone normalization, and clinic generation.
- `App\Services\DataQualityService`: Core analytical service executing bulk deduplication, duplicate grouping, prefix scanning, and transactional merges.
- `App\Http\Controllers\Admin\DataQualityController`: Inertia controller handling route authorization, index prop hydration, and POST actions.

### Routes (`routes/web.php`):
```php
Route::middleware(['auth', 'role:Super Admin|Admin'])->prefix('admin/data-quality')->name('admin.data-quality.')->group(function () {
    Route::get('/', [DataQualityController::class, 'index'])->name('index');
    Route::post('/merge-doctors', [DataQualityController::class, 'mergeDoctors'])->name('merge-doctors');
    Route::post('/merge-pharmacies', [DataQualityController::class, 'mergePharmacies'])->name('merge-pharmacies');
    Route::post('/standardize-doctor-names', [DataQualityController::class, 'standardizeDoctorNames'])->name('standardize-doctor-names');
    Route::post('/clean-double-clinics', [DataQualityController::class, 'cleanDoubleClinics'])->name('clean-double-clinics');
});
```

---

## 8. Testing & Verification

Comprehensive automated feature tests are maintained in `tests/Feature/DataQualityAndDeduplicationTest.php`:

```bash
# Execute full Data Quality & Deduplication test suite
php artisan test --filter=DataQualityAndDeduplicationTest
```

**Validated Test Scenarios**:
1. Unauthorized users are blocked from `/admin/data-quality`.
2. Exact phone duplicates for doctors are identified and safely merged.
3. Area + normalized name duplicates are accurately identified.
4. Pharmacy duplicates are merged with visit and inventory reassignment.
5. Doctor honorific title prefix scanner accurately isolates `د.` and `دكتور` records while preserving `أديب` and other genuine Arabic names.
6. Doctor name standardization merges collisions and updates associated clinic names.
7. Double clinic prefixes (`عيادة د. د.`) are cleaned while preserving names beginning with Dal (`د`).