CGW-Quote-Builder/information/AI_PARSING_INSTRUCTIONS.md

# AI Data Parsing Instructions

## 📋 Quick Reference

**Input**: `data/products.csv` (1000 rows with product and accessory data)

**Outputs**: 
1. `data/navigation.json` - Smart navigation flow with conditional questions
2. `data/products.json` - Product catalog with materials, colors, and accessory links
3. `data/accessories.json` - Accessory options with compatibility rules

**Key Features**:
- ✅ Conditional material questions (only show if multiple materials exist in sub-type)
- ✅ Conditional color questions (only show if multiple colors available)
- ✅ Auto-skip questions when only one option exists
- ✅ Smart product-accessory linking by sub-type, category, or specific product code
- 🔄 Multi-group assignment using bit values (optional enhancement)

---

## Overview
Parse the `data/products.csv` file to generate three separate JSON files that structure product data, navigation, and accessory options for a doors and windows e-commerce application.

## Source File Structure

### CSV File: `data/products.csv`
The CSV contains product and accessory data with the following column structure:

**Row 1**: Category headers (informational)
**Row 2**: Column names (actual headers)

### Column Mapping (0-indexed):
- **A (0)**: DISCONT - Discontinued flag (TRUE/FALSE)
- **B (1)**: LOC_CODE - Location code (e.g., "Iola")
- **C (2)**: PROD_CODE - Product code (unique identifier)
- **D (3)**: CATEGORY - Category code
- **E (4)**: CATEGORY_NEW - New category (usually empty)
- **F (5)**: DESCRIPTION - Full product description
- **G (6)**: Base Type - Primary type ("Window", "Door", or empty)
- **H (7)**: Sub-type Door - Door subcategory (e.g., "Storm Door")
- **I (8)**: Sub-type Window - Window subcategory (e.g., "Storm Window")
- **J (9)**: Accessory Yes - Boolean indicating if item is an accessory
- **K (10)**: This Item - Boolean for specific item relationship
- **L+ (11+)**: Material and Color availability (Aluminum, Vinyl, Black, White, Bronze, Tan, Mill, Sandstone, etc.)

---

## Output File 1: `data/navigation.json`

### Purpose
Generate a navigation flow with questions and button options to guide users through product selection.

### ⚡ Key Navigation Principles

1. **Question 1 (Start)**: Built from Column G (Base Type) - Door, Window, etc.
2. **Question 2 (Sub-Type)**: Built from Column H (Doors) OR Column I (Windows) based on Q1 selection
3. **Question 3 (Material)**: **CONDITIONAL** - Only show if sub-type has products with multiple materials (Aluminum AND Vinyl)
   - Example: Storm Doors (all Aluminum) → SKIP this question
   - Example: Mixed Windows (some Aluminum, some Vinyl) → SHOW this question
4. **Question 4 (Color)**: **CONDITIONAL** - Only show if filtered products have multiple color options
5. **Question 5 (Dimensions)**: Always show - final step before product display
6. **Dynamic Linking**: The "next" property must skip questions that aren't needed for that path

### Structure
```json
{
  "start": {
    "type": "question",
    "inputType": "button",
    "title": "What are you looking for?",
    "subtitle": "Select the product category",
    "answers": [
      {
        "caption": "PRODUCT_TYPE",
        "image": "EMOJI",
        "next": "NEXT_QUESTION_ID",
        "filter": {
          "baseType": "VALUE"
        }
      }
    ]
  },
  "q-QUESTION_ID": {
    "type": "question",
    "inputType": "button|form",
    "title": "Question Title",
    "subtitle": "Question subtitle",
    "answers": [...],
    "fields": [...]
  }
}
```

### Generation Rules

#### Question 1: Base Type Selection (Start)
**Source**: Column G (Base Type)
1. Extract all unique values from column G where DISCONT=FALSE
2. Create button option for each unique base type (e.g., "Window", "Door")
3. Assign appropriate emojis (🪟 for windows, 🚪 for doors)
4. Each answer links to the corresponding sub-type question

**Example**: "What are you looking for?" → Window / Door options

#### Question 2: Sub-Type Selection
**Source**: Column H (for Doors) OR Column I (for Windows)
1. **For Door selection**: Use column H (Sub-type Door)
   - Extract unique values where column G = "Door" and DISCONT=FALSE
   - Examples: "Storm Door", "Patio Door", etc.
   
2. **For Window selection**: Use column I (Sub-type Window)
   - Extract unique values where column G = "Window" and DISCONT=FALSE
   - Examples: "Storm Window", "Casement", etc.

3. Create separate question branches:
   - `q-door-type`: For door sub-types
   - `q-window-type`: For window sub-types

4. Each answer links to either material question (if needed) or dimensions question

**Example**: "What type of door?" → Storm Door / Patio Door options

#### Question 3: Material Selection (CONDITIONAL)
**Source**: Columns L (Aluminum) and M (Vinyl)
**Important**: This question should **only appear** if products in the selected sub-type have multiple material options.

**Logic**:
1. For each sub-type group, count materials:
   - Count products where Aluminum (column L) = TRUE
   - Count products where Vinyl (column M) = TRUE
   
2. **Show material question** if:
   - Some products have Aluminum=TRUE AND some have Vinyl=TRUE
   - OR any single product has BOTH Aluminum=TRUE AND Vinyl=TRUE
   
3. **Skip material question** if:
   - ALL products in the sub-type have only one material type
   - Example: All "Storm Door" products only have Aluminum=TRUE → Skip material question

4. If shown, create radio buttons or select dropdown with:
   - Only materials that exist in the sub-type group
   - Link to color question or dimensions

**Example Skip Case**: Storm Doors (all Aluminum only) → Skip directly to dimensions
**Example Show Case**: Windows (some Aluminum, some Vinyl) → Show material selection

#### Question 4: Color Selection (CONDITIONAL)
**Source**: Color columns (N through S+)
Similar conditional logic as materials:
1. Only show if products in the filtered group have multiple color options
2. Present only colors available for the selected material (if material was selected)
3. Skip if all products have the same single color

#### Question 5: Dimensions Entry (Always Show)
**Source**: User input
1. Width input field (number, required)
2. Height input field (number, required)
3. This is typically the final question before showing filtered products

#### Filter Integration
Each answer should accumulate filter criteria:
```json
"filter": {
  "baseType": "Window|Door",
  "subType": "Storm Door|etc",
  "material": "Aluminum|Vinyl",    // Only if material question was shown
  "color": "White|Bronze|etc",     // Only if color question was shown
  "category": "CATEGORY_CODE"
}
```

#### Navigation Flow Schema
```
start (Column G)
  → q-door-type (Column H) OR q-window-type (Column I)
    → q-material (Columns L,M) [CONDITIONAL - only if multiple materials exist]
      → q-color (Columns N+) [CONDITIONAL - only if multiple colors exist]
        → q-dimensions (User Input)
          → results (Filtered products)
```

**Dynamic Linking**: The "next" value for each question must be calculated based on whether the next conditional question is needed:
- If material question not needed → link sub-type directly to color or dimensions
- If color question not needed → link material (or sub-type) directly to dimensions

---

## Output File 2: `data/products.json`

### Purpose
Store all product data for display on product detail pages.

### Structure
```json
[
  {
    "id": "PROD_CODE",
    "productCode": "PROD_CODE",
    "category": "CATEGORY",
    "description": "DESCRIPTION",
    "discontinued": false,
    "location": "LOC_CODE",
    "baseType": "Window|Door",
    "subType": {
      "door": "VALUE_OR_NULL",
      "window": "VALUE_OR_NULL"
    },
    "materials": ["Aluminum", "Vinyl"],
    "colors": ["Black", "White", "Bronze", "Tan", "Mill", "Sandstone"],
    "isAccessory": false,
    "compatibleAccessories": ["PROD_CODE1", "PROD_CODE2"]
  }
]
```

### Generation Rules
1. **Include all rows** where `DISCONT` (column A) is FALSE
2. **Skip accessories** (column J = TRUE) - those go in accessories.json
3. **Materials array**: Include all material columns (L+) where value is TRUE
4. **Colors array**: Include all color columns where value is TRUE
5. **baseType**: Copy from column G
6. **subType**: Create object with "door" and "window" keys from columns H and I
7. **compatibleAccessories**: Populate based on accessories linked to this product (see Accessories section)

---

## Output File 3: `data/accessories.json`

### Purpose
Store accessory/option data that can be applied to products (e.g., handles for storm doors, glass inserts).

### Structure
```json
[
  {
    "id": "PROD_CODE",
    "accessoryCode": "PROD_CODE",
    "category": "CATEGORY",
    "description": "DESCRIPTION",
    "discontinued": false,
    "location": "LOC_CODE",
    "baseType": "Window|Door",
    "materials": ["Aluminum"],
    "colors": ["Black", "White"],
    "compatibilityRules": {
      "type": "subType|category|specific",
      "subTypeDoor": "Storm Door",
      "subTypeWindow": null,
      "categories": ["STD", "INSERTS"],
      "specificProducts": ["6100I", "8100I"],
      "requiresMatch": {
        "material": true,
        "color": false
      }
    },
    "optionType": "handle|insert|hardware|glass",
    "metadata": {
      "specificItemLink": false
    }
  }
]
```

### Generation Rules
1. **Include all rows** where `Accessory Yes` (column J) is TRUE
2. **Skip discontinued** items (column A = TRUE)
3. **compatibilityRules.type**: Determine based on available data
   - If column H (Sub-type Door) or column I (Sub-type Window) has value → "subType"
   - If column K (This Item) is TRUE → "specific" (requires product linking)
   - Otherwise → "category"

4. **compatibilityRules.subTypeDoor/Window**: Copy from columns H and I
5. **compatibilityRules.categories**: Extract from column D (CATEGORY)
6. **compatibilityRules.specificProducts**: To be populated based on column K logic
   - If column K is TRUE, this accessory is for specific products
   - You may need to analyze patterns in PROD_CODE or CATEGORY to determine relationships
   - Example: "BGIST" (inserts for storm doors) might be compatible with all storm door products

7. **optionType**: Infer from DESCRIPTION or CATEGORY
   - If DESCRIPTION contains "HANDLE" → "handle"
   - If DESCRIPTION contains "INSERT" → "insert"
   - If DESCRIPTION contains "HARDWARE" → "hardware"
   - If DESCRIPTION contains "GLASS" → "glass"
   - Default → "option"

8. **requiresMatch**: Set based on whether accessory materials/colors must match product
   - For handles/hardware: material matching often required
   - For inserts: usually flexible

---

## Multi-Group Assignment (Bit Values Consideration)

### Current Structure
Currently, each product belongs to a single category/sub-type based on columns G, H, and I.

### Proposed Enhancement: Bit Flags
To allow products to appear in multiple navigation groups, consider implementing bit flags for categories:

```json
{
  "productCode": "EXAMPLE",
  "description": "Multi-purpose product",
  "categoryFlags": 7,  // Binary: 111 (belongs to groups 1, 2, and 4)
  "categoryBits": {
    "residential": 1,    // 2^0 = 1
    "commercial": 2,     // 2^1 = 2  
    "industrial": 4,     // 2^2 = 4
    "custom": 8          // 2^3 = 8
  },
  "navigationGroups": ["residential", "commercial", "industrial"]
}
```

### Implementation Options

#### Option A: Additional CSV Columns
Add bit value columns to track multiple group memberships:
- Column U: Navigation Group Bits (integer)
- Column V: Secondary Category
- Column W: Tertiary Category

#### Option B: Parse from Description/Category
Analyze DESCRIPTION and CATEGORY fields to identify products that could belong to multiple groups:
- Keywords indicating dual-purpose (e.g., "residential/commercial")
- Multiple category codes separated by delimiter

#### Option C: JSON-Only Enhancement
Generate single-group assignments from CSV, then manually or programmatically enhance JSON with additional group memberships based on business rules.

### Navigation Impact
With bit values:
1. Multiple sub-type paths could lead to the same product
2. Products appear in search results for multiple filter combinations
3. Requires modification to filter logic to use bitwise operations

**Example**: A storm door that works for both residential and commercial applications could appear in both navigation paths.

### Refinement Needed
This feature requires:
- ✅ Business rules for multi-group assignment
- ✅ Decision on implementation approach (CSV vs JSON)
- ✅ Filter logic updates to handle bitwise comparisons
- ✅ Testing to ensure products appear in correct groups
- ✅ UI considerations (showing product appears in multiple categories)

---

## Data Relationships

### Products ↔ Accessories Linking

1. **By Sub-Type**: Accessories with matching sub-type values (columns H or I) are compatible
   - Example: Accessory with subTypeDoor="Storm Door" → compatible with all products where subType.door="Storm Door"

2. **By Category**: Accessories linked to specific CATEGORY codes
   - Example: Accessory with category="INSERTS" might be compatible with products in "STD" category

3. **By Specific Product Code**: Use column K (This Item) flag
   - If TRUE, requires manual mapping or pattern analysis
   - Consider adding a "specificProductCodes" field to link directly

### Recommendation Algorithm
When displaying accessories for a product:
```
1. Match by specificProducts first (exact match)
2. Match by subType (door or window)
3. Match by category
4. Filter by material/color compatibility if requiresMatch is true
5. Exclude discontinued accessories
```

---

## Processing Steps

### Step 1: Parse CSV
1. Read products.csv starting from row 3 (skip header rows 1-2)
2. Split each row by comma delimiter
3. Handle empty fields appropriately
4. Parse boolean values (TRUE/FALSE → true/false in JSON)

### Step 2: Categorize Rows
1. Separate products (column J = FALSE) from accessories (column J = TRUE)
2. Filter out discontinued items (column A = TRUE) or include with flag

### Step 3: Extract Navigation Data
1. Collect unique values from columns G, H, I
2. For each sub-type group, analyze material and color diversity
3. Build question hierarchy with conditional logic:
   - Level 1: Base Type (Window, Door) - from column G
   - Level 2: Sub-Type (Storm Door, Casement, etc.) - from columns H/I
   - Level 3: Material (CONDITIONAL) - from columns L, M
   - Level 4: Color (CONDITIONAL) - from color columns
   - Level 5: Dimensions & final inputs
4. Generate question flow with proper linking (next values)

#### Algorithm: Determine Material Question Necessity
```
For each sub-type group:
  products = filter products where subType matches AND DISCONT=FALSE
  
  aluminumCount = count products where Aluminum (col L) = TRUE
  vinylCount = count products where Vinyl (col M) = TRUE
  bothCount = count products where Aluminum=TRUE AND Vinyl=TRUE
  
  IF (aluminumCount > 0 AND vinylCount > 0) OR bothCount > 0:
    SHOW material question for this sub-type
    Create q-material-{subtype} with options for available materials
  ELSE:
    SKIP material question
    Link sub-type answer directly to color question or dimensions
    Auto-apply the single material to filter
```

#### Algorithm: Determine Color Question Necessity
```
For each (sub-type, material) combination:
  products = filter products where match AND DISCONT=FALSE
  
  availableColors = []
  For each color column (N through S+):
    IF any product has this color = TRUE:
      add color to availableColors
  
  IF len(availableColors) > 1:
    SHOW color question for this path
  ELSE IF len(availableColors) = 1:
    SKIP color question
    Auto-apply the single color to filter
  ELSE:
    SKIP color question (no color data)
```

### Step 4: Build Products Array
1. For each non-accessory, non-discontinued row:
   - Extract all product fields
   - Parse material/color columns into arrays
   - Create product object
   - Add to products array

### Step 5: Build Accessories Array
1. For each accessory row:
   - Extract accessory fields
   - Determine compatibility rules
   - Infer option type from description
   - Create accessory object
   - Add to accessories array

### Step 6: Link Products to Accessories
1. For each product, find compatible accessories based on:
   - Sub-type matching
   - Category matching
   - Specific product code matching
2. Populate compatibleAccessories array in products.json
3. Verify bidirectional relationships

### Step 7: Validate Output
1. Ensure all JSON is valid and properly formatted
2. Check that all question flows have valid "next" links
3. Verify product-accessory relationships are logical
4. Confirm no duplicate IDs exist

---

## Advanced Considerations

### Column Extensions
If additional columns are added beyond column T (~):
- Check for additional material/color flags
- Look for price, availability, or specification data
- Include in metadata or as new product properties

### Future Enhancements
You may want to extend the schema with:
1. **Pricing**: Add price fields to products and accessories
2. **Images**: Add image URLs or filenames
3. **Specifications**: Add detailed specs (dimensions, ratings, etc.)
4. **Availability**: Add stock levels or lead times
5. **Sorting**: Add sort order or priority fields

---

## Example Outputs

### Example Product Object
```json
{
  "id": "6100I",
  "productCode": "6100I",
  "category": "STD",
  "description": "STAR 6100 FULL VIEW STORM DOOR",
  "discontinued": false,
  "location": "Iola",
  "baseType": "Door",
  "subType": {
    "door": "Storm Door",
    "window": null
  },
  "materials": ["Aluminum"],
  "colors": ["White", "Bronze"],
  "isAccessory": false,
  "compatibleAccessories": ["BGIST", "TGIODD"]
}
```

### Example Accessory Object
```json
{
  "id": "BGIST",
  "accessoryCode": "BGIST",
  "category": "INSERTS",
  "description": "INSERTS FOR STORM DOORS",
  "discontinued": false,
  "location": "Iola",
  "baseType": "Door",
  "materials": ["Aluminum"],
  "colors": ["Black", "White", "Bronze", "Mill", "Sandstone"],
  "compatibilityRules": {
    "type": "subType",
    "subTypeDoor": "Storm Door",
    "subTypeWindow": null,
    "categories": ["STD", "PD"],
    "specificProducts": [],
    "requiresMatch": {
      "material": true,
      "color": false
    }
  },
  "optionType": "insert",
  "metadata": {
    "specificItemLink": false
  }
}
```

### Example Navigation Flow
```json
{
  "start": {
    "type": "question",
    "inputType": "button",
    "title": "What are you looking for?",
    "subtitle": "Select the product category",
    "answers": [
      {
        "caption": "Door",
        "image": "🚪",
        "next": "q-door-type",
        "filter": { "baseType": "Door" }
      },
      {
        "caption": "Window",
        "image": "🪟",
        "next": "q-window-type",
        "filter": { "baseType": "Window" }
      }
    ]
  },
  "q-door-type": {
    "type": "question",
    "inputType": "button",
    "title": "What type of door?",
    "subtitle": "Select the door category",
    "answers": [
      {
        "caption": "Storm Door",
        "image": "🚪",
        "next": "q-dimensions",
        "filter": { 
          "baseType": "Door",
          "subType": "Storm Door",
          "material": "Aluminum"
        },
        "note": "Material question skipped - all storm doors are aluminum only"
      },
      {
        "caption": "Patio Door",
        "image": "🚪",
        "next": "q-material-patio",
        "filter": { 
          "baseType": "Door",
          "subType": "Patio Door"
        },
        "note": "Material question shown - patio doors have multiple material options"
      }
    ]
  },
  "q-material-patio": {
    "type": "question",
    "inputType": "button",
    "title": "Select Material",
    "subtitle": "Choose your preferred material for Patio Door",
    "answers": [
      {
        "caption": "Aluminum",
        "image": "🔩",
        "next": "q-color-patio-aluminum",
        "filter": {
          "baseType": "Door",
          "subType": "Patio Door",
          "material": "Aluminum"
        }
      },
      {
        "caption": "Vinyl",
        "image": "🪟",
        "next": "q-color-patio-vinyl",
        "filter": {
          "baseType": "Door",
          "subType": "Patio Door",
          "material": "Vinyl"
        }
      }
    ],
    "conditional": {
      "showIf": "multipleOptionsExist",
      "check": "materials",
      "fallbackNext": "q-dimensions"
    }
  },
  "q-color-patio-aluminum": {
    "type": "question",
    "inputType": "button",
    "title": "Select Color",
    "subtitle": "Choose your preferred color",
    "answers": [
      {
        "caption": "White",
        "next": "q-dimensions",
        "filter": {
          "baseType": "Door",
          "subType": "Patio Door",
          "material": "Aluminum",
          "color": "White"
        }
      },
      {
        "caption": "Bronze",
        "next": "q-dimensions",
        "filter": {
          "baseType": "Door",
          "subType": "Patio Door",
          "material": "Aluminum",
          "color": "Bronze"
        }
      }
    ]
  },
  "q-dimensions": {
    "type": "question",
    "inputType": "form",
    "title": "Enter Product Dimensions",
    "subtitle": "Please provide the measurements",
    "fields": [
      {
        "name": "width",
        "label": "Width (inches)",
        "type": "number",
        "required": true,
        "placeholder": "e.g., 36"
      },
      {
        "name": "height",
        "label": "Height (inches)",
        "type": "number",
        "required": true,
        "placeholder": "e.g., 80"
      }
    ],
    "next": "results"
  }
}
```

**Key Points in Example**:
- Storm Door skips material question (goes directly to dimensions)
- Patio Door shows material question (multiple materials available)
- Material selection leads to color-specific questions
- All paths eventually reach dimensions entry

---

## Final Notes

1. **Data Quality**: Some rows may have inconsistent data. Handle gracefully with defaults.
2. **Empty Values**: Treat empty strings as null in JSON
3. **Boolean Conversion**: CSV TRUE/FALSE should become JSON true/false
4. **Unique IDs**: PROD_CODE serves as the unique identifier
5. **Relationships**: The linking between products and accessories may require iterative refinement based on business rules

## Questions to Consider

When implementing, clarify:
1. Should discontinued items be included in ANY output file?
2. How should accessories with column K=TRUE be specifically linked?
3. Are there additional columns beyond column S that need parsing?
4. Should colors and materials be validated against a master list?
5. What should happen if a product has no compatible accessories?
6. **Should the navigation include filters for materials/colors early, or determine dynamically based on product availability?** ✅ RESOLVED: Dynamic based on availability
7. **For sub-types with only one material option, should that material be auto-applied to the filter?** → YES, skip question and auto-apply
8. **Multi-group assignment (bit values):**
   - Should products be able to appear in multiple navigation paths?
   - If yes, how should this be indicated in the CSV? (new column, description parsing, manual JSON editing?)
   - What business rules determine multi-group membership?
   - Should search results indicate a product appears in multiple categories?
9. **Material/Color question threshold:**
   - Current logic: Show if > 1 option exists
   - Alternative: Show only if > X% of products have multiple options (e.g., 20% threshold)
   - Should we show questions even if only 1-2 products have alternative options?

---

## Success Criteria

The parsing is complete when:
- ✅ All three JSON files are generated and valid
- ✅ Products.json contains all non-accessory, non-discontinued products
- ✅ Accessories.json contains all accessory items with proper compatibility rules
- ✅ Navigation.json provides a complete question flow from start to product selection
- ✅ Product-accessory relationships are established and logical
- ✅ All materials and colors are properly extracted into arrays
- ✅ No data loss from the original CSV