CGW-Quote-Builder/planning/ENCODING_TABLE_GENERATION.md

Encoding Table Generation System

Problem Statement

Currently, encoding lookup tables require manual maintenance of both forward and reverse mappings:

// Must maintain both of these manually
const PRODUCT_COLOR = {
    'BLACK': 1,
    'BRONZE': 2,
    // ... add new color here
};

const PRODUCT_COLOR_REVERSE = {
    1: 'BLACK',
    2: 'BRONZE',
    // ... and remember to add it here too!
};

Issues:

• ❌ Prone to human error (forgetting reverse mapping)
• ❌ Risk of mismatches between forward/reverse
• ❌ No validation for duplicate values
• ❌ No documentation of available slots
• ❌ Difficult for non-technical users to add values

Proposed Solution: Python-Generated Tables

Architecture

User edits simple source file
         ↓
Python validation script
         ↓
Generate JavaScript + Python constants
         ↓
Auto-served by Flask (or manual build step)

File Structure

app/
  data/
    encoding_values.json (or .yaml)  ← USER EDITS THIS
  js/
    encoding_tables.js               ← AUTO-GENERATED
  generate_encoding_tables.py        ← GENERATOR SCRIPT

Source File Format

Option 1: JSON with Comments Support

app/data/encoding_values.json

{
  "metadata": {
    "version": 0,
    "description": "Product encoding value definitions"
  },
  "tables": {
    "product_type": {
      "max_value": 15,
      "bit_size": 4,
      "values": [
        {"name": "PATIO_DOOR", "value": 1, "display": "Patio Door"},
        {"name": "STORM_DOOR", "value": 3, "display": "Storm Door"},
        {"name": "STORM_WINDOW", "value": 5, "display": "Storm Window"},
        {"name": "PRIMARY_WINDOW", "value": 7, "display": "Primary Window"}
      ]
    },
    "product_color": {
      "max_value": 15,
      "bit_size": 4,
      "values": [
        {"name": "BLACK", "value": 1, "display": "Black"},
        {"name": "BRONZE", "value": 2, "display": "Bronze"},
        {"name": "SANDSTONE", "value": 3, "display": "Sandstone"},
        {"name": "WHITE", "value": 4, "display": "White"},
        {"name": "TAN", "value": 5, "display": "Tan"},
        {"name": "MILL", "value": 6, "display": "Mill"}
      ]
    },
    "product_material": {
      "max_value": 15,
      "bit_size": 4,
      "values": [
        {"name": "ALUMINUM", "value": 1, "display": "Aluminum"},
        {"name": "VINYL", "value": 2, "display": "Vinyl"}
      ]
    },
    "hardware_type": {
      "max_value": 31,
      "bit_size": 5,
      "values": [
        {"name": "NONE", "value": 0, "display": "None"},
        {"name": "LEVER", "value": 1, "display": "Lever"},
        {"name": "PULL", "value": 2, "display": "Pull"},
        {"name": "PULL_HANDLE", "value": 3, "display": "Pull Handle"},
        {"name": "DEADBOLT", "value": 4, "display": "Deadbolt"},
        {"name": "HINGE", "value": 5, "display": "Hinge"}
      ]
    },
    "hardware_style": {
      "max_value": 31,
      "bit_size": 5,
      "values": [
        {"name": "NONE", "value": 0, "display": "None"},
        {"name": "STANDARD", "value": 1, "display": "Standard"},
        {"name": "PUSH", "value": 2, "display": "Push"},
        {"name": "ALTERNATIVE", "value": 3, "display": "Alternative"},
        {"name": "CONTEMPORARY", "value": 4, "display": "Contemporary"},
        {"name": "TRADITIONAL", "value": 5, "display": "Traditional"}
      ]
    },
    "hardware_color": {
      "max_value": 31,
      "bit_size": 5,
      "values": [
        {"name": "NONE", "value": 0, "display": "None"},
        {"name": "BRASS", "value": 1, "display": "Brass"},
        {"name": "WHITE", "value": 2, "display": "White"},
        {"name": "BLACK", "value": 3, "display": "Black"},
        {"name": "SATIN", "value": 4, "display": "Satin"},
        {"name": "NICKEL", "value": 5, "display": "Nickel"},
        {"name": "BRONZE", "value": 6, "display": "Bronze"}
      ]
    }
  }
}

Option 2: YAML (Better for Comments)

app/data/encoding_values.yaml

version: 0
description: Product encoding value definitions

tables:
  product_type:
    max_value: 15
    bit_size: 4
    values:
      - name: PATIO_DOOR
        value: 1
        display: Patio Door
      - name: STORM_DOOR
        value: 3
        display: Storm Door
      # Reserved: 2, 4, 6, 8-15
      
  product_color:
    max_value: 15
    bit_size: 4
    values:
      - name: BLACK
        value: 1
        display: Black
      - name: BRONZE
        value: 2
        display: Bronze
      # ... etc
      # Reserved: 7-15 (9 slots available)

User Workflow

Adding a New Color

Before (Manual):

1. Open encoding_helper.js
2. Find PRODUCT_COLOR object
3. Add new entry with unused value
4. Find PRODUCT_COLOR_REVERSE object
5. Add matching reverse entry
6. Hope you didn't make a mistake

After (Generated):

1. Open encoding_values.json
2. Add one line: {"name": "CHARCOAL", "value": 7, "display": "Charcoal"}
3. Run python generate_encoding_tables.py (or auto-regenerated by Flask)
4. Done! JavaScript and Python constants updated

Generator Script

app/generate_encoding_tables.py

#!/usr/bin/env python3
"""
Generate JavaScript and Python encoding tables from source JSON/YAML.

Usage:
    python generate_encoding_tables.py
    
    Or add to Flask app for auto-generation on file change.
"""

import json
import os
from datetime import datetime
from pathlib import Path

def load_source_file(filepath):
    """Load encoding values from JSON or YAML."""
    with open(filepath, 'r') as f:
        if filepath.endswith('.yaml') or filepath.endswith('.yml'):
            import yaml
            return yaml.safe_load(f)
        else:
            return json.load(f)

def validate_table(table_name, table_data):
    """Validate a single table for errors."""
    errors = []
    max_value = table_data['max_value']
    values = table_data['values']
    
    used_values = set()
    used_names = set()
    
    for item in values:
        name = item['name']
        value = item['value']
        
        # Check for duplicate values
        if value in used_values:
            errors.append(f"{table_name}: Duplicate value {value}")
        used_values.add(value)
        
        # Check for duplicate names
        if name in used_names:
            errors.append(f"{table_name}: Duplicate name '{name}'")
        used_names.add(name)
        
        # Check value range
        if value < 0 or value > max_value:
            errors.append(f"{table_name}: Value {value} out of range (0-{max_value})")
        
        # Validate naming convention (uppercase with underscores)
        if not name.isupper() or not name.replace('_', '').isalpha():
            errors.append(f"{table_name}: Invalid name format '{name}' (use UPPERCASE_WITH_UNDERSCORES)")
    
    # Report available slots
    available = sorted(set(range(max_value + 1)) - used_values)
    info = f"{table_name}: {len(used_values)}/{max_value + 1} slots used, {len(available)} available"
    if available and len(available) <= 10:
        info += f" {available}"
    
    return errors, info

def generate_javascript(data, output_path):
    """Generate JavaScript constants file."""
    lines = [
        "// AUTO-GENERATED FILE - DO NOT EDIT MANUALLY",
        "// Generated from: app/data/encoding_values.json",
        f"// Generated on: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}",
        "//",
        "// To add new values:",
        "//   1. Edit app/data/encoding_values.json",
        "//   2. Run: python app/generate_encoding_tables.py",
        "//   3. Restart Flask server (or it auto-reloads)",
        "",
        "// ============================================================================",
        "// ENCODING CONSTANTS",
        "// ============================================================================",
        "",
        f"const ENCODING_VERSION = {data['metadata']['version']};",
        ""
    ]
    
    for table_name, table_data in data['tables'].items():
        const_name = table_name.upper()
        
        # Generate forward mapping
        lines.append(f"// {const_name} (max: {table_data['max_value']}, bits: {table_data['bit_size']})")
        lines.append(f"const {const_name} = {{")
        for item in table_data['values']:
            lines.append(f"    '{item['name']}': {item['value']},")
        lines.append("};")
        lines.append("")
        
        # Generate reverse mapping
        lines.append(f"const {const_name}_REVERSE = {{")
        for item in table_data['values']:
            lines.append(f"    {item['value']}: '{item['name']}',")
        lines.append("};")
        lines.append("")
        
        # Generate display mapping
        lines.append(f"const {const_name}_DISPLAY = {{")
        for item in table_data['values']:
            lines.append(f"    '{item['name']}': '{item['display']}',")
        lines.append("};")
        lines.append("")
    
    # Write file
    with open(output_path, 'w') as f:
        f.write('\n'.join(lines))
    
    print(f"✓ Generated: {output_path}")

def generate_python(data, output_path):
    """Generate Python constants file (optional)."""
    lines = [
        '"""',
        "AUTO-GENERATED FILE - DO NOT EDIT MANUALLY",
        "Generated from: app/data/encoding_values.json",
        f"Generated on: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}",
        '"""',
        "",
        f"ENCODING_VERSION = {data['metadata']['version']}",
        ""
    ]
    
    for table_name, table_data in data['tables'].items():
        const_name = table_name.upper()
        
        lines.append(f"# {const_name} (max: {table_data['max_value']}, bits: {table_data['bit_size']})")
        lines.append(f"{const_name} = {{")
        for item in table_data['values']:
            lines.append(f"    '{item['name']}': {item['value']},")
        lines.append("}")
        lines.append("")
        
        lines.append(f"{const_name}_REVERSE = {{")
        for item in table_data['values']:
            lines.append(f"    {item['value']}: '{item['name']}',")
        lines.append("}")
        lines.append("")
    
    with open(output_path, 'w') as f:
        f.write('\n'.join(lines))
    
    print(f"✓ Generated: {output_path}")

def generate_documentation(data, output_path):
    """Generate markdown documentation."""
    lines = [
        "# Encoding Values Reference",
        "",
        f"**Version:** {data['metadata']['version']}  ",
        f"**Generated:** {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}  ",
        "",
        "## Overview",
        "",
        "This document lists all currently defined encoding values.",
        "",
        "## Quick Reference",
        ""
    ]
    
    for table_name, table_data in data['tables'].items():
        lines.append(f"### {table_name.replace('_', ' ').title()}")
        lines.append("")
        lines.append(f"**Capacity:** {len(table_data['values'])}/{table_data['max_value'] + 1} slots used  ")
        lines.append(f"**Bit Size:** {table_data['bit_size']} bits  ")
        lines.append("")
        lines.append("| Value | Name | Display |")
        lines.append("|-------|------|---------|")
        
        for item in sorted(table_data['values'], key=lambda x: x['value']):
            lines.append(f"| {item['value']} | `{item['name']}` | {item['display']} |")
        
        lines.append("")
    
    with open(output_path, 'w') as f:
        f.write('\n'.join(lines))
    
    print(f"✓ Generated: {output_path}")

def main():
    """Main generation function."""
    base_dir = Path(__file__).parent
    source_file = base_dir / 'data' / 'encoding_values.json'
    
    # Load source data
    print(f"Loading: {source_file}")
    data = load_source_file(source_file)
    
    # Validate all tables
    print("\nValidating tables...")
    all_errors = []
    for table_name, table_data in data['tables'].items():
        errors, info = validate_table(table_name, table_data)
        print(f"  {info}")
        all_errors.extend(errors)
    
    if all_errors:
        print("\n❌ VALIDATION ERRORS:")
        for error in all_errors:
            print(f"  - {error}")
        return 1
    
    print("\n✓ All tables valid")
    
    # Generate files
    print("\nGenerating files...")
    generate_javascript(data, base_dir / 'js' / 'encoding_tables.js')
    generate_python(data, base_dir / 'encoding_tables_constants.py')
    generate_documentation(data, base_dir.parent / 'planning' / 'ENCODING_VALUES_REFERENCE.md')
    
    print("\n✓ Generation complete!")
    return 0

if __name__ == '__main__':
    exit(main())

Flask Integration (Auto-Generation)

Add to app.py:

import os
from datetime import datetime

@app.before_request
def check_encoding_tables():
    """Auto-regenerate encoding tables if source has changed."""
    source_file = 'data/encoding_values.json'
    output_file = 'js/encoding_tables.js'
    
    # Check if regeneration needed
    if not os.path.exists(output_file):
        print("Encoding tables not found, generating...")
        os.system('python generate_encoding_tables.py')
    else:
        source_mtime = os.path.getmtime(source_file)
        output_mtime = os.path.getmtime(output_file)
        
        if source_mtime > output_mtime:
            print("Encoding values changed, regenerating tables...")
            os.system('python generate_encoding_tables.py')

Benefits

For Non-Technical Users

• ✅ Edit simple JSON file (no JavaScript knowledge needed)
• ✅ Clear structure with examples
• ✅ Automatic validation catches errors
• ✅ Can't create mismatches between forward/reverse

For Developers

• ✅ Single source of truth
• ✅ Auto-generated documentation
• ✅ Validation prevents bugs
• ✅ Easy to add new tables
• ✅ Can generate for multiple targets (JS, Python, docs)

For Maintenance

• ✅ Version control friendly (one file to track)
• ✅ Easy to review changes (just JSON diff)
• ✅ Reports available slots automatically
• ✅ Enforces naming conventions

Migration Path

Phase 1: Create Source File

1. Extract existing values to encoding_values.json
2. Test generator script
3. Verify output matches current encoding_helper.js

Phase 2: Integrate Generator

1. Add generator script to project
2. Update build process / Flask app
3. Test auto-generation

Phase 3: Switch to Generated Tables

1. Update encoding_helper.js to import generated file
2. Remove manual mappings
3. Update documentation

Phase 4: User Documentation

1. Create guide for adding new values
2. Document validation rules
3. Add examples

Future Enhancements

• Web UI for editing values (no need to edit JSON directly)
• Export to CSV for client review
• Import from Excel/CSV
• Visual capacity indicators
• Conflict detection across tables
• Automatic value assignment (find next available slot)

Related Files

• ENCODING_SYSTEM.md - Technical specification
• app/js/encoding_helper.js - Current manual implementation
• app/js/encoding_tables.js - Future generated file

---

Status: Planning
Priority: Medium
Effort: ~4-6 hours implementation + testing
Dependencies: None