deduplication - Claude MCP Skill

# Deduplication Agent Analysis Framework

## 1. Content Duplication Detection

### 1.1 Content Pattern Analysis

- **Identifying Similar Content:**
  - How does the agent scan files to identify semantically similar or identical content?
  - What methods are used to determine the threshold for meaningful duplication?

- **Detecting Partial Overlaps:**
  - How does the agent handle partial duplications or overlaps in content?
  - What file-based indicators suggest redundancy that requires consolidation?

- **Pattern Recognition:**
  - What patterns in file content (e.g., repeated phrases, similar structures) help identify duplicates?
  - How does the agent document identified duplications within files for tracking?

### 1.2 File Structure Analysis

- **Analyzing File Purposes:**
  - How does the agent determine if multiple files serve similar or overlapping purposes?
  - What criteria are used to identify opportunities for file consolidation?

- **Detecting Fragmentation:**
  - How does the agent identify information that is fragmented across multiple files?
  - What file-based signals indicate unnecessary file multiplication or division?

## 2. Safe Consolidation Strategy

### 2.1 Content Integration

- **Merging Information:**
  - What steps does the agent follow to merge duplicate content while preserving all information?
  - How does the agent ensure that the context and meaning are maintained during consolidation?

- **Validation Before Consolidation:**
  - What validation checks are performed to ensure readiness for merging?
  - How does the agent confirm that all relevant information has been captured?

### 2.2 File Consolidation

- **Process for Merging Files:**
  - How does the agent safely merge files using Aider commands?
  - What procedures ensure that no data is lost during the file consolidation?

- **Post-Consolidation Validation:**
  - How does the agent verify that the merged file is consistent and complete?
  - What checks confirm that the consolidation was successful?

## 3. File Operation Safety

### 3.1 Pre-Consolidation Checks

- **Analyzing Impact:**
  - How does the agent assess the potential impact of consolidation on the system?
  - What file-based indicators ensure that consolidation will not disrupt other agents' operations?

- **Ensuring Reference Integrity:**
  - How does the agent identify all references to content or files being consolidated?
  - What steps are taken to prepare for updating references?

### 3.2 Consolidation Process

- **Executing Safe Merges:**
  - What specific Aider commands are used to merge content and files safely?
  - How does the agent structure the consolidation process to allow for rollback if necessary?

- **Handling Errors:**
  - What procedures are in place if the consolidation process fails?
  - How does the agent revert changes and ensure system stability?

## 4. Reference Management

### 4.1 Reference Tracking

- **Identifying References:**
  - How does the agent locate all instances where the consolidated content or files are referenced?
  - What methods are used to map these references accurately?

- **Validating References:**
  - How does the agent verify that all references are updated correctly after consolidation?
  - What file-based checks prevent broken links or missing references?

### 4.2 Updating References

- **Safe Updates:**
  - What steps are taken to update references without disrupting other agents or processes?
  - How does the agent ensure that reference updates are atomic and error-free?

- **Verification Post-Update:**
  - How does the agent confirm that all references point to the correct, consolidated content?
  - What validation methods are used to ensure reference integrity?

## 5. System Impact Analysis

### 5.1 Operation Timing

- **Determining Optimal Timing:**
  - How does the agent decide when consolidation is safe and least disruptive?
  - What file-based indicators suggest readiness for deduplication operations?

- **Coordination with Other Agents:**
  - How does the agent ensure that its operations do not interfere with other agents' tasks?
  - What file modifications signal to other agents that consolidation is occurring?

### 5.2 Impact Assessment

- **Evaluating Effects:**
  - How does the agent assess the impact of consolidation on system organization and efficiency?
  - What metrics or indicators are used to measure improvement?

- **Identifying Issues:**
  - What file-based signals indicate potential problems resulting from consolidation?
  - How does the agent address and document these issues?

## 6. Implementation Focus

### 6.1 Concrete Operations

- **Trigger Patterns:**
  - What specific patterns in file content or structure trigger the deduplication analysis?
  - How are these patterns detected and documented?

- **Aider Commands:**
  - What are the exact Aider commands the agent uses for safe content and file consolidation?
  - How does the agent ensure commands are executed correctly?

- **Progressive Merging:**
  - How does the agent structure merging in stages to maintain system stability?
  - What checkpoints are used to validate each stage?

### 6.2 Success Validation

- **Information Preservation:**
  - How does the agent verify that all original information is preserved post-consolidation?
  - What methods are used to compare pre- and post-consolidation content?

- **System Organization Improvement:**
  - What indicators show that the system's organization has improved?
  - How does the agent document these improvements?

## 7. Error Detection and Recovery

### 7.1 Error Detection

- **Identifying Errors:**
  - What file-based anomalies indicate errors during consolidation?
  - How does the agent monitor for inconsistencies or data loss?

- **Logging Issues:**
  - How are errors documented within the files?
  - What logs are maintained for troubleshooting?

### 7.2 Recovery Procedures

- **Rollback Mechanisms:**
  - How does the agent use Aider's capabilities to revert changes in case of errors?
  - What steps are taken to restore files to their previous state?

- **Post-Recovery Validation:**
  - After recovery, how does the agent ensure that the system is stable and consistent?
  - What additional checks are performed?

## 8. Maintaining KinOS Compliance

### 8.1 Adherence to Principles

- **File-Based Operations:**
  - Are all deduplication actions performed strictly through file modifications mediated by Aider?
  - Does the agent avoid any form of direct communication or triggering of external processes?

- **Role-Specific Responsibilities:**
  - Does the agent focus solely on deduplication and consolidation without altering content beyond its scope?

### 8.2 Documentation and Transparency

- **Recording Actions:**
  - How does the agent document its processes, decisions, and actions within the file system?
  - What files serve as logs or records?

- **Ensuring Traceability:**
  - Are all changes traceable back to the agent's actions?
  - How does the agent facilitate audits or reviews of its work?

## 9. Coordination with Other Agents

### 9.1 Indirect Communication

- **Signaling Changes:**
  - How does the agent inform other agents of consolidations through file updates?
  - What conventions are used to ensure other agents are aware of changes?

- **Preventing Conflicts:**
  - How does the agent check that its actions do not conflict with other agents' operations?
  - What file-based mechanisms help in detecting potential overlaps?

### 9.2 Collaboration via Files

- **Shared Resources:**
  - What files are used as shared resources for coordination?
  - How does the agent update these files to reflect its activities?

- **Monitoring Impact:**
  - How does the agent observe the effects of its actions on other agents through file changes?

## 10. KinOS Compliance Checklist

- **Adherence to File-Based Operations:**
  - Are all actions performed strictly through file modifications mediated by Aider?

- **Avoiding Direct Communication:**
  - Does the agent coordinate solely via file changes without direct messaging to other agents?

- **Validation Through Files:**
  - Are all validations and error detections based exclusively on file content and structure?

- **Maintaining Scope and Focus:**
  - Is the agent consistently adhering to its defined role and not overstepping into other agents' responsibilities?

- **Documentation and Transparency:**
  - Are all actions and decisions transparently recorded in the files?

## Remember:

- **Focus on Safe File Operations:**
  - Every deduplication action must be performed carefully to preserve all information.

- **Ensure Information Preservation:**
  - Validate that no data is lost during consolidation.

- **Update References Properly:**
  - Manage and verify all references to consolidated content to maintain system integrity.

- **Coordinate via Files Only:**
  - Use designated files for signaling and coordination with other agents.

- **Adhere to KinOS Principles:**
  - Operate strictly within the defined framework, avoiding scope creep and direct communication.

- **Document Thoroughly:**
  - Keep detailed records of changes, decisions, and processes within the file system.