MonaNorouzi/GitHub-architecture-reverse-engineering

# Comprehensive Reverse Engineering of GitHub: Architecture, SCM, and Database Design ## Executive Summary This repository contains a high-level architectural and data-level reverse engineering of the GitHub platform, grounded in the principles of Software Configuration Management (SCM) and distributed systems. By utilizing both top-down (UI-to-architecture) and bottom-up (GraphQL-to-database) analytical methodologies, we expose the underlying structural paradigms of the platform. Key technical discoveries documented herein include: * **SCM Engine & Git DAG:** The fundamental version control mechanism operating as a Content-Addressable File System, relying on mathematical traversal of a Directed Acyclic Graph (DAG) comprised of `Blob`, `Tree`, and `Commit` objects. * **Architectural Evolution:** The transition from a monolithic Ruby on Rails core to an event-driven microservices ecosystem (e.g., GitHub Actions), heavily supported by a highly available caching layer (Redis/Memcached). * **Database Polymorphism:** The relational database implementation where `Pull Requests` and `Issues` utilize Single Table Inheritance concepts, verifiable via GitHub's GraphQL API structure. * **State Transitions:** The asynchronous, transactional database state changes that occur during a Pull Request lifecycle, orchestrated by background workers and external CI Webhooks. ## 1. Top-Down Discovery: Use Case & Behavioral Modeling The following diagram illustrates the primary system actors and the core operational processing logic during the Pull Request lifecycle. flowchart LR %% External Actors Dev([Developer]) Maintainer([Repository Maintainer]) CIBot([CI/CD Microservice]) %% System Boundary subgraph GitHub Core Platform UC1((Initialize Branch & Push)) UC2((Instantiate Pull Request)) UC3((Execute Integration Tests)) UC4((Conduct Code Review)) UC5((Execute Merge Transaction)) end %% Actor Relationships Dev --> UC1 Dev --> UC2 CIBot --> UC3 Maintainer --> UC4 Maintainer --> UC5 %% Internal Dependencies UC2 -. Async Trigger .-> UC3 UC3 -. CI Status Payload .-> UC5 ## 2. Bottom-Up Discovery: Polymorphic Data Architecture By reverse-engineering the GraphQL API endpoints, we map the underlying relational database structure. Notice the polymorphic relationship where a Pull Request acts as a specialized extension of an Issue. erDiagram USER ||--o{ REPOSITORY : owns USER ||--o{ ISSUE : authors REPOSITORY ||--o{ ISSUE : contains REPOSITORY ||--o{ COMMIT : stores ISSUE ||--o| PULL_REQUEST : "polymorphic extension" PULL_REQUEST ||--o{ COMMIT : encapsulates USER { int id PK string username string email } REPOSITORY { int id PK int owner_id FK string repository_name } ISSUE { int id PK int repository_id FK int author_id FK string lifecycle_state "Open / Closed" } PULL_REQUEST { int issue_id PK, FK string head_branch_ref string base_branch_ref boolean mergeable_status } COMMIT { string sha_hash PK int repository_id FK string tree_sha_pointer } ## 3. SCM Processing Logic: Pull Request State Transitions This Sequence Diagram meticulously traces the database state transitions and the orchestration between the Rails Monolith, Background Workers (Mergeability checks), and GitHub Actions (CI). sequenceDiagram actor Dev as Developer participant API as API Gateway (Routing) participant Rails as Rails Monolith participant DB as Relational DB (MySQL) participant Worker as Async Job (Diffing) participant CI as GitHub Actions actor Maint as Maintainer Dev->>API: POST /pulls (Create PR) API->>Rails: Route Request Rails->>DB: INSERT Issue (State: Open) Rails->>DB: INSERT PR (Mergeable: NULL) Rails-->>API: 201 Created Response Rails->>Worker: Enqueue Background Mergeability Check Rails->>CI: Dispatch Webhook (Event: pull_request) Worker->>Worker: Traverse Git DAG & Calculate Diff Worker->>DB: UPDATE PR (Mergeable: True) CI->>CI: Execute CI/CD Pipeline CI->>Rails: PATCH /status (Payload: Success) Rails->>DB: UPDATE Commit Status (Status: Success) Maint->>API: POST /merge (Accept PR) API->>Rails: Route Merge Request Rails->>Worker: Execute Low-level `git merge` Worker-->>Rails: Merge Confirmation Rails->>DB: UPDATE Issue (State: Closed) Rails->>DB: UPDATE PR (State: Merged) Rails-->>API: Return Finalized State UI Render ## 4. High-Level System Component Architecture The physical deployment and interaction of GitHub's distributed components, highlighting the centralized Rails monolith, caching optimization layers, and decoupled microservices. flowchart TB %% External Interfaces Client[Client Browser / Git CLI] %% Network Routing LB[Load Balancers / HAProxy] %% Transient Data / Optimization subgraph Caching Layer Redis[(Redis / Memcached Clusters)] end %% Core Application subgraph Monolithic Core Routing[API Gateway / Router] Rails[Ruby on Rails Application] Routing --> Rails end %% Decoupled Services subgraph Event-Driven Architecture Kafka[Kafka Message Broker] Actions[GitHub Actions K8s Cluster] Kafka --> Actions end %% Persistent Storage subgraph Persistence Layer DB[(MySQL / Vitess Shards)] GitFS[(Spokes / Git File Servers)] end %% Component Interconnections Client --> LB LB --> Routing Routing --> Redis Rails --> DB Rails --> GitFS Rails --> Kafka Actions -. Status Callback .-> Routing