comiiit

2026-06-11 15:25:40 +02:00
parent 95d1839f05
commit 3fa11ef0f6
50 changed files with 9336 additions and 33 deletions
--- a/DATABASES.en.md
+++ b/DATABASES.en.md
@@ -0,0 +1,323 @@
+# 🗄️ Database Architecture
+
+## Database Classification
+
+### Relational (SQL)
+
+| DB | License | Use Case | Details |
+|----|---------|----------|--------|
+| **PostgreSQL** | Open source | Universal, geospatial, analytics, AI | [POSTGRESQL.md](POSTGRESQL.md) |
+| **MySQL / MariaDB** | Open source | Web, LAMP stack, e-commerce | [MYSQL.md](MYSQL.md) |
+| **Microsoft SQL Server** | Proprietary | Enterprise .NET, Windows ecosystem | — |
+| **Oracle DB** | Proprietary | Enterprise, finance, mainframe, RAC cluster | [ORACLE.md](ORACLE.md) |
+| **Amazon Aurora** | Managed | MySQL/PostgreSQL compatible, cloud-native | — |
+
+### NoSQL
+
+| Type | DB | Use Case | Details |
+|-----|----|----------|--------|
+| **Document** | MongoDB, Couchbase | JSON data, flexible schema | [MONGODB.md](MONGODB.md) |
+| **Key-Value / Cache** | Redis, Memcached, DynamoDB | Cache, session store, real-time | [REDIS.md](REDIS.md) |
+| **Wide-column** | Cassandra, ScyllaDB | Time-series, IoT, big data | [CASSANDRA.md](CASSANDRA.md) |
+| **Vector** | Pinecone, Qdrant, Milvus, pgvector | Embeddings, RAG, semantic search | [VEKTOROVE-DB.md](VEKTOROVE-DB.md) |
+| **Graph** | Neo4j, Dgraph | Relationships, recommendations, social graphs | — |
+
+### Storage Engines
+
+Common concepts across databases: [DATABASE-ENGINES.en.md](DATABASE-ENGINES.en.md)
+
+---
+
+## Transaction Isolation Levels
+
+| Level | Dirty Read | Non-repeatable Read | Phantom Read | Serialization Anomaly |
+|--------|-----------|---------------------|-------------|----------------------|
+| **Read Uncommitted** | Yes (possible) | Yes | Yes | Yes |
+| **Read Committed** | No (prevented) | Yes | Yes | Yes |
+| **Repeatable Read** | No | No | No (PostgreSQL: No) | Yes |
+| **Serializable** | No | No | No | No |
+
+**Anomalies**:
+- **Dirty Read** — reading data from an uncommitted transaction (data may be rolled back)
+- **Non-repeatable Read** — same query returns different data (another transaction updated the row in the meantime)
+- **Phantom Read** — same query returns new rows (another transaction inserted data matching the condition)
+- **Serialization Anomaly** — the result of transactions is not equivalent to any serial order
+
+### PostgreSQL vs MySQL Differences
+
+- **PostgreSQL**: Read Uncommitted behaves like Read Committed. Repeatable Read = Snapshot Isolation (also prevents phantom reads). Serializable = SSI.
+- **MySQL InnoDB**: Repeatable Read uses next-key locking (prevents phantom reads).
+
+---
+
+## CAP Theorem
+
+In a distributed system, only 2 out of 3 are possible: **C**onsistency, **A**vailability, **P**artition tolerance.
+
+In practice: P is always required, we choose between CP (consistency) and AP (availability).
+
+### PACELC Extension
+
+PACELC extends CAP with behavior under normal conditions (no partition):
+- **P**artition → **A**vailability vs **C**onsistency
+- **E**lse (no partition) → **L**atency vs **C**onsistency
+
+| DB | Partition Choice | Else Choice |
+|----|----------------|------------|
+| Cassandra | AP (availability) | LC (low latency, eventual consistency) |
+| DynamoDB (default) | AP | LC |
+| MongoDB | CP (primary) | LC |
+| PostgreSQL (single) | CP | CC |
+| CockroachDB | CP | CC |
+
+### Quorum Details
+
+- **R** (read quorum) + **W** (write quorum) > **N** (replication factor)
+- Typical: N=3, R=2, W=2 (tolerates 1 node down)
+- **Sloppy quorum** — when a node is unavailable, data is temporarily stored on another node
+- **Hinted handoff** — temporary write to another node with a hint, data is transferred upon recovery
+
+---
+
+## Replication
+
+| Type | Description | Latency |
+|-----|-------|---------|
+| Synchronous | Write confirmed only after replication to all nodes | High, but consistent |
+| Asynchronous | Write confirmed immediately, replication in the background | Low, possible data loss |
+| Semi-synchronous | Confirmation from majority of nodes | Compromise |
+
+### Topologies
+
+- **Leader-Follower** (Master-Slave) — reads from replicas
+- **Leader-Leader** (Multi-master) — writes to multiple nodes
+- **Quorum-based** — R + W > N (Cassandra, DynamoDB)
+
+---
+
+## Sharding
+
+Data distribution across nodes based on a shard key.
+
+```
+                     ┌─────────┐
+                     │  Proxy  │
+                     │  Router │
+                     └────┬────┘
+               ┌──────────┼──────────┐
+          ┌────▼───┐ ┌───▼────┐ ┌───▼────┐
+          │Shard A │ │Shard B │ │Shard C │
+          │ 0-100  │ │101-200 │ │201-300 │
+          └────────┘ └────────┘ └────────┘
+```
+
+### Methods
+
+| Method | Description | Advantage | Disadvantage |
+|--------|-------|--------|----------|
+| **Hash-based** | `shard_id = hash(key) % N` | Even distribution | Loss of range queries |
+| **Range-based** | Data by range (A-M, N-Z) | Preserves ordering | Hot spots |
+| **Consistent hashing** | Hash ring, vnodes | Min. rebalancing when number of shards changes | More complex |
+
+### Routing
+
+- **Proxy-based** — application goes to proxy, which routes (Vitess, ProxySQL, mongos)
+- **Client-side** — application knows which shard to target
+- **DNS-based** — each shard has its own endpoint
+
+---
+
+## Data Consistency Patterns
+
+| Pattern | Description | Example |
+|---------|-------|---------|
+| **Strong consistency** | After a write, every read sees the latest data | Single DB, Raft, Spanner |
+| **Eventual consistency** | After a write, data propagates over time | DNS, DynamoDB (default), Cassandra |
+| **Read-after-write** | The author always sees their own write (others are eventual) | Social networks, comments |
+| **Causal consistency** | Causally dependent operations are seen in the correct order | COPS, Orbe, MongoDB (causal clusters) |
+| **Monotonic reads** | You do not see older data after seeing newer data | Cassandra (MONOTONIC_READ consistency) |
+| **Monotonic writes** | Writes from a single client are in order | Queue-based, single leader |
+
+---
+
+## Data Migration
+
+### Schema Migration
+
+```
+V1__initial_schema.sql
+V2__add_users_table.sql
+V3__add_email_index.sql
+V4__add_orders_table.sql
+```
+
+### Zero-Downtime Migration
+
+1. **Expand** — add new column/table (application tolerates both states)
+2. **Migrate** — backfill data, update application to new schema
+3. **Contract** — remove old column/table
+
+### Tools
+
+| Tool | Language | Strategy | Zero-Downtime | Rollback |
+|---------|-------|-----------|--------------|----------|
+| **Flyway** | Java (multi-lang CLI) | Versioned SQL | Limited (additive only) | `undo` (limited, enterprise) |
+| **Liquibase** | Java (multi-lang CLI) | Changesets (XML/YAML/JSON/SQL) | Yes (changeset design) | `rollback <count>` |
+| **Alembic** | Python | Auto-generation, versioned | Yes (branching) | `downgrade` |
+| **Prisma Migrate** | TypeScript | Declarative schema → diff | Yes (shadow DB) | `migrate diff` |
+| **gh-ost** | Go | Triggerless online DDL (MySQL) | Yes (binlog stream) | No (progressive) |
+| **pgroll** | Go | Online schema migration (PG) | Yes (views, multiple versions) | Yes (immediate) |
+
+---
+
+## SQL Antipatterns
+
+Based on *More SQL Antipatterns* (Karwin, 2026) — 14 new antipatterns:
+
+### Language Antipatterns
+
+| Antipattern | Problem | Solution |
+|-------------|---------|--------|
+| **Fear of JOINs** | Manual pairing in application instead of JOIN | Use JOIN correctly |
+| **Relational Division** | Finding sets in WHERE | Relational division (subquery with GROUP BY/HAVING) |
+| **Pagination via OFFSET** | OFFSET is O(n) — the larger the offset, the slower | Keyset pagination (WHERE id > last_seen) |
+| **Non-Sargable queries** | Functions on columns in WHERE (`WHERE YEAR(date) = 2026`) | Rewrite as range condition |
+
+### Optimization Antipatterns
+
+| Antipattern | Problem | Solution |
+|-------------|---------|--------|
+| **Premature denormalization** | Denormalization without reason | Measure, then optimize |
+| **JSON overuse** | JSON as a universal solution | Use JSON only for genuinely flexible data |
+| **Cacheless transactions** | Relying on query cache (removed in MySQL 8) | Application-level caching |
+
+### Application Antipatterns
+
+| Antipattern | Problem | Solution |
+|-------------|---------|--------|
+| **Polling** | Regularly querying for changes | LISTEN/NOTIFY, Kafka, Change Data Capture |
+| **Transaction encapsulation** | Each model manages its own transaction | Unit of Work pattern |
+| **Fear of deadlocks** | Trying to prevent all deadlocks | Mitigation, not prevention |
+| **Data hoarding** | Storing everything forever | Data retention policies, archiving |
+
+### Mini-Antipatterns
+
+- `LIMIT` without `ORDER BY` — nondeterministic results
+- `NATURAL JOIN` — fragile, implicit join condition
+- `N+1 queries` — query in a loop instead of JOIN/batch
+- Redundant indexes — duplicate/overlapping indexes unnecessarily slow writes
+
+---
+
+## Designing Data-Intensive Applications (2nd Edition)
+
+*Kleppmann, Riccomini (2026)* — substantially revised edition.
+
+### What's New Compared to 1st Edition
+
+| Area | What's New |
+|--------|-----------|
+| **Cloud-native** | Storage = object store (S3, Blob), not local disk. Separation of control/data/compute plane |
+| **AI workloads** | Vector indexes, DataFrames as a data model, batch processing for training data |
+| **Local-first software** | DuckDB, PGlite, SQLite — databases running on laptop/edge, sync when connected |
+| **Formal methods** | Randomized testing, formal verification (important for AI-generated code) |
+| **Legal & ethics** | GDPR, ethics of predictive analytics, bias, algorithmic accountability |
+| **Streaming → SQL views** | Materialize, incremental view maintenance — streaming as SQL |
+
+### Key Principles (unchanged)
+
+**Reliability**, **Scalability**, **Maintainability** — the three pillars of good data systems.
+
+---
+
+## Apache Iceberg Lakehouse
+
+Based on *Architecting an Apache Iceberg Lakehouse* (Merced, 2026):
+
+### What is a Data Lakehouse
+
+An architecture combining the flexibility and low cost of a **data lake** (object storage) with the performance and governance of a **data warehouse**. Apache Iceberg is an open source table format.
+
+### Iceberg Metadata Architecture
+
+```
+Table metadata (.metadata.json)
+    └── Snapshot manifest list
+            └── Manifests (file-level stats)
+                    └── Data files (Parquet/ORC/Avro)
+```
+
+### Key Features
+
+| Feature | Description |
+|-----------|-------|
+| **ACID transactions** | Safe concurrent read/write |
+| **Schema evolution** | Add/drop/rename columns without rewrite |
+| **Time travel** | Query historical snapshots |
+| **Partition evolution** | Change partition strategy without data rewrite |
+| **Hidden partitioning** | Automatic partition filters (user does not need to specify) |
+| **Multi-engine** | Spark, Flink, Trino, Dremio, Snowflake over the same data |
+
+### When to Use Iceberg
+
+- Multi-tool access to the same governed data
+- ACID on lake data
+- Streaming + batch in a single table
+- Reducing duplication (one canonical copy instead of ETL to warehouse)
+
+---
+
+## Best Practices
+
+- **Connection pooling** — PgBouncer, RDS Proxy, ProxySQL
+- **Indexing based on query patterns** — do not have unnecessary indexes
+- **Read replicas** for reporting and analytics
+- **Backup & recovery** — point-in-time recovery (PITR), regular tests
+- **Query monitoring** — slow query log, pg_stat_statements, performance_schema
+- **Encryption at rest & in transit**
+- **Migrations in CI/CD** — part of the pipeline, not manual
+- **Choose DB based on workload** — no single universal DB (polyglot persistence)
+
+---
+
+## Database License Model Comparison
+
+| DB | License | Price (self-hosted) | Price (managed cloud) | Vendor lock-in | Note |
+|----|---------|-------------------|---------------------|----------------|----------|
+| **PostgreSQL** | PostgreSQL license (MIT-like) | $0 | ~$0.10-1.00/hr (RDS, CloudSQL, Aurora) | Low | Fully open source, no restrictions |
+| **MySQL** | GPL v2 / Commercial (Oracle) | $0 (GPL) / ~$2,000/server/year (commercial) | ~$0.10-1.00/hr (RDS, PlanetScale) | Medium (Oracle owned) | GPL = need to release application? (depends on distribution) |
+| **MariaDB** | GPL v2 / Business Source | $0 (GPL) | ~$0.10-1.00/hr (SkySQL) | Low | Fully compatible MySQL fork, no Oracle influence |
+| **Oracle SE2** | Proprietary (per core) | ~$17,500/core + 22% support/year | ~$1-5/hr (RDS, OCI) | High | Core factor 0.5 (EPYC/Xeon), max 16 threads |
+| **Oracle EE** | Proprietary (per core + options) | ~$47,500/core + options + 22% support | ~$2-30/hr (OCI, RDS) | High | Options double the price (RAC, partitioning, compression) |
+| **SQL Server Standard** | Proprietary (per core + CAL) | ~$1,000/core + $200/CAL | ~$0.20-1.00/hr (Azure SQL) | Medium | Windows Server license required additionally |
+| **SQL Server Enterprise** | Proprietary (per core + CAL) | ~$7,000/core + $200/CAL | ~$1-5/hr (Azure SQL) | Medium | AlwaysOn, partitioning, in-memory OLTP |
+| **MongoDB** | SSPL (Community) / Commercial (Enterprise) | $0 (Community) / ~$10k/server/year (Enterprise) | ~$0.10-5.00/hr (Atlas) | Medium | SSPL restricts managed cloud services |
+| **Redis** | RSALv2 + SSPL (7.4+) / BSD (Valkey) | $0 (Valkey) | ~$0.10-1.00/hr (ElastiCache, Memorystore → Valkey) | Low (Valkey) | Redis 7.4+ license change → Valkey fork |
+| **Cassandra** | Apache 2.0 | $0 | ~$0.10-1.00/hr (Keyspaces, Amazon Managed) | Low | Fully open source, no restrictions |
+| **ScyllaDB** | Apache 2.0 (OSS) / Enterprise | $0 (OSS) / Enterprise subscription | ~$0.50-3.00/hr (ScyllaDB Cloud) | Low (OSS) | Enterprise: monitoring, security, support |
+| **CockroachDB** | BSL (Business Source License) / Enterprise | $0 (core) / Enterprise subscription | ~$0.50-3.00/hr (CockroachDB Cloud) | Medium | BSL: converts to MIT after 3 years. Enterprise: multi-region, backup |
+
+**Key Recommendations**:
+- **Lowest TCO**: PostgreSQL (no license, broadest cloud support)
+- **Highest vendor lock-in**: Oracle (PL/SQL, proprietary options, expensive migration)
+- **License risk**: Redis (license change) → use Valkey for new projects
+- **Cloud-native licensing**: MongoDB Atlas, CockroachDB Cloud, ScyllaDB Cloud — pay-per-use, no license management
+
+## Resources
+
+Links, books and standards: [sources/databases/sources.md](sources/databases/sources.md)
+
+### Recommended Reading
+
+| Book | Authors | ISBN | Key Takeaway |
+|-------|--------|------|----------------|
+| Database Internals | Alex Petrov | 978-1492040346 | In-depth explanation of storage engines (B-Tree, LSM-Tree, WAL, MVCC), distributed systems |
+| Designing Data-Intensive Applications (2nd ed.) | Kleppmann, Riccomini | — | Cloud-native, AI, local-first, formal methods |
+| High Performance MySQL (4th ed.) | Schwartz, Zaitsev, Tkachenko | 978-1492075292 | MySQL architecture, schema/index optimization |
+| Expert Oracle Architecture (3rd ed.) | Kyte, Kuhn | 978-1484249602 | Oracle architecture, RAC, Data Guard, tuning |
+| AI-Ready PostgreSQL 18 | Kumar, Linster | — | PostgreSQL as a unified platform for AI |
+| More SQL Antipatterns | Bill Karwin (2026) | — | 14 antipatterns, keyset pagination |
+| Vector Databases | Borwankar (2026) | — | Embeddings, vector indexes, RAG |
+| Architecting an Apache Iceberg Lakehouse | Merced (2026) | — | Lakehouse architecture, Iceberg metadata |
+
+*Last revision: 2026-06-03*