The 30+ vendor integrations that feed Schoolytics' data warehouse. Each is its own JSON-RPC service so vendor-specific concerns (auth quirks, rate limits, flaky pagination, API versioning) stay isolated.
Shape of an integration
Every integration is a Flask + Flask-JsonRPC service deployed to Cloud Run (or Cloud Functions for the older ones). It exposes vendor-specific methods covering whatever the vendor surfaces (rosters, grades, attendance, assignments, assessment results). All methods follow a shared signature: one customer id plus vendor-specific params, returning a dict.
At request time the RPC fetches the customer's credentials by customer id, instantiates the vendor client, and executes the sync.
Why RPCs instead of a monolith
- Isolation. Vendor-specific rate-limit logic, retry policy, and auth flow live next to the vendor they serve, a bug in one integration doesn't risk the others.
- Independent resource profiles. Some vendors' syncs want generous memory and long timeouts; others are tiny. Per-service tuning instead of one-size-fits-all.
- Blast radius. Deploys are scoped. Updating one integration doesn't ship the others.
Versioning in place
The larger integrations register multiple versions of the same method in their JSON-RPC dispatcher. Callers invoke the version they're compatible with. New implementations mature as the latest version while older ones remain callable, no breaking-migration required.
Async by default
Long-running syncs are enqueued to Cloud Tasks rather than held open on the RPC connection. The RPC returns immediately; the task queue handles retries with exponential backoff, and writes to BigQuery are batched via Pub/Sub to avoid throughput throttles. Decouples API latency from actual sync time.
Shared substrate
A shared Python package provides the Spanner ORM, encrypted-field implementations, Cloud Tasks publisher, and the BigQuery / Pub/Sub writer. A new integration doesn't bootstrap the platform each time, it pulls the shared lib and focuses on the vendor-specific surface.
Stack
- Python, Flask + Flask-JsonRPC
- Cloud Run (newer) and Cloud Functions (older) for deployment
- Cloud Spanner for customer metadata and encrypted credentials
- Cloud Tasks for async sync orchestration
- Cloud Pub/Sub + BigQuery for data egress
- Cloud Build / Docker for CI