Sign up
All challenges
Most candidates spend 6–12 hours; take what you need.

Data Pipeline Resilience

Build a scraper or ingestor for a public Indian market data source that keeps working when the source changes.

The problem

Our exchange and data-vendor scrapers break in ways that are boring but expensive: an endpoint gets retired without notice, a response schema quietly adds or renames a field, a rate limit kicks in only under load, or an upstream ID gets reassigned to a different entity.

When this happens today, the failure is often silent — the pipeline "succeeds" but writes wrong, stale, or partial data, and nobody notices until a user does.

What to deliver

  • A scraper/ingestor for any public Indian market data source of your choice (an exchange, a regulator, a data aggregator, a news source — your call), covering discovery, extraction and normalized storage.
  • A concrete strategy for detecting when the source has changed shape or gone stale — not just try/except around a request.
  • A retry/backoff and idempotency design so re-running the job never double-writes or corrupts state.
  • A short write-up: what would page you at 3am, and why.

What a good submission looks like

  • Treats "the source changed" as the expected case, not the exception.
  • Has an explicit contract for what "success" means for a single run (rows written, freshness, schema match) and validates against it.
  • Shows restraint — a small, correct pipeline beats a large, fragile one.

We're evaluating

Failure handlingIdempotencyMonitoring & alertingSchema drift tolerance

How to submit

  1. Build your solution, using any tools or stack you prefer.
  2. Write up your thinking — decisions made, tradeoffs, what you'd do with more time.
  3. Zip your code + write-up, name it yourname-pipeline-resilience.zip.
  4. Email it to us — the button below pre-fills the subject and a template.