Overview
This repository provides a robust data pipeline for Polymarket, a decentralized prediction market. It automates the fetching, processing, and structuring of crucial trading data, including market details, order-filled events, and derived trade information. This system empowers users to gain deep insights into Polymarket's trading activity.
Key Features
- Comprehensive Data Collection: Gathers extensive market metadata and real-time order-filled events from the Goldsky subgraph.
- Automated Trade Structuring: Transforms raw order data into clean, analyzable trade records, including price, direction, and USD value.
- Resumable Operations: All pipeline stages intelligently resume from where they left off, ensuring data integrity and minimizing downtime during updates.
- Robust Error Handling: Implements automatic retries, rate limit management with exponential backoff, and graceful handling of server errors.
- Missing Market Discovery: Automatically identifies and fetches market data that may have been missed during initial collection, saving manual effort.
- Fast Package Management: Leverages
uvfor accelerated and reliable dependency installation. - Detailed Data Schemas: Clearly defines the structure and content of raw and processed data files for easy integration.
Typical Use Cases
- Algorithmic Trading: Develop and backtest trading strategies based on historical Polymarket price action and order flow.
- Market Analysis: Gain insights into market liquidity, trading volume, and the behavior of key participants.
- Data Science & Research: Utilize structured trade data for academic research, sentiment analysis, and prediction modeling.
- Portfolio Tracking: Reconstruct personal trading history and analyze performance on the Polymarket platform.
- Building Custom Dashboards: Create real-time or historical dashboards to visualize Polymarket trading dynamics.
