> ## Documentation Index
> Fetch the complete documentation index at: https://docs.kubox.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Processing One Trillion Rows

> Compute the minimum, maximum and mean temperatures per weather station, sorted alphabetically for one trillion rows.

The example is inspired from the [One Trillion Row Challenge](https://docs.coiled.io/blog/1trc.html) by Coiled - the company behind popular distributed pythonic open-source library [Dask](https://www.dask.org).

<Tip> At its core, Kubox offers a freemium CLI that empowers you to effortlessly spin up Kubernetes clusters in your own cloud. You have the flexibility to deploy your own data processing frameworkds and tools. </Tip>

> The source code is available in our Github repository at [https://github.com/kubox-ai/1trc](https://github.com/kubox-ai/1trc)

This repository is a **work in progress** as we iterate and learn before the final submission. It contains the code to quickly spin up a Kubox cluster in AWS `us-east-1` and process 1 trillion rows of data. It give you **two options** for tackling this challenge:

1. [ClickHouse](https://clickhouse.com) – A powerful, high-performance analytics database.
2. [Daft](https://www.getdaft.io) and [Ray](https://www.ray.io) – A dynamic duo for distributed computing and cutting-edge data processing.

We’re excited to have you explore and experiment with Kubox. Feel free to dive in and share your feedback as we continue to enhance this project!

<Note>USD prices for AWS EC2 spot instances are used for the below calculations. The prices are subject to change based on the current market rates.</Note>

| Metric/Framework | Daft + Ray | Clickhouse |
| ---------------- | ---------- | ---------- |
| Startup time     | 320s       | 313s       |
| Running time     | 1189s      | 527s       |
| Delete time      | 122s       | 123s       |
| Estimate cost    | \$2.75     | \$1.37     |

Work in progress..

## Dataset

The [One Trillion Row Challenge](https://docs.coiled.io/blog/1trc.html) originated as an ambitious benchmark task:

* **Goal**: Compute the minimum, mean, and maximum temperatures per weather station, sorted alphabetically.
* **Dataset**:
  * Format: Parquet
  * Size: 2.5 TB (100,000 files, each `24 MiB` in size with 10 million rows)
  * Location: `s3://coiled-datasets-rp/1trc` (AWS S3 Requester Pays Bucket)

Here is how to download one of the `24 MiB` file:

```bash theme={null}
aws s3 cp s3://coiled-datasets-rp/1trc/measurements-0.parquet . --request-payer requester
```