Practice Now
CSV loading
Synchronous processing
Data handling
HTTP requests
Backend development

How to synchronously load a csv file into memory before handling HTTP requests

System Design practice on Codemia

Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.

Practice system design
Networking & APIs

Introduction

Many backend applications need reference data — lookup tables, configuration, pricing rules — loaded from CSV files before the server can handle requests. If you start accepting HTTP requests before the data is ready, you risk returning incomplete or empty responses. The solution is to synchronously load the CSV into memory during application startup, before binding the HTTP server to a port.

Node.js: Synchronous CSV Loading

Using fs.readFileSync with Manual Parsing

javascript
1const fs = require('fs');
2const http = require('http');
3
4// Load CSV synchronously BEFORE creating the server
5const csvData = fs.readFileSync('products.csv', 'utf-8');
6
7const products = csvData
8  .split('\n')
9  .slice(1) // skip header row
10  .filter(line => line.trim())
11  .map(line => {
12    const [id, name, price, category] = line.split(',');
13    return { id, name, price: parseFloat(price), category };
14  });
15
16console.log(`Loaded ${products.length} products`);
17
18// Server starts AFTER data is loaded
19const server = http.createServer((req, res) => {
20  if (req.url === '/products') {
21    res.writeHead(200, { 'Content-Type': 'application/json' });
22    res.end(JSON.stringify(products));
23  }
24});
25
26server.listen(3000, () => console.log('Server ready'));

Using a CSV Library (csv-parse)

For complex CSVs with quoted fields, newlines in values, or custom delimiters:

javascript
1const { parse } = require('csv-parse/sync');
2const fs = require('fs');
3
4const input = fs.readFileSync('data.csv', 'utf-8');
5const records = parse(input, {
6  columns: true,       // use first row as keys
7  skip_empty_lines: true,
8  cast: true           // auto-convert numbers and booleans
9});

Python: Loading CSV Before Flask/FastAPI

Flask Example

python
1import csv
2from flask import Flask, jsonify
3
4# Load data BEFORE creating the app
5data = []
6with open('products.csv', newline='') as f:
7    reader = csv.DictReader(f)
8    data = list(reader)
9
10print(f"Loaded {len(data)} records")
11
12app = Flask(__name__)
13
14@app.route('/products')
15def get_products():
16    return jsonify(data)

FastAPI with Startup Event

python
1import csv
2from fastapi import FastAPI
3
4app = FastAPI()
5products = []
6
7@app.on_event("startup")
8def load_csv():
9    global products
10    with open('products.csv', newline='') as f:
11        products = list(csv.DictReader(f))
12    print(f"Loaded {len(products)} products")
13
14@app.get("/products")
15def get_products():
16    return products

Pandas for Large CSVs

python
1import pandas as pd
2
3# Pandas handles encoding, dtypes, and large files efficiently
4df = pd.read_csv('products.csv', dtype={'id': str, 'price': float})
5products = df.to_dict(orient='records')

Go: Loading CSV at Init

go
1package main
2
3import (
4    "encoding/csv"
5    "log"
6    "net/http"
7    "os"
8    "encoding/json"
9)
10
11type Product struct {
12    ID    string `json:"id"`
13    Name  string `json:"name"`
14    Price string `json:"price"`
15}
16
17var products []Product
18
19func init() {
20    f, err := os.Open("products.csv")
21    if err != nil {
22        log.Fatal("Cannot open CSV:", err)
23    }
24    defer f.Close()
25
26    reader := csv.NewReader(f)
27    records, err := reader.ReadAll()
28    if err != nil {
29        log.Fatal("Cannot parse CSV:", err)
30    }
31
32    for _, r := range records[1:] { // skip header
33        products = append(products, Product{ID: r[0], Name: r[1], Price: r[2]})
34    }
35    log.Printf("Loaded %d products", len(products))
36}
37
38func main() {
39    http.HandleFunc("/products", func(w http.ResponseWriter, r *http.Request) {
40        json.NewEncoder(w).Encode(products)
41    })
42    log.Fatal(http.ListenAndServe(":8080", nil))
43}

Handling Large Files

For CSVs that are too large for memory:

  • Data Chunking: Divide the CSV into manageable chunks and only load necessary chunks on-demand.
  • Data Compression: Compress the CSV file to reduce memory usage.
  • Database Integration: If feasible, load the CSV data into a database and query it directly.
  • Memory-Mapped Files: Use mmap to map the file into virtual memory without loading it all at once.
python
1# Chunked loading with pandas
2chunks = pd.read_csv('huge_file.csv', chunksize=10000)
3lookup = {}
4for chunk in chunks:
5    for _, row in chunk.iterrows():
6        lookup[row['key']] = row['value']

Common Pitfalls

  • Async loading without waiting: In Node.js, using fs.readFile (async) without awaiting before server.listen() means requests may arrive before data is ready. Use readFileSync or await the async version before binding the port.
  • File path issues: Relative paths resolve from the working directory, not the script location. Use path.join(__dirname, 'data.csv') in Node.js or pathlib.Path(__file__).parent / 'data.csv' in Python.
  • Encoding errors: CSV files may use UTF-8 with BOM, Latin-1, or other encodings. Specify encoding explicitly: open('file.csv', encoding='utf-8-sig').
  • Memory limits: A 1 GB CSV loaded into objects can consume 3-5 GB of RAM due to object overhead. Consider SQLite or a database for files over 100 MB.
  • Hot reloading: Development servers that auto-reload (Flask debug, nodemon) re-read the CSV on every restart. For large files, this adds significant startup time.

Summary

  • Load CSV data synchronously during application startup, before the HTTP server starts listening
  • In Node.js, use fs.readFileSync or await async reads before server.listen()
  • In Python, load at module level or in a startup event handler
  • In Go, use init() functions to load before main() runs
  • For large files, consider chunked loading, databases, or memory-mapped files
Related reading
Course
Beginner
27 lessons
10 hours
System Design Fundamentals

Build a strong foundation in designing scalable, reliable distributed systems.

View the course
Track what you have practised

A free account saves your progress, solutions and study plan across every problem on Codemia.

System Design practice on Codemia

Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.

Practice system design
Codemia
© 2026 Codemia
Resources
Legal
Social
All Rights Reserved.