Write out file with google colab
System Design practice on Codemia
Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.
Introduction
Writing files in Google Colab is straightforward, but durability depends on where you save them. Files written under /content are ephemeral and can disappear when the runtime restarts. For persistent output, you should write to mounted Google Drive or explicitly download artifacts.
Understand Colab Storage Layers
Colab notebooks commonly use two storage locations:
- Runtime disk under
/content. - Mounted Drive under
/content/drive.
Runtime disk is fast and ideal for temporary intermediate files. Drive is persistent across sessions but usually slower for heavy write workloads.
You should decide location based on whether the output must survive runtime disconnects.
Write a Simple Text File to Runtime
Use standard Python file APIs for quick output during one session.
This method is useful for logs, debug snapshots, or temporary exports you will process immediately.
Write Structured Files with Pandas
For tabular outputs, pandas provides clear and reliable writers.
The file appears in the Colab file sidebar and can be downloaded manually.
Persist Files to Google Drive
Mounting Drive is the standard way to keep generated files beyond runtime lifetime.
Then write to a folder in MyDrive.
If you rerun the notebook later, the file still exists in Drive.
Download Output to Local Machine
When you need quick local export, use Colab download helpers.
This triggers browser download and is practical for occasional small files.
Handle Large Outputs Safely
For large writes, stream incrementally and avoid keeping huge strings in memory.
For large datasets destined for Drive, write to runtime first, then copy finalized artifacts. This often performs better than writing directly to Drive on every small operation.
Create Compressed Artifacts for Easy Sharing
When notebooks produce many files, package them into one archive.
A zip artifact is easier to download and track than multiple loose files.
Verify Paths to Avoid Silent Mistakes
Long notebooks can change context and make path bugs hard to spot. Add simple checks before writing important output.
Use absolute paths for critical files so behavior is consistent across notebook reruns.
Practical Persistence Strategy for Training Runs
For long ML jobs, save progress incrementally:
- Write checkpoints at intervals.
- Append metrics to a log file on Drive.
- Keep a small metadata file with run parameters.
This strategy reduces data loss risk when runtimes disconnect unexpectedly.
Common Pitfalls
- Saving critical output only under
/contentand losing it after runtime reset. - Forgetting to mount Drive before writing Drive paths.
- Using relative paths that depend on changing working directory.
- Writing huge files directly to Drive in many tiny operations.
- Generating many artifacts but not packaging or naming them consistently.
Summary
- Use
/contentfor temporary fast local runtime files. - Use mounted Drive for persistence across Colab sessions.
- Use
files.downloadfor quick local exports. - Prefer absolute paths and explicit path checks in long notebooks.
- Save important outputs incrementally to reduce runtime disconnect risk.
Related reading
- Write parquet from AWS Kinesis firehose to AWS S3
- Writing a pickle file to an s3 bucket in AWS
- You have requested more vCPU capacity than your current vCPU limit of 0
- You may not specify a referenced group id for an existing IPv4 CIDR rule. prompt when editing the Inbound rule in AWS Security Group
- Write to UTF-8 file in Python
- Writelines writes lines without newline, Just fills the file
- Your current user or role does not have access to Kubernetes objects on this EKS cluster - EKS
- -bash aws command not found

System Design Fundamentals
Build a strong foundation in designing scalable, reliable distributed systems.
View the courseTrack what you have practised
A free account saves your progress, solutions and study plan across every problem on Codemia.
System Design practice on Codemia
Work through 120+ system design problems with detailed solutions, from rate limiters to multi-region storage.