Currently, we have client libraries available for the following languages:
pip install --upgrade scaleapi conda install -c conda-forge scaleapi # If using Anaconda package management
npm install scaleapi --save
Don't see your favorite language?
Let us know if you want (or are interested in writing) a library for a language not represented here!
Introducing Sail, a data pipeline starter kit with all of our best practices already included:
- Project level Pipeline Config (with versioning abstracted)
- Batch Creation and Finalization (both Easy and Complex cases)
- Task Creation based on a .csv mapping, .json file, or just a Python Dictionary. A longer-term goal is to support passing in a folder location or S3 Bucket URI.
- Concurrency for every operation so scripts can run ~30x faster
- Logging built-in
- Error Handling and retries on every part + Idempotency for Task creation
We're proud of the code we've pushed to date and it's usable, but we're still fleshing out the documentation, code samples, and related materials.
Get in touch with [email protected] if you'd like to know more.