Quilt Teams Registry
Quilt Data | 2.9.15Linux/Unix, Amazon Linux 2018.03.0 - 64-bit Amazon Machine Image (AMI)
Missing element in the ML flow
Quilt simplified our flow in data maintenance and versioning. Now, it became extremely easy to keep track of changes in a dataset and refer in a reproducible manner a specific revision without worrying if someone overwrites the data. We have it already integrated into our flow, so the dataset updates interfere with model building no more.
Quilt team provides us with ongoing support. Bugs happen in every software, but in the case of small bug we found, we received a fixup in no time, so we could smoothly continue our work.
At this moment we spotted several drawbacks, but these are mostly addressed in the upcoming Quilt 3. The most crucial for us was limit of a number of files included in a single package. It is pointed out on the product page (but easy to be missed), that Quilt handles packages up to 40 000 files. Even though, we successfully pushed packages with 50 000 files already and are awaiting Quilt 3 that is supposed to handle millions of files.
It is incredibly useful to provide separation between different projects just by using permissions. However, missing permission group system introduces some overhead into managing all permissions manually (for each data package, for each user).
What we love most about Quilt, is the caching feature. We reduced data transfer costs while keeping low complexity of scripts.
Overall grade is 5/5 since that tool was missing heavily in the flow we had for Machine Learning. At this moment we use it also for versioning models (especially that we generate models in a bunch of formats each time) and Jupyter Notebooks (for which Git isn't the best option)
- Leave a Comment |
- Mark review as helpful
Great tool for sharing and versioning datasets
Quilt Data Registry has been incredibly useful to us in sharing datasets with our clients and managing access to them. Quilt handles versioning and packaging with no effort on our part which]allows us to be able to share our analysis externally with ease. Clients are able to import the transformed datasets into their workflow with full portability. Definitely worth trying out!