Deep Lake

My main use case for Deep Lake  is storing unstructured data, which can be in the form of images and PDFs, and we have some audios and videos as well. We have used it to store this data in a single place with built-in versioning. We have used Deep Lake  for searching and storing the data.

We have used Deep Lake to store data with built-in versioning. It is used for storing and searching data plus vectors while building LLM applications and for managing datasets while training deep learning models. The typical use case is that we have used it in our support and legal team and research team who were manually digging through thousands of documents to answer questions. We wanted a bot that can read this entire data, whether it is in PDF, image, audio, or video format. We have used Deep Lake in that area.

The best features Deep Lake offers include the ability to store any type of data. Deep Lake stores any format of data: PDFs, images, audios, and videos in a single place. It has flexibility to store any kind of data, which is unstructured and structured. This is the main advantage.

The flexibility to store any format of data makes the process easier. We wanted to recall everything when we needed to review instead of going through every document, opening every PDF and searching for the specific information. Deep Lake made our process easier by storing the data and giving us exactly what we are looking for at a particular point in time instead of requiring manual searching.

Deep Lake has positively impacted our organization because it has saved our time and we can directly and indirectly see the impact on our FTEs. We have reduced one or two FTEs per month. The whole FTE reduction occurred because a few hours per week previously spent by analysis or support colleagues manually searching shared drives has been reduced. It has helped in terms of FTE reductions, faster speed, and reducing manual work that we used to do multiple times.

I can say we have saved two FTEs per story, but a few hours per week. I cannot say this is the exact hour, but it can be a few hours a day because we roughly used to spend one or two hours previously.

Deep Lake can be improved by having filtering options that can give us more options to filter when we are comparing anything. Security-wise and cost predictability are areas that could be improved.

Deep Lake could benefit from combined filtering that can handle heavy metadata filtering alongside search functionality. This is what I can recall at this point.

I have been using Deep Lake for six months.

Deep Lake is stable and very stable compared to other solutions.

Deep Lake handles data in larger volumes regularly. In terms of scalability, it is very scalable and reliable. Bottlenecks can be addressed effectively over time. It performs quite well.

We have not reached out to customer support for major issues, but in the initial days, we raised some queries to customer support. They are available through some channels, but we need to manually follow up with them occasionally. Overall, their support is good.

Before choosing Deep Lake, we looked at LanceDB . LanceDB  uses IVFPQ for ANN search, which is better for smaller datasets. Deep Lake uses linear search, so it can be used for lakh rows and HNSW based ANN beyond that. Both prioritize accuracy at a small scale and switch to approximate methods as the data grows. That is where we moved to Deep Lake.

Deep Lake is not deployed as a public instance. We actually get it from the libraries. Installing Deep Lake does not require downloading from somewhere else. We can choose our own cloud for data storage. It is actually installed from the library.

We use S3  for our own cloud deployment.

I have not seen a concrete return on investment in terms of money saved because it is an indirect impact. The direct impact was on time saving. That time saving has indirectly impacted money saving. People were actually spending a lot of time searching everything. This has been reduced to a few hours per week. Two employees were reduced if you look at the overall picture. I cannot give an exact number for the money saved, but it has made a very significant impact in terms of reducing manual work.

I am not part of the licensing team, but I have an idea because I have been on calls regarding this matter. Deep Lake is open source and available from anywhere. We can download it and use it.

The switching part is the main reason we chose Deep Lake because it is open source and we can install and use it. Other tools exist which are common alternatives like LanceDB and PGVector. In terms of using and installing, we do not need to purchase anything or have any contract license-based arrangements. That is where we moved to Deep Lake.

The advice I would give to others looking into using Deep Lake differs from business to business. Whether you have a small-scale or large-scale organization, if you do not have any budget to spend on these kinds of tools, Deep Lake will definitely establish itself on your platform. It depends on the use case and modeling training and multi-modal analysis. A good approach would be to acknowledge the variance rather than giving one blanket recommendation. Deep Lake is cost-sensitive, scalable, and reliable. I would recommend that all teams pilot Deep Lake with a real subset of data before full commitment since it differs from business to business. In terms of free access and everything, Deep Lake establishes itself well. I rate this product an eight out of ten.

Private Cloud

Amazon Web Services  (AWS )