Lilac

What is Lilac?

Lilac is an open-source tool designed to empower data and AI practitioners to improve their products by enhancing the quality of their data. It provides capabilities for searching, quantifying, and editing data specifically for large language models (LLMs).

Key Features and Benefits

• Semantic & Keyword Search: Enables users to quickly find relevant data points within large datasets.
• Clustering: Facilitates the grouping of similar data points, making it easier to identify patterns and themes.
• Data Quality Control: Inspect and evaluate datasets to ensure high quality and reliability.
• Fuzzy-Concept Search: Refine searches to discover related concepts even when exact matches are not available.
• Blazing Fast Dataset Computations: Lilac can cluster and title 1 million data points in just 20 minutes and embed datasets at half a billion tokens per minute.

How to Use Lilac

1. Install: Use pip to install Lilac: pip install lilac
2. User Interface: Access Lilac's intuitive user interface to start exploring and editing your data.

Why is Lilac important?

Lilac helps users understand the concepts within datasets and select the right data for specific tasks. It is a critical part of data quality evaluation pipelines and aids in democratizing data across organizations.

User Testimonials

• Jonathan Talmi, Lead of Data Acquisition: "Lilac is an incredibly powerful tool for data exploration and quality control. We use Lilac daily to inspect and evaluate datasets, and then democratize them across the org. It is a critical part of our data quality evaluation pipeline."
• Jonathan Frankle, Chief Neural Network Scientist: "Lilac provides a simple path to understanding the concepts in datasets and selecting the right data for a task."
• Teknium, Co-founder, NousResearch: "Everyone working with LLM Datasets should check out @lilac_ai data platform…Their clustering helped determine a lot of topics Hermes-2.5 covers today."