Optimize machine learning model performance through data attribute selection and integration

Some organizations have a well-defined data catalog that you can browse through, and in other organizations, such data can be found only through word of mouth. But before you go hunting for the data, take a minute to think about what data you might need.

For example, property detail data might provide you insights into why a property was sold at a specific rate—was it because it had a two-car garage, a jacuzzi, or near a good school? And Geographic income per capita information might indicate why sale prices are low or high or how to position the house.

Make a list and then explore these options:

1. If available in your organization, explore the data catalog and look for data sources that are likely to include data points meaningful to the task at hand.

2. Engage other business units that may hold data relevant or that may have insights into which data points have historically proven relevant to the objective.

3. Also consider public data sets that may provide additional context to your model.

4. Consider commercial data sets that could also hold valuable features that will improve your model’s performance.

For our scenario, you will look at three data sets available within your organization as you build the first version of your algorithm:

1. Property sale information, being a primary dataset for the company, is compiled by a line-of-business process and put into structured JSON files in an Amazon Simple Storage Service (Amazon S3) bucket.

2. Neighborhood information from a survey provided by a 3rd party vendor.

3. Property details which are maintained in a relational database in Amazon Relational Database Service (Amazon RDS).

Here is the schema of these data sets.

Not all data is made equal, ensure data quality and relevance

Start by inspecting samples of the various data sets you’ll be working with and run some basic tests. Remember the garbage in, garbage out (GIGO) idiom; basing your model on poor quality or poorly maintained data will affect the quality of your output.

Some basic questions you will want to have an answer for are:

• Is the data clean? What you are looking for is ensuring data is accurate, consistent and complete.

• Is the data sufficient? This doesn’t necessarily mean that you are looking for petabytes of data, particularly when fine-tuning existing models, you can work with reasonably sized datasets.

• Is the data well distributed? To avoid overfitting, you want data that is diverse and well distributed.

Let’s say that you find the results acceptable, and you believe that the data you have can help you predict the valuation of the property well. The next step is to get and then join full datasets.

You now have the data you need and have figured out how to keep it up to date using ByteHouse’s Data Express service. As an ML engineer, finding important features and using them for training an ML model is the next step in and the one where the value of data can ultimately be materialized.

When it comes to using data, you know that although 18 columns might not look like a lot of data, when you multiply that by 1 million rows the total number of unique attributes and potential combinations can explode in orders of magnitude.

In real world production data sets, the actual number of columns will be in the hundreds, which, when multiplied by tens or hundreds of millions of rows, turns into a real challenge of scale across environments.

Although Machine Learning (ML) training costs have decreased, the more data, the more time and cost that will need to be invested in training the model and the larger and more compute intensive the model will end up being which may take longer to return a prediction and will require a lot more compute capacity per inference.

And there’s something all users have in common today, they all want and expect instantaneous responses. So, the key to building a model that will run quickly, inexpensively, and still deliver the expected value and experience to users lies in simplicity, efficiency and optimization.

These are some of feature selection techniques that can help:

1. Regularized trees, a technique that penalizes using a variable that is similar to variables selected for splitting previous tree nodes.

2. Lasso regression, which adds a penalty term to the cost function of regression to encourage sparsity in the coefficients.

3. Tree-based techniques such as Random Forest and Gradient Boosting provide us with feature importance to select features as well.

Now that you’ve aggregated your data, let’s go back to a more familiar tool, Jupyter, which you can access either from AWS Glue Studio, Amazon SageMaker Studio, or your computer to connect to ByteHouse.

Step 1: Connect to ByteHouse from your Jupyter notebook

With a few lines of Python code, you can easily connect your Jupyter Notebook to your ByteHouse data warehouse and run queries on it.

Step 2: Identify important features

Once you have connected to Bytehouse and loaded data into a Pandas data frame, you can use the scikit-learn library’s functions to split data into train and test sets. Then, train a model using the RandomForest Algorithm, as shown in the following screenshot.

During the experimentation shown in that screenshot, I got a good enough coefficient of determination (a metric that determines whether the model is good or not). The algorithm also identified the importance of each feature. Here is a graph that shows sqft, bedrooms, median_income, and year_built as important determinants.

When I took just these four features and re-trained a regression model, I got 0.82 as the coefficient of determination, which was quite good. So, despite shedding more than three-quarters of the total features, I was able to train an almost equally performant ML model that is also light.