10.1 Support Vector Machines

Support vector machines (SVMs) are supervised machine learning (ML) models used to solve regression and classification problems. However, it is widely used for solving classification problems. The main goal of SVM is to segregate the n-dimensional space into labels or classes by defining a decision boundary or hyperplanes. In this chapter, we shall explore SVM for solving classification problems.

10.1.1 SVM Working Principle

SVM Working Principle | Parteek Bhatia, https://youtu.be/UhzBKrIKPyE

To understand the working principle of the SVM classifier, we will take a standard ML problem where we want a machine to distinguish between a peach and an apple based on their size and color.

Let us suppose the size of the fruit is represented on the X-axis and the color of the fruit is on the Y-axis. The distribution of the dataset of apple and peach is shown in Figure 10.1.

To classify it, we must provide the machine with some sample stock of fruits and label each of the fruits in the stock as an “apple” or “peach”. For example, we have a labeled dataset of some 100 fruits with corresponding labels, i.e., “apple” or “peach”. When this data is fed into a machine, it will analyze these fruits and train itself. Once the training is completed, if some new fruit comes into the stock, the machine will classify whether it is an “apple” or a “peach”.

Most of the traditional ML algorithms would learn by observing the perfect apples and perfect peaches in the stock, i.e., they will train themselves by observing the ideal apples of stock (apples which are very much like apples in terms of their size and color) and the perfect peaches of stock (peaches which are very much like peaches in terms of their size and color). These standard samples are likely to be found in the heart of stock. The heart of the stock is shown in Figure 10.2.

1.1 Introduction

In today’s data-driven world, information flows through the digital landscape like an untapped river of potential. Within this vast data stream lies the key to unlocking a new era of discovery and innovation. Machine learning (ML), a revolutionary field, acts as the gateway to this wealth of opportunities. With its ability to uncover patterns, make predictive insights, and adapt to evolving information, ML has transformed industries, redefined technology, and opened the door to limitless possibilities. This book is your gateway to the fascinating realm of ML—a journey that empowers you to harness the power of data, enabling you to build intelligent systems, make informed decisions, and explore the boundless possibilities of the digital age.

ML has emerged as the dominant approach for solving problems in the modern world, and its wide-ranging applications have made it an integral part of our lives. Right from search engines to social networking sites, everything is powered by ML algorithms. Your favorite search engine uses ML algorithms to get you the appropriate search results. Smart home assistants like Alexa and Siri use ML to serve us better. The influence of ML in our day-to-day activities is so much that we cannot even realize it. Online shopping sites like Amazon, Flipkart, and Myntra use ML to recommend products. Facebook is using ML to display our feed. Netflix and YouTube are using ML to recommend videos based on our interests.

Data is growing exponentially with the Internet and smartphones, and ML has just made this data more usable and meaningful. Social media, entertainment, travel, mining, medicine, bioinformatics, or any field you could name uses ML in some form.

To understand the role of ML in the modern world, let us first discuss the applications of ML.

16.1 Introduction to Artificial Neural Network

Neural networks and deep learning are the buzzwords in modern-day computer science. And, if you think that these are the latest entrants in this field, you probably have a misconception. Neural networks have been around for quite some time, and they have only started picking up now, putting up a huge positive impact on computer science.

Artificial neural network (ANN) was invented in the 1960s and 1970s. It became a part of common tech talks, and people started thinking that this machine learning (ML) technique would solve all the complex problems that were challenging the researchers during that time. But sooner, the hopes and expectations died off over the next decade.

The decline could not be attributed to some loopholes in neural networks, but the major reason for the decline was the “technology” itself. The technology back then was not up to the right standard to facilitate neural networks as they needed a lot of data for training and huge computation resources for building the model. During that time, both data and computing power were scarce. Hence, the resulting neural network remained only on paper rather than taking centerstage of the machine to solve some real-world problems.

Later on, at the beginning of the 21st century, we saw a lot of improvements in storage techniques resulting in reduced cost per gigabyte of storage. Humanity witnessed a huge rise in big data due to the Internet boom and smartphones.

13.1 Implementation of k-means Clustering and Hierarchical Clustering

In the previous chapter, we discussed various clustering algorithms. We learned that clustering algorithms are broadly classified into partitioning methods, hierarchical methods, and density-based methods. The k-means clustering algorithm follows partitioning method; agglomerative and divisive algorithms follow the hierarchical method, while DBSCAN is based on density-based clustering methods.

In this chapter, we will implement each of these algorithms by considering various case studies by following a step-by-step approach. You are advised to perform all these steps on your own on the mentioned databases stated in this chapter.

The k-means algorithm is considered a partitioning method and an unsupervised machine learning (ML) algorithm used to identify clusters of data items in a dataset. It is one of the most prominent ML algorithms, and its implementation in Python is quite straightforward. This chapter will consider three case studies, i.e., customers shopping in the mall dataset, the U.S. arrests dataset, and a popular Iris dataset. We will understand the significance of k-means clustering techniques to implement it in Python through these case studies. Along with the clustering of data items, we will also discuss the ways to find out the optimal number of clusters. To compare the results of the k-means algorithm, we will also implement hierarchical clustering for these problems.

We will kick-start the implementation of the k-means algorithm in Spyder IDE using the following steps.

• Step 1: Importing the libraries and the dataset—The dataset for the respective case study would be downloaded, and then the required libraries would be imported.

• Step 2: Finding the optimal number of clusters—We will find the optimal number of clusters by the elbow method for the given dataset.

• Step 3: Fitting k-means to the dataset—A k-means model will be prepared by training the model over the acquired dataset.

• Step 4: Visualizing the clusters—The clusters formed by the k-means model would then be visualized in the form of scatter plots.

14.1 Introduction to Association Rule Mining

Association rule mining is a rule-based technique to discover the relation between the attributes of a dataset. It is used to find the relation between the sales of item X and item Y. It is often called a “market basket” analysis, as shown in Figure 14.1. Here, the market analyst examines the items that consumers often purchase together to find the relation between the sale of item X and item Y.

In other words, when customers visit a store, they may buy a certain type of items together during a shopping trip. For example, as shown in Figure 14.1, a database of customer’s transactions (e.g., shopping baskets) is shown where each transaction consists of a set of items (e.g., products) purchased during a visit, machine learning (ML) engineers can use association mining for finding out a group of items which are frequently purchased together (customers purchasing behavior). This is also referred to as an analysis of customer purchasing behavior. For example, “IF one buys bread, THEN there is a high probability of buying butter with it”, as it is common that people who buy bread often buy butter with it. The store manager can use this information and arrange the items accordingly to increase sales and the overall efficiency of the store.

Let us consider a situation where the store manager feels that there is a lot of rush and customers always complain about the slow working of his store. He is exploring different ways to improve the efficiency of his store. He performed an association analysis and prepared a list of associated items like bread and butter. He may decide to put all these associated items together on the same shelf or near each other so that customers can find them quickly, reducing their shopping time. It will also improve the overall efficiency of the store and the sale of the products. To further improve the shopping experience of his customers, he can create different combos and put sales over these combos.

22.1 Intuition of Genetic Algorithm

Genetic algorithm (GA) is inspired by nature, and it plays a vital role in the field of machine learning (ML). It selects the best-optimized solution from all available possible solutions or candidates. As nature selects the best possible candidates using the theory of evolution, in the same way, the GA selects the best possible solution from the available solutions.

One of the applications of GAs in ML is to select the global minima from all possible (local) minima by using natural selection. In earlier chapters, we learned that during the training of an artificial neural network, the main goal is to obtain the weights with a minimum cost function value. The gradient descent algorithm is commonly used to find the local minima of the cost function. But, we must find the global minima to reach the optimal weights. A GA can be used to find the global minima out of all available local minima or possible solutions. In this case, the set of possible local minima becomes the population containing possible candidates.

In this chapter, we will discuss inspiration from nature which is the main driving concept in working of GAs and their implementation. To get a good idea about the GA, we will discuss the basics of natural selection by revisiting the theory of evolution in the next section.

22.2 The Inspiration behind Genetic Algorithm

The concepts discussed in this chapter are also available in the form of the free online Udemy Course, Genetic Algorithm for Machine Learning by Parteek Bhatia,

https://www.udemy.com/course/genetic-algorithm-for-machine-learning/

The GA is one of the first and most well-regarded evolutionary algorithms in computer science literature. John Holland, a researcher at the University of Michigan, gave this algorithm in the 1970s, but it became popular in the ‘90s.

3.1 Need for Data Pre-processing

We live in an age where data is considered oil because we need data to train machine learning (ML) algorithms. The most important job for a data analyst is to collect, clean, and analyze the data and build ML models on the cleaned dataset. But often, the raw data that we obtain is noisy. It consists of many discrepancies, inconsistencies, and often missing values. To understand this situation, let us consider an example.

Suppose we have to predict the house price, and for this, we have collected data from a few previous transactions, as shown in Figure 3.1.

In a perfect situation, the captured data should be of this format, as shown in Figure 3.1. Here, we have the size of the house and the number of bedrooms as input features, while the price is the output attribute. We can predict the price of an unknown instance through regression.

But practically, in most situations, the captured data is not of good quality, and usually, we have a dataset, as shown in Figure 3.2.

You can see that this data is messy. There are a lot of unknown or missing values, and if we trained the model on this data, its prediction would be very poor. Also, you can identify the noise and incorrect labels like the second record price is incorrect and will result in poor model training.

We can also consider some more examples like if someone entered –1 in the “salary credited” column in the case of employee dataset. It does not make any sense and will be considered noise. Sometimes, we may have an unrealistic and impossible combination of data; for example, let us consider a record where we have Gender–Male and Pregnant–Yes.

Today, Python is known to be one of the most in-demand programming languages. As per the stats of GitHub (a provider of Internet hosting for software development), Python is the second most popular programming language, following JavaScript, as shown in Figure 2.1, and soon it may be on the top of the chart. Python surpassed Java, PHP, and other prominent languages in 2019.

Python is easy and versatile. So it is acclaimed as the major programming language to work on many new-age technologies like machine learning (ML), artificial intelligence, data science, and natural language processing. The creator of Python, Guido van Rossum, in 1991, stated that Python is a high-level programming language, and its core design philosophy is about code readability and syntax, which allows programmers to express concepts in a few lines of code. Interestingly, the name Python is inspired by Guido's favorite television show Monty Python's Flying Circus.

In this chapter, we will discuss various programming constructs of Python so that you can easily implement ML algorithms by using it. Before writing the actual code in Python, let us focus on the features of Python that make it so popular and unique.

2.1 Features of Python

Features offered by Python can be visualized in Figure 2.2. Talking about them profoundly, the main features of Python are as follows:

• • Beginner's Language: Python is not only just easy to code and learn, but also fast to grasp, and hence it is a suitable choice for any novice user who wants to learn to program. This is why nowadays this language is introduced to students in schools.

• • Interpreted: Unlike other programming languages such as C or C++, Python does not require you to compile programs before executing them. It is an interpreted language, i.e., the code written in Python gets processed in real-time line by line.

• • Interactive: The interactive feature of Python enables real-time feedback, allowing programmers to experiment, debug, and make adjustments on the go.