Data Analytics And Visualization

What makes a data scientist different from another? The answer is simple : How good he plays with the data! Before applying machine learning algorithms it is important to interpret the data well. A data scientist can make the data more valuable just by communicating with it. 

Goals

The main goals of a data scientist are :

1. Getting a better understanding of data
2. Getting a better understanding of problem statement
3. Identifying different patterns in data
4. Identifying of hidden patterns
5. Predicting the likelihood of something happening

Fundamental Steps of a Data Analytics Project

• STEP 1 : Understanding the Problem

The problem should be clear, concise and measurable. Many companies are too vague when it comes to defining the data problem, which makes it difficult for data scientists to translate them into machine code. Enough data should be available in usable format. 

• STEP 2 : Get the data

We can get data using databases, APIs and some open sources. Some popular sources are as follows :

1. Kaggle Datasets
2. Google datasets search engines
3. Datasets via AWS
4. Open Ml

• STEP 3 : Clean the data

Now, it’s time to check whether your data is homogeneous or clean.

• Handling missing values

We just can’t ignore missing values because many machine learning algorithms don’t accept missing values. Either we can simply drop them or can impute them with different techniques.

1. Impute with central tended values
2. Impute with most correlated feature

• Treating Outliers

An outlier is an extremely large and extremely small data value relative to the rest of the data set. It may represent a data entry error or a genuine error. An outlier can skew our data, so they must be treated. We can detect outliers using z-values. There are different ways to deal with outliers :

1. Deleting Observations
2. Transforming Values
3. Imputation
4. Separate Training

• Removing duplicate values

Duplicates are the extreme case of non random sampling or multiple entry of data. Duplicates can bias your fitted model and can cause overfitting , so they must be removed. You can use the drop_duplicates() function to get rid of all duplicate values.

• STEP 4 : Data Enrichment

Data Enrichment is required to get most from the data. After cleaning the data, we manipulate the data so we can get the most value out of it. Group the data from different sources and narrow them to get an essential feature. For example, take the date-time data, extract its components (like day,month,year) and then can use them for finding, just say national holidays. 

It helps brands to better understand their customers and gain deeper insights into their lives. Through data enrichment we can improve user experience, maximize customer engagement, boost targeted marketing, etc.

• STEP 5 : Visualizations

Pictures talk better than words, this implies our data too. Let data talk via visualization rather than numbers. We generally use visualization because they are more interactive and give better understanding. When you are dealing with a large volume of data then it becomes easy with the help of visualizations. 

Some Data Visualization Techniques :

1. Heatmaps
2. Box plots
3. Histograms
4. Scatterplots
5. Distribution Plots

• STEP 7 : Data Preprocessing

Machines don’t understand human language so before applying any machine learning algorithm we should preprocess our data. Data Preprocessing is important to check the interpretability of the data. 

• Encoding categorical data

There are two ways to encode categorical data :

1. Label Encoder
2. One Hot Encoding

• Encoding numerical data

Here, we can use standard scaling and min-max scaling.

1. Standard Scaling
2. Min-Max Scaling

• STEP 8 : Predictive Model

Now our data is ready for a machine learning model. Now you can build models to uncover trends in the data that were not distinguishable in graphs and stats and can also predict future trends.

     Well now you have learned the basics of data engineering!