Mfcttrf

I have a data set of around 3M row. I has only 2 category (category- 2:1 ratio). Now i want to visualize(scatter plot) it's distribution to understand can the data linearly separable or not(In order to choose model type).I already try this and the plot is not understandable. What will be the best way to visualize this data set?

I have three suggestions that may help.

1. Reduce the point size


2. Make the points highly transparent


3. Downsample the points

Since you do not provide any sample data, I will use some random data to illustrate.

## The purpose of S1 is to intermix the two classes at random
S1 = sample(3000000)
x = c(rnorm(2000000, 0, 1), rnorm(1000000, 3,1))[S1]
y = c(rnorm(2000000, 0, 1), rnorm(1000000, 3,1))[S1]
z = c(rep(1,2000000), rep(2,1000000))[S1]

plot(x,y, pch=20, col=rainbow(3)[z])

The base plot without any adjustments is not very nice. Let's apply suggestions 1 and 2.

plot(x,y, pch=20, cex=0.4, col=rainbow(3, alpha=0.01)[z]) 

Reducing the point size and making the points highly transparent helps some. This gives a better idea of the overlap between the two distributions.

If we downsample, we don't need quite as much transparency.

## The purpose of S2 is to downsample the data
S2 = sample(3000000, 100000)
plot(x[S2],y[S2], pch=20, cex=0.4, col=rainbow(3, alpha=0.1)[z[S2]]) 

This gives a different view that provides a similar, but not identical understanding of the two distributions.

These are not magic, but I think that they are helpful.

Assuming you're using Python, the datashader module was created to effectively display very large number of points.

I however recommend using the hvplot package instead as it includes datashader support and provides a pandas compatible API.

# import modules
import pandas as pd
import hvplot.pandas

# read your data into dataframe (or whatever source).
df = pd.read_csv('large_file.csv')

# plot using hvplot; normally df.plot
df.hvplot.scatter('x_column', 'y_column', datashade=True')

datashader in effect creates a series of images and only shows the data to the required resolution without over plotting. As you zoom in it updated the view with the refined detail.

Note

If you're reading in larger than RAM datasets you may want to check out dask as well.