Converting numpy ndarray into pandas dataframe with column names and types

Question

Edit: As explained below in @floydian's comment, the problem was that calling a = np.array(a, dtype=d) creates an a double array which was causing the problem.

I am aware that this has been already asked multiple times, and in fact am looking at Creating a Pandas DataFrame with a numpy array containing multiple types answer right now. But I still seem to have a problem while converting. It must be something very simple that I am missing. I hope that someone can be so kind and point it out. Sample code below:

import numpy as np
import pandas as pd

a = np.array([[1, 2], [3, 4]])
d = [('x','float'), ('y','int')]
a = np.array(a, dtype=d)

# Try 1
df= pd.DataFrame(a)
# Result - ValueError: If using all scalar values, you must pass an index

# Try 2
i = [1,2]
df= pd.DataFrame(a, index=i)
# Result - Exception: Data must be 1-dimensional

I mean, yeah, but what does it look like? Is the first column int and the second float? — cs95
– cs95, Commented Mar 6, 2018 at 19:44
oh right, yes that's the idea: the first column int and the second float — shiftyscales
– shiftyscales, Commented Mar 6, 2018 at 19:45

Floydian · Accepted Answer · 2018-03-06 20:15:58Z

2

I would define the array like this:

a = np.array([(1, 2), (3, 4)], dtype=[('x','float'), ('y', 'int')])
pd.DataFrame(a)

gets what you want.

answered Mar 6, 2018 at 20:15

Floydian

4001 silver badge14 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

shiftyscales Over a year ago

Unfortunately in my original problem the array is defined in the way I did it above.

Floydian Over a year ago

a = np.array(a, dtype=d) creates two 2x2 arrays, a['x'] with float as dtype applied to all values and a['y'] with int as dtype applied to all values. I am guessing this is the reason why you're getting all those errors.

shiftyscales Over a year ago

That was indeed the reason. Thank you for pointing me out in the right direction.

erocoar · Accepted Answer · 2018-03-06 21:16:15Z

1

One option to separate it after the fact could be e.g.

pd.DataFrame(a.astype("float32").T, columns=a.dtype.names).astype({k: v[0] for k, v in a.dtype.fields.items()})

Out[296]: 
     x  y
0  1.0  3
1  2.0  4

answered Mar 6, 2018 at 21:16

erocoar

5,9333 gold badges26 silver badges45 bronze badges

Collectives™ on Stack Overflow

Converting numpy ndarray into pandas dataframe with column names and types

2 Answers 2

3 Comments

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

Comments

Linked

Related