16) Data Manipulation with Pandas

Related references:

First, let’s check that we have Pandas installed

Previous lectures had instructions to install Pandas. Let’s check if we are good to go!

In [1]:

# Just as NumPy is commonly abbreviated as 'np', Pandas is abbreviated as 'pd'
import pandas as pd

pd.__version__

Out[1]:

'0.21.0'

Pandas extends features of NumPy

Pandas is a new library built on NumPy. Ever wish you could have column and row labels on a NumPy ndarray? The Pandas DataFrame answers that wish, and can also handle heterogeneous types and/or missing data!

The three main data types in Pandas are built on the Numpy ndarray:

  • Series
  • DataFrame
  • Index

We’ll also be using NumPy later, so we’ll import that now.

In [2]:

import numpy as np

The Pandas Series Object

A Series is a one-dimensional array-like object containing a sequence of values (of similar types to NumPy types) and an associated array of data labels, called its index. The simplest Series is formed from only an array of data:

In [3]:

obj = pd.Series([4, 7, -5, 3])
obj

Out[3]:

0    4
1    7
2   -5
3    3
dtype: int64

The string representation of a Series displayed interactively shows the index on the left and the values on the right. Since we did not specify an index for the data, a default one consisting of the integers 0 through N - 1 (where N is the length of the data) is created. You can get the array representation and index object of the Series via its values and index attributes, respectively:

In [4]:

obj.values

Out[4]:

array([ 4,  7, -5,  3])

In [5]:

obj.index  # like range(4)

Out[5]:

RangeIndex(start=0, stop=4, step=1)

Often it will be desirable to create a Series with an index identifying each data point with a label:

In [6]:

obj2 = pd.Series([4, 7, -5, 3], index=['d', 'b', 'a', 'c'])
obj2

Out[6]:

d    4
b    7
a   -5
c    3
dtype: int64

In [7]:

obj2.index

Out[7]:

Index(['d', 'b', 'a', 'c'], dtype='object')

Like with a NumPy array, data can be accessed by the associated index via the familiar Python square-bracket notation:

In [8]:

obj2[1:3]

Out[8]:

b    7
a   -5
dtype: int64

Compared with NumPy arrays, you can use labels in the index when selecting single values or a set of values:

In [9]:

obj2['a']

Out[9]:

-5

In [10]:

obj2['d'] = 6
obj2[['c', 'a', 'd']]

Out[10]:

c    3
a   -5
d    6
dtype: int64

Here [‘c’, ‘a’, ‘d’] is interpreted as a list of indices, even though it contains strings instead of integers.

Using NumPy functions or NumPy-like operations, such as filtering with a boolean array, scalar multiplication, or applying math functions, will preserve the index-value link:

In [11]:

obj2[obj2 > 0]

Out[11]:

d    6
b    7
c    3
dtype: int64

In [12]:

obj2 * 2

Out[12]:

d    12
b    14
a   -10
c     6
dtype: int64

In [13]:

np.exp(obj2)

Out[13]:

d     403.428793
b    1096.633158
a       0.006738
c      20.085537
dtype: float64

Another way to think about a Series is as a fixed-length, ordered dict, as it is a mapping of index values to data values. It can be used in many contexts where you might use a dict:

In [14]:

'b' in obj2

Out[14]:

True

In [15]:

'e' in obj2

Out[15]:

False

Should you have data contained in a Python dict, you can create a Series from it by passing the dict:

In [16]:

sdata = {'Ohio': 35000, 'Texas': 71000, 'Oregon': 16000, 'Utah': 5000}
obj3 = pd.Series(sdata)
obj3

Out[16]:

Ohio      35000
Oregon    16000
Texas     71000
Utah       5000
dtype: int64

When you are only passing a dict, the index in the resulting Series will have the dict’s keys in sorted order. You can override this by passing the dict keys in the order you want them to appear in the resulting Series:

In [17]:

states = ['Texas', 'California', 'Ohio', 'Oregon']
obj4 = pd.Series(sdata, index=states)
obj4

Out[17]:

Texas         71000.0
California        NaN
Ohio          35000.0
Oregon        16000.0
dtype: float64

In [18]:

obj4.values

Out[18]:

array([ 71000.,     nan,  35000.,  16000.])

Here, three values found in sdata were placed in the appropriate locations, but since no value for California was found, it appears as NaN (not a number), which is considered in pandas to mark missing or NA values. Since ‘Utah’ was not included in states, it is excluded from the resulting object.

The isnull and notnull functions in pandas should be used to detect missing data:

In [19]:

pd.isnull(obj4)

Out[19]:

Texas         False
California     True
Ohio          False
Oregon        False
dtype: bool

In [20]:

pd.notnull(obj4)

Out[20]:

Texas          True
California    False
Ohio           True
Oregon         True
dtype: bool

Series also has these as instance methods:

In [21]:

obj4.isnull()

Out[21]:

Texas         False
California     True
Ohio          False
Oregon        False
dtype: bool

A useful Series feature for many applications is that it automatically aligns by index label in arithmetic operations:

In [22]:

obj3

Out[22]:

Ohio      35000
Oregon    16000
Texas     71000
Utah       5000
dtype: int64

In [23]:

obj4

Out[23]:

Texas         71000.0
California        NaN
Ohio          35000.0
Oregon        16000.0
dtype: float64

In [24]:

obj3 + obj4

Out[24]:

California         NaN
Ohio           70000.0
Oregon         32000.0
Texas         142000.0
Utah               NaN
dtype: float64

Both the Series object itself and its index have a name attribute, which integrates with other key areas of pandas functionality:

In [25]:

obj4.name = 'population'
obj4.index.name = 'state'
obj4

Out[25]:

state
Texas         71000.0
California        NaN
Ohio          35000.0
Oregon        16000.0
Name: population, dtype: float64

A Series’s index can be altered in-place by assignment:

In [26]:

obj

Out[26]:

0    4
1    7
2   -5
3    3
dtype: int64

In [27]:

obj.index = ['Bob', 'Steve', 'Jeff', 'Ryan']
obj

Out[27]:

Bob      4
Steve    7
Jeff    -5
Ryan     3
dtype: int64

The Pandas DataFrame Object

A DataFrame represents a rectangular table of data and contains an ordered collection of columns, each of which can be a different value type (numeric, string, boolean, etc.). The DataFrame has both a row and column index; it can be thought of as a dict of Series all sharing the same index. Under the hood, the data is stored as one or more two-dimensional blocks rather than a list, dict, or some other collection of one-dimensional arrays.

FYI: While a DataFrame is physically two-dimensional, you can use it to represent higher dimensional data in a tabular format using hierarchical indexing, which we’ll come back to later.

There are many ways to construct a DataFrame, though one of the most common is from a dict of equal-length lists or NumPy arrays:

In [28]:

data = {'state': ['Ohio', 'Ohio', 'Ohio', 'Nevada', 'Nevada', 'Nevada'],
        'year': [2000, 2001, 2002, 2001, 2002, 2003],
        'pop': [1.5, 1.7, 3.6, 2.4, 2.9, 3.2]}
frame = pd.DataFrame(data)

The resulting DataFrame will have its index assigned automatically as with Series, and the columns are placed in sorted order:

In [29]:

frame

Out[29]:
pop state year
0 1.5 Ohio 2000
1 1.7 Ohio 2001
2 3.6 Ohio 2002
3 2.4 Nevada 2001
4 2.9 Nevada 2002
5 3.2 Nevada 2003

FYI: Thank Jupyter notebooks for the nice DataFrame output format.

For large DataFrames, the head method selects only the first five rows:

In [30]:

frame.head()

Out[30]:
pop state year
0 1.5 Ohio 2000
1 1.7 Ohio 2001
2 3.6 Ohio 2002
3 2.4 Nevada 2001
4 2.9 Nevada 2002

If you pass a column that isn’t contained in the dict, it will appear with missing values in the result:

In [31]:

frame2 = pd.DataFrame(data, columns=['year', 'state', 'pop', 'debt'],
                      index=['one', 'two', 'three', 'four',
                             'five', 'six'])
frame2

Out[31]:
year state pop debt
one 2000 Ohio 1.5 NaN
two 2001 Ohio 1.7 NaN
three 2002 Ohio 3.6 NaN
four 2001 Nevada 2.4 NaN
five 2002 Nevada 2.9 NaN
six 2003 Nevada 3.2 NaN

A column in a DataFrame can be retrieved as a Series either by dict-like notation or by attribute:

In [32]:

frame2['state']

Out[32]:

one        Ohio
two        Ohio
three      Ohio
four     Nevada
five     Nevada
six      Nevada
Name: state, dtype: object

Rows can also be retrieved by position or name with the special loc attribute:

In [33]:

frame2.loc['three']

Out[33]:

year     2002
state    Ohio
pop       3.6
debt      NaN
Name: three, dtype: object

Columns can be modified by assignment. For example, the empty ‘debt’ column could be assigned a scalar value or an array of values:

In [34]:

frame2['debt'] = 16.5
frame2

Out[34]:
year state pop debt
one 2000 Ohio 1.5 16.5
two 2001 Ohio 1.7 16.5
three 2002 Ohio 3.6 16.5
four 2001 Nevada 2.4 16.5
five 2002 Nevada 2.9 16.5
six 2003 Nevada 3.2 16.5 
In [35]:

frame2['debt'] = np.arange(6.)
frame2

Out[35]:
year state pop debt
one 2000 Ohio 1.5 0.0
two 2001 Ohio 1.7 1.0
three 2002 Ohio 3.6 2.0
four 2001 Nevada 2.4 3.0
five 2002 Nevada 2.9 4.0
six 2003 Nevada 3.2 5.0

When you are assigning lists or arrays to a column, the value’s length must match the length of the DataFrame. If you assign a Series, its labels will be realigned exactly to the DataFrame’s index, inserting missing values in any holes:

In [36]:

val = pd.Series([-1.2, -1.5, -1.7], index=['two', 'four', 'five'])
frame2['debt'] = val
frame2

Out[36]:
year state pop debt
one 2000 Ohio 1.5 NaN
two 2001 Ohio 1.7 -1.2
three 2002 Ohio 3.6 NaN
four 2001 Nevada 2.4 -1.5
five 2002 Nevada 2.9 -1.7
six 2003 Nevada 3.2 NaN 
In [37]:

val2 = pd.Series([-1.2, -1.5, -1.7], index=['one', 'three', 'six'])
frame2['debt'] = val
frame2

Out[37]:
year state pop debt
one 2000 Ohio 1.5 NaN
two 2001 Ohio 1.7 -1.2
three 2002 Ohio 3.6 NaN
four 2001 Nevada 2.4 -1.5
five 2002 Nevada 2.9 -1.7
six 2003 Nevada 3.2 NaN

Assigning a column that doesn’t exist will create a new column. The del keyword will delete columns as with a dict.

As an example of del, let’s add a new column of boolean values where the state column equals ‘Ohio’

In [38]:

frame2['eastern'] = frame2['state'] == 'Ohio'
frame2

Out[38]:
year state pop debt eastern
one 2000 Ohio 1.5 NaN True
two 2001 Ohio 1.7 -1.2 True
three 2002 Ohio 3.6 NaN True
four 2001 Nevada 2.4 -1.5 False
five 2002 Nevada 2.9 -1.7 False
six 2003 Nevada 3.2 NaN False

The del method can then be used to remove this column:

In [39]:

del frame2['eastern']
frame2.columns

Out[39]:

Index(['year', 'state', 'pop', 'debt'], dtype='object')

Caution: The column returned from indexing a DataFrame is a view on the underlying data, not a copy. Thus, any in-place modifications to the Series will be reflected in the DataFrame. The column can be explicitly copied with the Series’s copy method.

Another common form of data is a nested dict of dicts:

In [40]:

pop = {'Nevada': {2001: 2.4, 2002: 2.9},
       'Ohio': {2000: 1.5, 2001: 1.7, 2002: 3.6}}

If the nested dict is passed to the DataFrame, pandas will interpret the outer dict keys as the columns and the inner keys as the row indices:

In [41]:

frame3 = pd.DataFrame(pop)
frame3

Out[41]:
Nevada Ohio
2000 NaN 1.5
2001 2.4 1.7
2002 2.9 3.6

Given a two-dimensional array of data, we can create a DataFrame with any specified column and index names. If omitted, an integer index will be used for each:

In [42]:

pd.DataFrame(np.random.rand(3, 2),
             columns=['foo', 'bar'],
             index=['a', 'b', 'c'])

Out[42]:
foo bar
a 0.480958 0.226559
b 0.760850 0.898359
c 0.503980 0.902335

You can transpose the DataFrame (swap rows and columns) with similar syntax to a NumPy array:

In [43]:

frame3.T

Out[43]:
2000 2001 2002
Nevada NaN 2.4 2.9
Ohio 1.5 1.7 3.6

The keys in the inner dicts are combined and sorted to form the index in the result. This isn’t true if an explicit index is specified:

In [44]:

pd.DataFrame(pop, index=[2001, 2002, 2003])

Out[44]:
Nevada Ohio
2001 2.4 1.7
2002 2.9 3.6
2003 NaN NaN

Dicts of Series are treated in much the same way:

In [45]:

pdata = {'Ohio': frame3['Ohio'][:-1],
         'Nevada': frame3['Nevada'][:2]}
pd.DataFrame(pdata)

Out[45]:
Nevada Ohio
2000 NaN 1.5
2001 2.4 1.7

If a DataFrame’s index and columns have their name attributes set, these will also be displayed:

In [46]:

frame3.index.name = 'year'
frame3.columns.name = 'state'
frame3

Out[46]:
state Nevada Ohio
year
2000 NaN 1.5
2001 2.4 1.7
2002 2.9 3.6

As with Series, the values attribute returns the data contained in the DataFrame as a two-dimensional ndarray:

In [47]:

frame3.values

Out[47]:

array([[ nan,  1.5],
       [ 2.4,  1.7],
       [ 2.9,  3.6]])

In [48]:

type(frame3.values)

Out[48]:

numpy.ndarray

If the DataFrame’s columns are different dtypes, the dtype of the values array will be chosen to accommodate all of the columns:

In [49]:

frame2.values

Out[49]:

array([[2000, 'Ohio', 1.5, nan],
       [2001, 'Ohio', 1.7, -1.2],
       [2002, 'Ohio', 3.6, nan],
       [2001, 'Nevada', 2.4, -1.5],
       [2002, 'Nevada', 2.9, -1.7],
       [2003, 'Nevada', 3.2, nan]], dtype=object)

In [50]:

type(frame2.values)

Out[50]:

numpy.ndarray

The Pandas Index Object

Index objects are responsible for holding the axis labels and other metadata (like the axis name or names). Any array or other sequence of labels you use when constructing a Series or DataFrame is internally converted to an Index:

In [51]:

obj = pd.Series(range(3), index=['a', 'b', 'c'])
index = obj.index
index

Out[51]:

Index(['a', 'b', 'c'], dtype='object')

In [52]:

index[1:]

Out[52]:

Index(['b', 'c'], dtype='object')

Index objects are immutable and thus can’t be modified by the user:

In [53]:

index[1] = 'd'  # TypeError

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-53-c25f1f037188> in <module>()
----> 1 index[1] = 'd'  # TypeError

~/.local/lib/python3.6/site-packages/pandas/core/indexes/base.py in __setitem__(self, key, value)
   1722
   1723     def __setitem__(self, key, value):
-> 1724         raise TypeError("Index does not support mutable operations")
   1725
   1726     def __getitem__(self, key):

TypeError: Index does not support mutable operations

Immutability makes it safer to share Index objects among data structures:

In [54]:

labels = pd.Index(np.arange(3))
labels

Out[54]:

Int64Index([0, 1, 2], dtype='int64')

In [55]:

obj2 = pd.Series([1.5, -2.5, 0], index=labels)
obj2

Out[55]:

0    1.5
1   -2.5
2    0.0
dtype: float64

In addition to being array-like, an Index also behaves like a fixed-size set:

In [56]:

frame3

Out[56]:
state Nevada Ohio
year
2000 NaN 1.5
2001 2.4 1.7
2002 2.9 3.6 
In [57]:

frame3.columns

Out[57]:

Index(['Nevada', 'Ohio'], dtype='object', name='state')

In [58]:

'Ohio' in frame3.columns

Out[58]:

True

In [59]:

2003 in frame3.index

Out[59]:

False

Caution: Unlike Python sets, a pandas Index can contain duplicate labels:

In [60]:

dup_labels = pd.Index(['foo', 'foo', 'bar', 'bar'])
dup_labels

Out[60]:

Index(['foo', 'foo', 'bar', 'bar'], dtype='object')

Selections with duplicate labels will select all occurrences of that label.

Some Index methods and properties

Method Description
append Concatenate with additional Index objects, producing a new Index
difference Compute set difference as an Index
intersection Compute set intersection
union Compute set union
isin Compute boolean array indicating whether each value is contained in the passed collection
delete Compute new Index with element at index i deleted
drop Compute new Index by deleting passed values
insert Compute new Index by inserting element at index i
is_monotonic Returns True if each element is greater than or equal to the previous element
is_unique Returns True if the Index has no duplicate values
unique Compute the array of unique values in the Index

For example:

In [61]:

indA = pd.Index([1, 3, 5, 7, 9])
indB = pd.Index([2, 3, 5, 7, 11])
indA.is_monotonic

Out[61]:

True

In [62]:

indA.intersection(indB)

Out[62]:

Int64Index([3, 5, 7], dtype='int64')

Essential Functionality

As throughout this class, we’ll focus on the most important features, leaving the less common (i.e., more esoteric) things for you to explore on your own.

Reindexing

An important method on pandas objects is reindex, which means to create a new object with the data conformed to a new index. Consider an example:

In [63]:

obj = pd.Series([4.5, 7.2, -5.3, 3.6], index=['d', 'b', 'a', 'c'])
obj

Out[63]:

d    4.5
b    7.2
a   -5.3
c    3.6
dtype: float64

Calling reindex on this Series rearranges the data according to the new index, introducing missing values if any index values were not already present:

In [64]:

obj2 = obj.reindex(['a', 'b', 'c', 'd', 'e'])
obj2

Out[64]:

a   -5.3
b    7.2
c    3.6
d    4.5
e    NaN
dtype: float64

For ordered data like time series, it may be desirable to do some interpolation or filling of values when reindexing. The method option allows us to do this, using a method such as ffill, which forward-fills the values:

In [65]:

obj3 = pd.Series(['blue', 'purple', 'yellow'], index=[0, 2, 4])
obj3

Out[65]:

0      blue
2    purple
4    yellow
dtype: object

In [66]:

obj3.reindex(range(6), method='ffill')

Out[66]:

0      blue
1      blue
2    purple
3    purple
4    yellow
5    yellow
dtype: object

In [67]:

# We can also backfill
obj3.reindex(range(6), method='bfill')

Out[67]:

0      blue
1    purple
2    purple
3    yellow
4    yellow
5       NaN
dtype: object

In [68]:

# or fill with a specified value
obj3.reindex(range(6), fill_value='white')

Out[68]:

0      blue
1     white
2    purple
3     white
4    yellow
5     white
dtype: object

With DataFrame, reindex can alter either the (row) index, columns, or both. When passed only a sequence, it reindexes the rows in the result:

In [69]:

frame = pd.DataFrame(np.arange(9).reshape((3, 3)),
                     index=['a', 'c', 'd'],
                     columns=['Ohio', 'Texas', 'California'])
frame

Out[69]:
Ohio Texas California
a 0 1 2
c 3 4 5
d 6 7 8 
In [70]:

frame2 = frame.reindex(['a', 'b', 'c', 'd'])
frame2

Out[70]:
Ohio Texas California
a 0.0 1.0 2.0
b NaN NaN NaN
c 3.0 4.0 5.0
d 6.0 7.0 8.0

The columns can be reindexed with the columns keyword:

In [71]:

states = ['Texas', 'Utah', 'California']
frame.reindex(columns=states)

Out[71]:
Texas Utah California
a 1 NaN 2
c 4 NaN 5
d 7 NaN 8

Dropping Entries from an Axis

Dropping one or more entries from an axis is easy if you already have an index array or list without those entries. As that can require a bit of munging and set logic, the drop method will return a new object with the indicated value or values deleted from an axis:

In [72]:

obj = pd.Series(np.arange(5.), index=['a', 'b', 'c', 'd', 'e'])
obj

Out[72]:

a    0.0
b    1.0
c    2.0
d    3.0
e    4.0
dtype: float64

In [73]:

new_obj = obj.drop('c')
new_obj

Out[73]:

a    0.0
b    1.0
d    3.0
e    4.0
dtype: float64

In [74]:

obj.drop(['d', 'c'])

Out[74]:

a    0.0
b    1.0
e    4.0
dtype: float64

In [75]:

obj

Out[75]:

a    0.0
b    1.0
c    2.0
d    3.0
e    4.0
dtype: float64

With DataFrame, index values can be deleted from either axis. To illustrate this, we first create an example DataFrame:

In [76]:

data = pd.DataFrame(np.arange(16).reshape((4, 4)),
                    index=['Ohio', 'Colorado', 'Utah', 'New York'],
                    columns=['one', 'two', 'three', 'four'])
data

Out[76]:
one two three four
Ohio 0 1 2 3
Colorado 4 5 6 7
Utah 8 9 10 11
New York 12 13 14 15

Calling drop with a sequence of labels will drop values from the row labels (axis 0):

In [77]:

data.drop(['Colorado', 'Ohio'])

Out[77]:
one two three four
Utah 8 9 10 11
New York 12 13 14 15

You can drop values from the columns by passing axis=1 or axis=‘columns’:

In [78]:

data.drop('two', axis=1)

Out[78]:
one three four
Ohio 0 2 3
Colorado 4 6 7
Utah 8 10 11
New York 12 14 15 
In [79]:

data.drop(['two', 'four'], axis='columns')

Out[79]:
one three
Ohio 0 2
Colorado 4 6
Utah 8 10
New York 12 14

Many functions, like drop, which modify the size or shape of a Series or DataFrame, can manipulate an object in-place without returning a new object:

In [80]:

obj.drop('c', inplace=True)
obj

Out[80]:

a    0.0
b    1.0
d    3.0
e    4.0
dtype: float64

Be careful with the inplace, as it destroys any data that is dropped.

Indexing, Selection, and Filtering

Series indexing (obj[…]) works analogously to NumPy array indexing, except you can use the Series’s index values instead of only integers. Here are some examples of this:

In [81]:

obj = pd.Series(np.arange(4.), index=['a', 'b', 'c', 'd'])
obj

Out[81]:

a    0.0
b    1.0
c    2.0
d    3.0
dtype: float64

In [82]:

obj['b']

Out[82]:

1.0

In [83]:

obj[1]

Out[83]:

1.0

In [84]:

obj[2:4]

Out[84]:

c    2.0
d    3.0
dtype: float64

In [85]:

obj[['b', 'a', 'd']]

Out[85]:

b    1.0
a    0.0
d    3.0
dtype: float64

In [86]:

obj[[1, 3]]

Out[86]:

b    1.0
d    3.0
dtype: float64

In [87]:

obj[obj < 2]

Out[87]:

a    0.0
b    1.0
dtype: float64

Caution: Slicing with labels behaves differently than normal Python slicing in that the endpoint is inclusive:

In [88]:

obj['b':'c']

Out[88]:

b    1.0
c    2.0
dtype: float64

Setting using these methods modifies the corresponding section of the Series:

In [89]:

obj['b':'c'] = 5
obj

Out[89]:

a    0.0
b    5.0
c    5.0
d    3.0
dtype: float64

Indexing into a DataFrame is for retrieving one or more columns either with a single value or sequence:

In [90]:

data = pd.DataFrame(np.arange(16).reshape((4, 4)),
                    index=['Ohio', 'Colorado', 'Utah', 'New York'],
                    columns=['one', 'two', 'three', 'four'])
data

Out[90]:
one two three four
Ohio 0 1 2 3
Colorado 4 5 6 7
Utah 8 9 10 11
New York 12 13 14 15 
In [91]:

data['two']

Out[91]:

Ohio         1
Colorado     5
Utah         9
New York    13
Name: two, dtype: int64

In [92]:

data[['three', 'one']]

Out[92]:
three one
Ohio 2 0
Colorado 6 4
Utah 10 8
New York 14 12

Selection with loc and iloc

These slicing and indexing conventions can be a source of confusion. For example, if your Series has an explicit integer index, an indexing operation such as data[1] will use the explicit indices, while a slicing operation like data[1:3] will use the implicit Python-style index.

In [93]:

data = pd.Series(['a', 'b', 'c'], index=[1, 3, 5])
data

Out[93]:

1    a
3    b
5    c
dtype: object

In [94]:

# explicit index when indexing
data[1]

Out[94]:

'a'

In [95]:

# implicit index when slicing
data[1:3]

Out[95]:

3    b
5    c
dtype: object

Because of this potential confusion in the case of integer indexes, Pandas provides some special indexer attributes that explicitly expose certain indexing schemes. These are not functional methods, but attributes that expose a particular slicing interface to the data in the Series.

First, the loc attribute allows indexing and slicing that always references the explicit index:

In [96]:

data.loc[1]

Out[96]:

'a'

In [97]:

data.loc[1:3]

Out[97]:

1    a
3    b
dtype: object

The iloc attribute allows indexing and slicing that always references the implicit Python-style index:

In [98]:

data.iloc[1]

Out[98]:

'b'

In [99]:

data.iloc[1:3]

Out[99]:

3    b
5    c
dtype: object

One guiding principle of Python code is that “explicit is better than implicit.” The explicit nature of loc and iloc make them very useful in maintaining clean and readable code; especially in the case of integer indexes, I recommend using these both to make code easier to read and understand, and to prevent subtle bugs due to the mixed indexing/slicing convention.

Next up: Data selection in dataframes!