38. Pandas的时间序列数据-时序处理

之前的章节在介绍时序数据的时候基本上时间作为index,提供values值产生了Series数据,一般时序index和values一一对齐,现实使用pandas处理数据会发现数据value和index存在位置差,需要将values前移或整体后移,这个时候可以借助pandas的shift函数来移动一下数值数据values.有的时候会发现index过密,想缩短时间学列的间隔值,这个时候可以考虑用asfreq和resample来调整时间序列的间隔。

38.1 移动对齐

这里是处理上述的第一个问题,也就是数据和时间序列位置存在对齐的问题,可以移动values数据,可以移动index即时间序列。

import numpy as np
import pandas as pd
v = [5, 4, 3, 2, 1]
t0 = pd.Series(v, index = pd.date_range('2018-12-19', periods = 5))
print t0
t1 = t0.shift(1)
print t1
t2 = t1.fillna(method = "bfill")
print t2

程序执行结果:

2018-12-19    5 # t0
2018-12-20    4
2018-12-21    3
2018-12-22    2
2018-12-23    1
Freq: D, dtype: int64
2018-12-19    NaN # t1
2018-12-20    5.0
2018-12-21    4.0
2018-12-22    3.0
2018-12-23    2.0
Freq: D, dtype: float64
2018-12-19    5.0 # t2
2018-12-20    5.0
2018-12-21    4.0
2018-12-22    3.0
2018-12-23    2.0
Freq: D, dtype: float64

从结果可以看出,series的values整体向下移动了一下,而index没有发生变化。对于移动后的series可以使用fillna函数来简单清洗一下

shift函数默认的freq参数为'D'即以天作为单位,通过修改freq的值,可以进行其他的修改,即实现对index的移动。

  • freq = 'B', 工作日为调整单位。
import numpy as np
import pandas as pd
v = [5, 4, 3, 2, 1]
t0 = pd.Series(v, index = pd.date_range('2018-12-19', periods = 5))
print t0
t1 = t0.shift(1, freq = "B")
print t1
  • freq = "H", 以小时为单位修改index时间。
import numpy as np
import pandas as pd
v = [5, 4, 3, 2, 1]
t0 = pd.Series(v, index = pd.date_range('2018-12-19 10:30:50', periods = 5))
print t0
t1 = t0.shift(1, freq = "2H")
print t1

程序执行结果:

2018-12-19 10:30:50    5 # t0
2018-12-20 10:30:50    4
2018-12-21 10:30:50    3
2018-12-22 10:30:50    2
2018-12-23 10:30:50    1
Freq: D, dtype: int64
2018-12-19 12:30:50    5 # t1
2018-12-20 12:30:50    4
2018-12-21 12:30:50    3
2018-12-22 12:30:50    2
2018-12-23 12:30:50    1
Freq: D, dtype: int64
import numpy as np
import pandas as pd
from pandas import DateOffset
v = [5, 4, 3, 2, 1]
t0 = pd.Series(v, index = pd.date_range('2018-12-19 10:30:50', periods = 5))
print t0
t1 = t0.shift(1, DateOffset(hours = 0.5))
print t1

程序执行结果:

2018-12-19 10:30:50    5 # t0
2018-12-20 10:30:50    4
2018-12-21 10:30:50    3
2018-12-22 10:30:50    2
2018-12-23 10:30:50    1
Freq: D, dtype: int64
2018-12-19 11:00:50    5 # t1
2018-12-20 11:00:50    4
2018-12-21 11:00:50    3
2018-12-22 11:00:50    2
2018-12-23 11:00:50    1
Freq: D, dtype: int64

可以看出时间序列整体往后调整了半小时。

38.2 时间频率调整

这里是处理的第二个问题,即原有的时间需类过密或者过稀,可以通过asfreq来调整时间序列的间隔时间,需要注意的是调整后数据是否能对应的上的问题,可采用均值、插值来填充调整后的时间序列所对应的数据。

import numpy as np
import pandas as pd
c = 31 * 24
v = np.arange(c)
t0 = pd.Series(v, index = pd.date_range('2018-12-19 10:00:00', periods = c, freq = "2H"))
print t0[:13]
t1 = t0.asfreq("D")
print t1[:13]

程序的执行结果:

2018-12-19 10:00:00     0
2018-12-19 12:00:00     1
2018-12-19 14:00:00     2
2018-12-19 16:00:00     3
2018-12-19 18:00:00     4
2018-12-19 20:00:00     5
2018-12-19 22:00:00     6
2018-12-20 00:00:00     7
2018-12-20 02:00:00     8
2018-12-20 04:00:00     9
2018-12-20 06:00:00    10
2018-12-20 08:00:00    11
2018-12-20 10:00:00    12
Freq: 2H, dtype: int64
2018-12-19 10:00:00      0
2018-12-20 10:00:00     12
2018-12-21 10:00:00     24
2018-12-22 10:00:00     36
2018-12-23 10:00:00     48
2018-12-24 10:00:00     60
2018-12-25 10:00:00     72
2018-12-26 10:00:00     84
2018-12-27 10:00:00     96
2018-12-28 10:00:00    108
2018-12-29 10:00:00    120
2018-12-30 10:00:00    132
2018-12-31 10:00:00    144
Freq: D, dtype: int64

通过asfreq函数,将原来的时间序列有间隔2小时变为了间隔一天,新生成的时间序列如果在原序列里有对应值,那么用原来的values,作为新时间序列的values,例如2018-12-20 10:00:00 12。但是如果调整后的时间序列没有原值能对应上,新时间序列里values会出现NaN。

import numpy as np
import pandas as pd
c = 31 * 24
v = np.arange(c)
t0 = pd.Series(v, index = pd.date_range('2018-12-19 10:00:00', periods = c, freq = "2H"))
print t0[:13]
t1 = t0.asfreq("H")
print t1[:13]

程序执行结果:

2018-12-19 10:00:00     0
2018-12-19 12:00:00     1
2018-12-19 14:00:00     2
2018-12-19 16:00:00     3
2018-12-19 18:00:00     4
2018-12-19 20:00:00     5
2018-12-19 22:00:00     6
2018-12-20 00:00:00     7
2018-12-20 02:00:00     8
2018-12-20 04:00:00     9
2018-12-20 06:00:00    10
2018-12-20 08:00:00    11
2018-12-20 10:00:00    12
Freq: 2H, dtype: int64
2018-12-19 10:00:00    0.0
2018-12-19 11:00:00    NaN
2018-12-19 12:00:00    1.0
2018-12-19 13:00:00    NaN
2018-12-19 14:00:00    2.0
2018-12-19 15:00:00    NaN
2018-12-19 16:00:00    3.0
2018-12-19 17:00:00    NaN
2018-12-19 18:00:00    4.0
2018-12-19 19:00:00    NaN
2018-12-19 20:00:00    5.0
2018-12-19 21:00:00    NaN
2018-12-19 22:00:00    6.0
Freq: H, dtype: float64

2018-12-19 11:00:00在原来的Series里t0,没有对应值,可以用fillna来处理填充。

import numpy as np
import pandas as pd
c = 31 * 24
v = np.arange(c)
t0 = pd.Series(v, index = pd.date_range('2018-12-19 10:00:00', periods = c, freq = "2H"))
print t0[:13]
t1 = t0.asfreq("H")
print t1[:13]
t2 = t1.fillna(method = "bfill")
print t2[:13]

程序执行结果:

2018-12-19 10:00:00     0
2018-12-19 12:00:00     1
2018-12-19 14:00:00     2
2018-12-19 16:00:00     3
2018-12-19 18:00:00     4
2018-12-19 20:00:00     5
2018-12-19 22:00:00     6
2018-12-20 00:00:00     7
2018-12-20 02:00:00     8
2018-12-20 04:00:00     9
2018-12-20 06:00:00    10
2018-12-20 08:00:00    11
2018-12-20 10:00:00    12
Freq: 2H, dtype: int64
2018-12-19 10:00:00    0.0
2018-12-19 11:00:00    NaN
2018-12-19 12:00:00    1.0
2018-12-19 13:00:00    NaN
2018-12-19 14:00:00    2.0
2018-12-19 15:00:00    NaN
2018-12-19 16:00:00    3.0
2018-12-19 17:00:00    NaN
2018-12-19 18:00:00    4.0
2018-12-19 19:00:00    NaN
2018-12-19 20:00:00    5.0
2018-12-19 21:00:00    NaN
2018-12-19 22:00:00    6.0
Freq: H, dtype: float64
2018-12-19 10:00:00    0.0
2018-12-19 11:00:00    1.0
2018-12-19 12:00:00    1.0
2018-12-19 13:00:00    2.0
2018-12-19 14:00:00    2.0
2018-12-19 15:00:00    3.0
2018-12-19 16:00:00    3.0
2018-12-19 17:00:00    4.0
2018-12-19 18:00:00    4.0
2018-12-19 19:00:00    5.0
2018-12-19 20:00:00    5.0
2018-12-19 21:00:00    6.0
2018-12-19 22:00:00    6.0
Freq: H, dtype: float64

或者在asfreq里使用method,例如:

import numpy as np
import pandas as pd
c = 31 * 24
v = np.arange(c)
t0 = pd.Series(v, index = pd.date_range('2018-12-19 10:00:00', periods = c, freq = "2H"))
print t0[:13]
t1 = t0.asfreq("H", method = "bfill")
print t1[:13]