Python学习之数据挖掘（三）

Pandas

    基础处理

        Pandas是什么？为什么用？

        核心数据结构

            DataFrame

            Panel

            Series

        基本操作

        运算

        画图

        文件的读取与存储

    高级处理

4.1Pandas介绍

    4.1.1 Pandas介绍 - 数据处理工具

        panel + data + analysis

        panel面板数据 - 计量经济学 三维数据

    4.1.2 为什么使用Pandas

        便捷的数据处理能力

        读取文件方便

        封装了Matplotlib、Numpy的画图和计算

    4.1.3 DataFrame

        结构：既有行索引，又有列索引的二维数组

        属性：

            shape

            index

            columns

            values

            T

        方法：

            head()

            tail()

        3 DataFrame索引的设置

            1）修改行列索引值

            2）重设索引

            3）设置新索引

        2 Panel

             DataFrame的容器

        3 Series

            带索引的一维数组

            属性

                index

                values

        总结：

            DataFrame是Series的容器

            Panel是DataFrame的容器

4.2 基本数据操作

    4.2.1 索引操作

        1）直接索引

            先列后行

        2）按名字索引

            loc

        3）按数字索引

            iloc

        4）组合索引

            数字、名字

    4.2.3 排序

        对内容排序

            dataframe

            series

        对索引排序

            dataframe

            series

4.3 DataFrame运算

    算术运算

    逻辑运算

        逻辑运算符

            布尔索引

        逻辑运算函数

            query()

            isin()

        统计运算

            min max mean median var std

             np.argmax()

            np.argmin()

        自定义运算

            apply(func, axis=0)True

                func:自定义函数

4.4 Pandas画图

    sr.plot()

4.5 文件读取与存储

    4.5.1 CSV

        pd.read_csv(path)

            usecols=

            names=

         dataframe.to_csv(path)

            columns=[]

            index=False

            header=False

    4.5.2 HDF5 、

        hdf5 存储 3维数据的文件

            key1 dataframe1二维数据

            key2 dataframe2二维数据

        pd.read_hdf(path, key=)

        df.to_hdf(path, key=)

    4.5.3 JSON

        pd.read_json(path)

            orient="records"

            lines=True

        df.to_json(patn)

            orient="records"

            lines=True

案例1：pandas数据帧DataFrame

# 数据帧DataFrame
import numpy as np
import pandas as pd
# 创建一个符合正态分布的10个股票5天的涨跌幅数据
stock_change = np.random.normal(0, 1, (10, 5))
print(pd.DataFrame(stock_change))
# 添加行索引
stock = ["股票{}".format(i) for i in range(10)]
print(pd.DataFrame(stock_change, index=stock))
# 添加列索引
date = pd.date_range(start="20180101", periods=5, freq="B")
data = pd.DataFrame(stock_change, index=stock, columns=date)
print(data)
# DataFrame的属性
print(data.shape)
# 行列表
print(data.index)
# 列列表
print(data.columns)
# ndarray
print(data.values)
# 转置
print(data.T)
# 前三条
print(data.head(3))
# 后三条
print(data.tail(3))
# 修改行列索引值
stock_ = ["股票_{}".format(i) for i in range(10)]
data.index = stock_
print(data.index)
# 重设索引
print(data.reset_index(drop=False))
# 设置新索引
df = pd.DataFrame({'month': [1, 4, 7, 10],
                    'year': [2012, 2014, 2013, 2014],
                    'sale':[55, 40, 84, 31]})
# 以月份设置新的索引
print(df.set_index("month", drop=True))
# 设置多个索引，以年和月份
new_df = df.set_index(["year", "month"])
print(new_df)
print(new_df.index)
print(new_df.index.names)
print(new_df.index.levels)

案例2：基本数据操作

# 基本数据操作
import pandas as pd
data = [[23.53,25.88,24.16,23.53],[22.8,23.78,23.53,22.80],[22.88,23.37,22.82,22.71]]
df = pd.DataFrame(data,columns=['open','high','close','low'],index=['2019-08-21','2019-08-20','2019-08-19'])
# 先列后行
print(df['open']['2019-08-19'])
# 先行后列
print(df.loc['2019-08-19','open'])
print(df.iloc[1, 0])
# 获取行第1天到第2天，['open', 'close', 'high', 'low']这个四个指标的结果
print(df.ix[:2, ['open', 'close', 'high', 'low']])
# 赋值操作
df.open = 100
df.iloc[1, 0] = 222
print(df)
print(df.sort_values(by=["high", "low"], ascending=False))
print(df.sort_index())
sr = df['high']
print(sr.sort_values(ascending=False).head())
print(sr.sort_index())
print(df["open"].add(3))
print(df.sub(100))
print(df["close"].sub(df["open"]).head())
print(df[df["high"] > 24])
print(df.query("close > 24 & low > 15"))
# 统计运算
print(df.describe())
print(df.max(axis=0))
print(df.idxmax(axis=0))
df["close"].sort_index().cumsum().plot()
#自定义运算
print(df.apply(lambda x: x.max() - x.min()))
# 散点图
df.plot(x="low", y="high", kind="scatter")

# 数据文件读取
# csv
import pandas as pd
pd.read_csv("./stock_day/stock_day.csv", usecols=["high", "low", "open", "close"]).head()
data = pd.read_csv("stock_day2.csv", names=["open", "high", "close", "low"])
# 保存'open'列的数据
data[:10].to_csv("test.csv", columns=["open"])
pd.read_csv("test.csv")
data[:10].to_csv("test.csv", columns=["open"], index=False, mode="a", header=False)
# hdf5
day_close = pd.read_hdf("./stock_data/day/day_close.h5")
day_close.to_hdf("test.h5", key="close")
print(pd.read_hdf("test.h5", key="close"))
day_open = pd.read_hdf("./stock_data/day/day_open.h5")
day_open.to_hdf("test.h5", key="open")
print(pd.read_hdf("test.h5", key="close").head())
# JSON
sa = pd.read_json("Sarcasm_Headlines_Dataset.json", orient="records", lines=True)
print(sa)
sa.to_json("test.json", orient="records", lines=True)