PCIE 调试过程记录

Posted 2022-08-22 smartvxworks

遇到的问题

• PCIE link不稳定
• 配置空间读写正常，Memory mapping空间读写异常

缘由

之前对PCIE的认识一直停留在概念的阶段，只知道是一个高速通讯协议，主要用于板内、板间的高速BUS。正好公司最近在调试一个PowerPC平台的PCIE BUS的BSP。需要一些PCIE的硬件、软件知识。下面通过解决实际问题过程的方法来进一步理解PCIE总线协议。但也仅仅限于工程应用（实际产品中调试、应用）层面。

具体解决过程

PCIE 物理层link 不稳定

启动u-boot后，可以看到PCIE link status信息log 

从log上分析：系统启动了3个PCIE controller，PCIE 号为：0.0.0 2.0.0 4.0.0。 
三个数字的具体含义为：0.0.0 ：Bus Number Device Number Function Number。 
这三个Host Bridge 下面分别挂了一个EP设备，PCIE 号为：1.0.0 、3.0.0 、5.0.0。 
正常看到这些PCIE bridge 和EP说明PCIE 读PCIE的配置空间是正确的，PCIE在physical 的link状态是OK的。 
可以使用pci header + PCIE号的方式查看Bridge和EP的Config 信息。 

 

若出现读取配置信息不稳定的情况，则说明PCIE link 不稳定。需要进一步排查硬件，软件辅助查看link status的方法是查看PCIE Host Bridge的link status的寄存器值。PCIE 规范里有一个LTSSM（Link Training and Status State Machine），各种status的code规范里都有定义。 
这个LTSSM在PCIE Extend Config Space里面。在P3041中的offset为0x404。 
可以使用命令：pci display.b 0.0.0 404查看LTSSM状态。 
正常情况下的输出： 

PCIE 初始化完成后会进入L0状态。异常状态见PCIE link 异常log。 
物理层link 不稳定，怀疑以下原因： 
- 高速串行信号质量问题 
- Serdes电源问题 
- 时钟问题

Serdes电源经过测量后，基本排除。 
高速信号完整性问题经过示波器测量后，也基本排除。 
时钟问题，最开始只是简单测试了下时钟的频率和幅值。没有实质性去分析时钟的jitter等时钟质量问题。后分析原理图发现，pcie的Host和EP端使用了不同的pcie时钟IC，也就是说使用了非同源时钟。在没有详细查看pcie specification关于pcie时钟的情况下，将host和EP端的时钟改为同源时钟（当然是选择飞线喽），pcie物理层link问题解决。 
项目没有那么紧张的时候，反思此事，仔细查看pcie specification中关于时钟的要求，并没有要求Host和EP端必须使用同源时钟。只是对同源、非同源时钟的精度做了要求。 
同源情况下要求100M. +-600ppm 
非同源时钟要求100M. +-300ppm 
哦，原来是这样，接着去看原理图中选用的PCIE时钟IC，发现EP端的pcie时钟IC为25M输入然后倍频出来100M时钟的，且使用了一个带有SSC功能的IC，原理配配置上打开了该功能。根据pcie规范中介绍的，该功能 
为pcie一项技术，该功能为了减少板级EMI对时钟做了特殊能量扩散处理。有SSC功能的时钟超出300ppm的抖动，但在600ppm内。 
这样得出一个结论，我们使用了非同源的设计，EP端使用的pcie时钟IC的jitter超出了300ppm。看来不能稳定的link也是情理之中。这样看来就有另一种解决方法，把EP端PCIE侧的SSC功能关闭，去实践发现将SSC功能关闭后，可以正常稳定link。 
我们使用的pcie时钟芯片支持SSC且配置打开，时钟ppm在300到600之间。又非同源，因此link不稳定。将此功能关闭，时钟在300ppm，因此可以稳定link。

解决方法： 
- 飞线统一成同一时钟源。 
- 将EP端的时钟IC的SSC功能关闭。

PCIE memap空间读写异常

问题：

• pcie可以正常读写配置空间，但无法正常读写memap 空间

最后定位问题： 
一个address是36bit的，但软件定义为一个32bit的变量，从而导致软件读写pcie memap空间失败（读错了位置）。 
解决方法：fixed the software bug。☺

这个问题定位主要需要一些关于PICE的一些地址的知识，要理解他们之间的关系。

附录

$$$develop from single control storeage based on P3041
$$$Hit any key to stop autoboot:  0 
=> pci display.b 0.0.0 400
00000400: 00 00 00 00 33 00 00 00 e2 04 00 00 00 00 00 00
00000410: 01 00 00 00 19 00 00 00 00 00 00 00 40 40 96 96
00000420: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000430: 00 00 00 00 00 00 00 00 5c 41 c2 00 00 00 00 00
=> pci display.b 0.0.0 400
00000400: 00 00 00 00 02 00 00 00 e2 04 00 00 00 00 00 00
00000410: 01 00 00 00 19 00 00 00 00 00 00 00 40 40 96 96
00000420: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000430: 00 00 00 00 00 00 00 00 5c 41 c2 00 00 00 00 00
=> pci display.b 0.0.0 400
00000400: 00 00 00 00 06 00 00 00 e2 04 00 00 00 00 00 00
00000410: 01 00 00 00 19 00 00 00 00 00 00 00 40 40 96 96
00000420: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000430: 00 00 00 00 00 00 00 00 5c 41 c2 00 00 00 00 00
=> pci display.b 0.0.0 400
00000400: 00 00 00 00 16 00 00 00 e2 04 00 00 00 00 00 00
00000410: 01 00 00 00 19 00 00 00 00 00 00 00 40 40 96 96
00000420: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000430: 00 00 00 00 00 00 00 00 5c 41 c2 00 00 00 00 00
=> pci display.b 0.0.0 400
00000400: 00 00 00 00 49 00 00 00 e2 04 00 00 00 00 00 00
00000410: 01 00 00 00 19 00 00 00 00 00 00 00 40 40 96 96
00000420: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000430: 00 00 00 00 00 00 00 00 5c 41 c2 00 00 00 00 00
=> pci display.b 0.0.0 400
00000400: 00 00 00 00 49 00 00 00 e2 04 00 00 00 00 00 00
00000410: 01 00 00 00 19 00 00 00 00 00 00 00 40 40 96 96
00000420: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000430: 00 00 00 00 00 00 00 00 5c 41 c2 00 00 00 00 00
=> pci display.b 0.0.0 400
00000400: 00 00 00 00 49 00 00 00 e2 04 00 00 00 00 00 00
00000410: 01 00 00 00 19 00 00 00 00 00 00 00 40 40 96 96
00000420: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000430: 00 00 00 00 00 00 00 00 5c 41 c2 00 00 00 00 00
=> pci display.b 0.0.0 400
00000400: 00 00 00 00 33 00 00 00 e2 04 00 00 00 00 00 00
00000410: 01 00 00 00 19 00 00 00 00 00 00 00 40 40 96 96
00000420: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000430: 00 00 00 00 00 00 00 00 5c 41 c2 00 00 00 00 00
=> pci display.b 0.0.0 400
00000400: 00 00 00 00 49 00 00 00 e2 04 00 00 00 00 00 00
00000410: 01 00 00 00 19 00 00 00 00 00 00 00 40 40 96 96
00000420: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000430: 00 00 00 00 00 00 00 00 5c 41 c2 00 00 00 00 00
=> pci display.b 0.0.0 400
00000400: 00 00 00 00 37 00 00 00 e2 04 00 00 00 00 00 00
00000410: 02 00 00 00 19 00 00 00 00 00 00 00 40 40 96 96
00000420: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000430: 00 00 00 00 00 00 00 00 5c 41 c2 00 00 00 00 00
=> pci display.b 0.0.0 400
00000400: 00 00 00 00 02 00 00 00 e2 04 00 00 00 00 00 00
00000410: 01 00 00 00 19 00 00 00 00 00 00 00 40 40 96 96
00000420: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000430: 00 00 00 00 00 00 00 00 5c 41 c2 00 00 00 00 00
=> pci display.b 0.0.0 400
00000400: 00 00 00 00 02 00 00 00 e2 04 00 00 00 00 00 00
00000410: 01 00 00 00 19 00 00 00 00 00 00 00 40 40 96 96
00000420: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000430: 00 00 00 00 00 00 00 00 5c 41 c2 00 00 00 00 00
=> pci display.b 0.0.0 400
00000400: 00 00 00 00 49 00 00 00 e2 04 00 00 00 00 00 00
00000410: 01 00 00 00 19 00 00 00 00 00 00 00 40 40 96 96
00000420: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000430: 00 00 00 00 00 00 00 00 5c 41 c2 00 00 00 00 00
=> pci display.b 0.0.0 400
00000400: 00 00 00 00 02 00 00 00 e2 04 00 00 00 00 00 00
00000410: 01 00 00 00 19 00 00 00 00 00 00 00 40 40 96 96
00000420: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000430: 00 00 00 00 00 00 00 00 5c 41 c2 00 00 00 00 00
=> pci display.b 0.0.0 400
00000400: 00 00 00 00 16 00 00 00 e2 04 00 00 00 00 00 00
00000410: 01 00 00 00 19 00 00 00 00 00 00 00 40 40 96 96
00000420: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000430: 00 00 00 00 00 00 00 00 5c 41 c2 00 00 00 00 00
=> pci display.b 0.0.0 400
00000400: 00 00 00 00 49 00 00 00 e2 04 00 00 00 00 00 00
00000410: 01 00 00 00 19 00 00 00 00 00 00 00 40 40 96 96
00000420: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000430: 00 00 00 00 00 00 00 00 5c 41 c2 00 00 00 00 00
=> pci display.b 0.0.0 400
00000400: 00 00 00 00 16 00 00 00 e2 04 00 00 00 00 00 00
00000410: 00 01 00 00 19 00 00 00 00 00 00 00 40 40 96 96
00000420: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000430: 00 00 00 00 00 00 00 00 5c 41 c2 00 00 00 00 00
=> pci display.b 0.0.0 400
00000400: 00 00 00 00 02 00 00 00 e2 04 00 00 00 00 00 00
00000410: 01 00 00 00 19 00 00 00 00 00 00 00 40 40 96 96
00000420: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000430: 00 00 00 00 00 00 00 00 5c 41 c2 00 00 00 00 00
=> pci display.b 0.0.0 400
00000400: 00 00 00 00 02 00 00 00 e2 04 00 00 00 00 00 00
00000410: 01 00 00 00 19 00 00 00 00 00 00 00 40 40 96 96
00000420: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000430: 00 00 00 00 00 00 00 00 5c 41 c2 00 00 00 00 00
=> pci display.b 0.0.0 400
00000400: 00 00 00 00 02 00 00 00 e2 04 00 00 00 00 00 00
00000410: 02 01 00 00 19 00 00 00 00 00 00 00 40 40 96 96
00000420: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000430: 00 00 00 00 00 00 00 00 5c 41 c2 00 00 00 00 00
=> pci display.b 0.0.0 400
00000400: 00 00 00 00 06 00 00 00 e2 04 00 00 00 00 00 00
00000410: 01 00 00 00 19 00 00 00 00 00 00 00 40 40 96 96
00000420: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000430: 00 00 00 00 00 00 00 00 5c 41 c2 00 00 00 00 00
=> pci display.b 0.0.0 400
00000400: 00 00 00 00 35 00 00 00 e2 04 00 00 00 00 00 00
00000410: 00 01 00 00 19 00 00 00 00 00 00 00 40 40 96 96
00000420: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000430: 00 00 00 00 00 00 00 00 5c 41 c2 00 00 00 00 00
=> pci display.b 0.0.0 400
00000400: 00 00 00 00 35 00 00 00 e2 04 00 00 00 00 00 00
00000410: 01 00 00 00 19 00 00 00 00 00 00 00 40 40 96 96
00000420: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000430: 00 00 00 00 00 00 00 00 5c 41 c2 00 00 00 00 00
=> pci display.b 0.0.0 400
00000400: 00 00 00 00 49 00 00 00 e2 04 00 00 00 00 00 00
00000410: 01 00 00 00 19 00 00 00 00 00 00 00 40 40 96 96
00000420: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000430: 00 00 00 00 00 00 00 00 5c 41 c2 00 00 00 00 00
=> pci display.b 0.0.0 400
00000400: 00 00 00 00 01 00 00 00 e2 04 00 00 00 00 00 00
00000410: 01 00 00 00 19 00 00 00 00 00 00 00 40 40 96 96
00000420: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000430: 00 00 00 00 00 00 00 00 5c 41 c2 00 00 00 00 00
=> pci display.b 0.0.0 400
00000400: 00 00 00 00 49 00 00 00 e2 04 00 00 00 00 00 00
00000410: 01 00 00 00 19 00 00 00 00 00 00 00 40 40 96 96
00000420: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000430: 00 00 00 00 00 00 00 00 5c 41 c2 00 00 00 00 00
=> pci display.b 0.0.0 400
00000400: 00 00 00 00 49 00 00 00 e2 04 00 00 00 00 00 00
00000410: 01 00 00 00 19 00 00 00 00 00 00 00 40 40 96 96
00000420: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000430: 00 00 00 00 00 00 00 00 5c 41 c2 00 00 00 00 00
=> pci display.b 0.0.0 400
00000400: 00 00 00 00 49 00 00 00 e2 04 00 00 00 00 00 00
00000410: 01 00 00 00 19 00 00 00 00 00 00 00 40 40 96 96
00000420: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000430: 00 00 00 00 00 00 00 00 5c 41 c2 00 00 00 00 00
=> pci display.b 0.0.0 400
00000400: 00 00 00 00 49 00 00 00 e2 04 00 00 00 00 00 00
00000410: 01 00 00 00 19 00 00 00 00 00 00 00 40 40 96 96
00000420: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000430: 00 00 00 00 00 00 00 00 5c 41 c2 00 00 00 00 00
=> pci display.b 0.0.0 400
00000400: 00 00 00 00 02 00 00 00 e2 04 00 00 00 00 00 00
00000410: 01 00 00 00 19 00 00 00 00 00 00 00 40 40 96 96
00000420: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000430: 00 00 00 00 00 00 00 00 5c 41 c2 00 00 00 00 00
=> pci display.b 0.0.0 400
00000400: 00 00 00 00 16 00 00 00 e2 04 00 00 00 00 00 00
00000410: 01 00 00 00 19 00 00 00 00 00 00 00 40 40 96 96
00000420: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000430: 00 00 00 00 00 00 00 00 5c 41 c2 00 00 00 00 00
=> pci display.b 0.0.0 400
00000400: 00 00 00 00 02 00 00 00 e2 04 00 00 00 00 00 00
00000410: 01 00 00 00 19 00 00 00 00 00 00 00 40 40 96 96
00000420: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000430: 00 00 00 00 00 00 00 00 5c 41 c2 00 00 00 00 00
=> pci display.b 0.0.0 400
00000400: 00 00 00 00 02 00 00 00 e2 04 00 00 00 00 00 00
00000410: 01 00 00 00 19 00 00 00 00 00 00 00 40 40 96 96
00000420: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000430: 00 00 00 00 00 00 00 00 5c 41 c2 00 00 00 00 00
=> pci display.b 0.0.0 400
00000400: 00 00 00 00 49 00 00 00 e2 04 00 00 00 00 00 00
00000410: 01 00 00 00 19 00 00 00 00 00 00 00 40 40 96 96
00000420: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000430: 00 00 00 00 00 00 00 00 5c 41 c2 00 00 00 00 00
=> pci display.b 0.0.0 400
00000400: 00 00 00 00 49 00 00 00 e2 04 00 00 00 00 00 00
00000410: 01 00 00 00 19 00 00 00 00 00 00 00 40 40 96 96
00000420: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000430: 00 00 00 00 00 00 00 00 5c 41 c2 00 00 00 00 00
=> pci display.b 0.0.0 400
00000400: 00 00 00 00 02 00 00 00 e2 04 00 00 00 00 00 00
00000410: 01 00 00 00 19 00 00 00 00 00 00 00 40 40 96 96
00000420: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000430: 00 00 00 00 00 00 00 00 5c 41 c2 00 00 00 00 00
=> pci display.b 0.0.0 400
00000400: 00 00 00 00 06 00 00 00 e2 04 00 00 00 00 00 00
00000410: 01 00 00 00 19 00 00 00 00 00 00 00 40 40 96 96
00000420: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000430: 00 00 00 00 00 00 00 00 5c 41 c2 00 00 00 00 00
=> pci display.b 0.0.0 400
00000400: 00 00 00 00 02 00 00 00 e2 04 00 00 00 00 00 00
00000410: 01 00 00 00 19 00 00 00 00 00 00 00 40 40 96 96
00000420: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000430: 00 00 00 00 00 00 00 00 5c 41 c2 00 00 00 00 00
=> pci display.b 0.0.0 400
00000400: 00 00 00 00 02 00 00 00 e2 04 00 00 00 00 00 00
00000410: 01 00 00 00 19 00 00 00 00 00 00 00 40 40 96 96
00000420: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000430: 00 00 00 00 00 00 00 00 5c 41 c2 00 00 00 00 00
=> pci display.b 0.0.0 400
00000400: 00 00 00 00 49 00 00 00 e2 04 00 00 00 00 00 00
00000410: 01 00 00 00 19 00 00 00 00 00 00 00 40 40 96 96
00000420: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000430: 00 00 00 00 00 00 00 00 5c 41 c2 00 00 00 00 00
=> pci display.b 0.0.0 400
00000400: 00 00 00 00 16 00 00 00 e2 04 00 00 00 00 00 00
00000410: 01 00 00 00 19 00 00 00 00 00 00 00 40 40 96 96
00000420: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000430: 00 00 00 00 00 00 00 00 5c 41 c2 00 00 00 00 00
=> pci display.b 0.0.0 400
00000400: 00 00 00 00 49 00 00 00 e2 04 00 00 00 00 00 00
00000410: 01 00 00 00 19 00 00 00 00 00 00 00 40 40 96 96
00000420: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000430: 00 00 00 00 00 00 00 00 5c 41 c2 00 00 00 00 00
=> pci display.b 0.0.0 400
00000400: 00 00 00 00 02 00 00 00 e2 04 00 00 00 00 00 00
00000410: 01 00 00 00 19 00 00 00 00 00 00 00 40 40 96 96
00000420: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000430: 00 00 00 00 00 00 00 00 5c 41 c2 00 00 00 00 00
=> pci display.b 0.0.0 400
00000400: 00 00 00 00 49 00 00 00 e2 04 00 00 00 00 00 00
00000410: 01 00 00 00 19 00 00 00 00 00 00 00 40 40 96 96
00000420: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000430: 00 00 00 00 00 00 00 00 5c 41 c2 00 00 00 00 00
=> pci display.b 0.0.0 400
00000400: 00 00 00 00 02 00 00 00 e2 04 00 00 00 00 00 00
00000410: 01 00 00 00 19 00 00 00 00 00 00 00 40 40 96 96
00000420: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000430: 00 00 00 00 00 00 00 00 5c 41 c2 00 00 00 00 00
=> pci display.b 0.0.0 400
00000400: 00 00 00 00 02 00 00 00 e2 04 00 00 00 00 00 00
00000410: 01 00 00 00 19 00 00 00 00 00 00 00 40 40 96 96
00000420: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000430: 00 00 00 00 00 00 00 00 5c 41 c2 00 00 00 00 00
=> pci display.b 0.0.0 400
00000400: 00 00 00 00 02 00 00 00 e2 04 00 00 00 00 00 00
00000410: 01 00 00 00 19 00 00 00 00 00 00 00 40 40 96 96
00000420: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000430: 00 00 00 00 00 00 00 00 5c 41 c2 00 00 00 00 00
=> pci display.b 0.0.0 400
00000400: 00 00 00 00 06 00 00 00 e2 04 00 00 00 00 00 00
00000410: 01 00 00 00 19 00 00 00 00 00 00 00 40 40 96 96
00000420: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000430: 00 00 00 00 00 00 00 00 5c 41 c2 00 00 00 00 00
=> pci display.b 0.0.0 400
00000400: 00 00 00 00 02 00 00 00 e2 04 00 00 00 00 00 00
00000410: 01 00 00 00 19 00 00 00 00 00 00 00 40 40 96 96
00000420: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000430: 00 00 00 00 00 00 00 00 5c 41 c2 00 00 00 00 00
=> pci display.b 0.0.0 400
00000400: 00 00 00 00 49 00 00 00 e2 04 00 00 00 00 00 00
00000410: 01 00 00 00 19 00 00 00 00 00 00 00 40 40 96 96
00000420: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000430: 00 00 00 00 00 00 00 00 5c 41 c2 00 00 00 00 00
=> pci display.b 0.0.0 400
00000400: 00 00 00 00 33 00 00 00 e2 04 00 00 00 00 00 00
00000410: 01 00 00 00 19 00 00 00 00 00 00 00 40 40 96 96
00000420: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000430: 00 00 00 00 00 00 00 00 5c 41 c2 00 00 00 00 00
=> pci display.b 0.0.0 400
00000400: 00 00 00 00 02 00 00 00 e2 04 00 00 00 00 00 00
00000410: 01 00 00 00 19 00 00 00 00 00 00 00 40 40 96 96
00000420: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00000430: 00 00 00 00 00 00 00 00 5c 41 c2 00 00 00 00 00

• PCIE空间的BAR
• CPU访问PCIE设备的高速外设（IO）的地址

• 应用层访问PCIE memap的地址

  来张图吧， 

• 

   

• 上图中，有几个关键的Address，Virtual Address这个大家比较好理解，但凡有一点在linux下开发经验的程序员都知道linux下的应用程序只能访问Virtual Address，那如果我们需要访问一些指定的Physical Address怎么办，这在driver开发当中十分常见。linux提供了mamap系统调用，用于Virtual Address和Physical Address的转换。另外，PICE设备在读写时使用PCIE的BAR地址，BAR地址是PCIE控制器以memap 方式读写PCIE EP设备时使用的地址，是PCIE协议使用的地址。

  我们以写PCIE EP的memap空间为例子说明一下地址之间的转换是怎么做的。PCIE的memap空间是OS分配的，一般是一个固定的地址和大小，然后OS启动后，PCIE的应用程序将PCIE的memap的Physical Address通过mamap系统调用映射为Virtual Address，然后应用程序就会通过此地址读写EP设备的memap空间，当CPU发出写EP设备的memap空间时，PCIE控制器会根据设置的BAR地址转换成相应的PICE协议要求的地址。BAR是在PCIE枚举的过程中设置的，一般通过枚举方式设置成合理的地址值，EP这一侧会根据配置的值将自身的memap地址空间映射到设置的地址上去。

• 举一个具体的例子： 
  CPU分配的PCIE Physical Address为0xc20000000，设置的BAR地址为0xe0000000。应用层APP实际使用的Virtual Address和OS有关系，不同的进程调用mamap映射为同一个Physical Address可能会产生不同的Virtual Address值。

  总结

  应用层调试方法，config space配置、memap空间的理解 
  Pcie控制器的配置、使用 
  从实际调试PCIE设备的过程看，设备调试和使用主要涉及到两个方面：config space和MMIO space。配置空间主要是用于配置PCIE设备。很多PCI设备仅仅支持64字节的配置空间。PCI和PCIe配置空间的区别如下： 
  PCI/PCI-X和PCIe设备还扩展了0x40和0xFF这段配置空间，这段空间主要存放一些与MSI或者MSI-X 中断机制相关的Capability结构。其中所有能够提交中断请求的PCIe设备，必须支持MSI或者MSI-X 中断机制相关的Capability结构。 
  PCIe设备还支持0x100 -0xFFF这段扩展配置空间。PCIe设备的扩展配置空间最大为4KB，在PCIe总线的扩展配置空间中，存放PCIe所独有的一些Capability结构，而PCI设备不能使用这段空间。 
  在x86处理器中，使用CONFIG_ADDRESS寄存器与CONFIG_DATA寄存器访问0x00-0xFF，而使用ECAM方式访问0x000-0xFFF这段空间；而在PowerPC处理器中，可以使用CFG_DATA和CFG_ADDR寄存器访问0x000-0xFFF。 
  MMIO(Memory mapping I/O)即内存映射I/O，它是PCI规范的一部分，I/O设备被放置在内存空间而不是I/O空间。从处理器的角度看，内存映射I/O后系统设备访问起来和内存一样。这样访问AGP/PCI-E显卡上的帧缓存，Bios，PCI设备就可以使用读写内存一样的汇编指令完成，简化了程序设计的难度和接口的复杂性。I/O作为CPU和外设交流的一个渠道，主要分为两种，一种是Port I/O，一种是MMIO(Memory mapping I/O)。 
  底层使用pcie接口的软件移植需要考虑以下方面：

• 确定address bus的宽度，是32bit还是64bit或者其他。
• 确定CPU的大小端，以及PCIE的大小端问题。
• check下PICE max payload问题。