Bug #118506

【AF】【EVT3】【Monkey】106版本(529版)Monkey测试72小时后黑屏(4+27个APK)

Added by CD TEST-方永红 about 2 years ago. Updated almost 2 years ago.

Status:ASSIGNEDStart date:2023-06-05
Priority:NormalDue date:
Assignee:CD TPM-王祥林% Done:

0%

Category:黑屏
Target version:-
Need_Info:-- Found Version:106(529版)
Resolution:-- Degrated:--
Severity:Normal Verified Version:
Reproducibility:Rarely Fixed Version:
Test Type:Monkey Test Root cause:

Description

【执行命令】
monkey --pkg-blacklist-file /sdcard/blacklist.txt --ignore-crashes --ignore-timeouts --ignore-security-exceptions --monitor-native-crashes --ignore-native-crashes -s 20 --throttle 500 --pct-touch 30 --pct-motion 15 --pct-nav 15 --pct-majornav 15 --pct-syskeys 5 --pct-appswitch 10 --pct-anyevent 10 --ignore-crashes -v -v -v 4432000 2>/sdcard/error.txt 1>/sdcard/info.txt

【测试结果】
monkey未执行完所有事件就黑屏了

add_ADSP_size.patch Magnifier (2.85 KB) BSP组-RD3_TSCD 罗聪, 2023-06-14 20:36

add_ADSP_size-new.patch Magnifier (3.36 KB) BSP组-RD3_TSCD 罗聪, 2023-06-16 11:01

串口输出.txt Magnifier (10.8 KB) BSP组-RD3_TSCD 罗聪, 2023-06-16 11:01

History

#2 Updated by CD TEST-方永红 about 2 years ago

  • Category changed from SYSTEM to 黑屏

#3 Updated by CD SYSTEM-夏旭 about 2 years ago

1. 从dump解析出来kernel log看ADSP Crash,导致的问题

[277881.709717] type=1400 audit(1685818315.631:971552): avc: denied { kill } for comm="swapper/2" capability=5 scontext=u:r:kernel:s0 tcontext=u:r:kernel:s0 tclass=capability permissive=0
[277881.914762] Fatal error on adsp!
[277881.914853] adsp subsystem failure reason: DALInterruptController.c:388:Failed to register interrupt 16 in big image! Status: 30.
[277881.914900] subsys-restart: subsystem_restart_dev(): Restart sequence requested for adsp, restart_level = SYSTEM.
[277882.017039] Kernel panic - not syncing: subsys-restart: Resetting the SoC - adsp crashed.

2. 请adsp ownner 继续分析

3. 软件版本
ftp://cdiot@192.168.87.46/Pre_figure/ReleaseBuild/Pre_figure_turbox-c2130c-la1.1-qssi12-dev/user/20230529

#4 Updated by CD SYSTEM-胡兵 about 2 years ago

  • Assignee changed from CD SYSTEM-夏旭 to CD TPM-王祥林

#5 Updated by CD TPM-王祥林 about 2 years ago

  • Assignee changed from CD TPM-王祥林 to CD BSP汪凯

汪凯

帮忙看一下。

#6 Updated by BSP组-RD3_TSCD 罗聪 about 2 years ago

之前将log提供给qcom:
高通通过他们的工具分析出来目前出现crash的问题已经确认在adsp里面,他们那边也会会安排ADSP工程师进行检查。这个还是需要去继续追下高通。

on the crash dump, the error comes from  ADSP interrupt.  issue will be checked by ADSP engineer further.

高通同时也提供了ADSP里面crash的时候调用的堆栈。

Call Stacks
=========================
File : core/systemdrivers/InterruptController/src/qurt/DALInterruptController.c
File Info: /home/scm/jenkins/workspace/DailyBuild_for_IOT_6490/build-dir/Pre_figure_turbox-c2130c-la1.1-qssi12-dev-3688/BP-CODE/adsp_proc/core/systemdrivers/InterruptController/src/qurt/DALInterruptController.c

-000|qurt_exception_raise_nonfatal()
-001|InterruptController_RegisterInterrupt()
-002|PdcGpio_Register()
-003|GPIOIntr_RegisterIsr()
-004|GPIOInt_RegisterIsr()
-005|pciedrv_gpio_register_isr()
-006|pciedrv_core_gateclkreq()
-007|pciedrv_core_l1ssto_handler()
-008|pciedrv_proc_l1ss_to_cmd()
-009|pciedrv_msg_proc()
-010|pcie_osal_thread_entry()
-011|_WorkLoopExecute()
-012|qurt_trampoline()
-013|qurt_trampoline()
----|end of frame

这个我们可以自己去调查下这些堆栈的调用,看看是否能挖掘出来一些信息。

#7 Updated by BSP组-RD3_TSCD 罗聪 about 2 years ago

根据高通的最新回复,导致crash的问题是因为ADSP的内存不够导致的,并给出了一个参考文档和相关思路:


Zhiping Pang
    True    6/13/2023 3:14 AM    不是pcie模块的问题,而是大块内存耗尽了。ADSP侧只剩一些小内存块了。
所以,你们在出问题之前干啥了?

Zhiping Pang
    True    6/14/2023 1:07 AM    Please check KBA 210525013529. 
Please change the {adsp}\config\8250.adsp\cust_config.xml
<heap_size name="heap_size" value = "0x180000" />
Please add at least 1 Mega bytes.

我这边根据文档修改了一部分,但是文档里面部分内容没有看明白,已经把修改的patch和不明白的地方给高通了,也继续在和高通沟通同,希望他能够进行一个疑惑解答。
同时今天本地修改了部分代码,本地先验证下是否有效,是否能开机,如果可以,进行VB测试,看是否有改善,如果没有改善,则继续和高通查看patch内容,修改。

今天的patch已经上传附件

#8 Updated by BSP组-RD3_TSCD 罗聪 about 2 years ago

最新的aptch打入导致现在设备起不来,串口的log显示无法进入kernel。已经将完整的log和修改的patch给qcom了,他们那边也在分析!
最新的patch和启动log如下:
patch: add_ADSP_size-new.patch
LOG:串口输出.txt

#10 Updated by BSP组-RD3_TSCD 罗聪 about 2 years ago

  • Assignee changed from BSP组-RD3_TSCD 罗聪 to CD BSP 叶鹏

hi peng:
我们目前还是在尝试增加ADSP的的的内存去避免因为内存不足导致的crash。
我们修改ADSP的内存,尝试去扩大2M的内存和1M的内存patch都已经上传,修改方案都是将ADSP的地址范围进行偏移,从现象看我们修改是生效了的,因为修改后从串口log中会出现相关的报错,目前的修改还是起不来。分析应该是修改了内核和对应的uefipil.cfg文件可能匹配校验还有问题,我们需要去修改正确。

最新的patch如下《add_ADSP_size-new.patch》 之前添加2m内存,高通反馈内存有重复的问题,这次之修改了1m,但是还是不可以。
我也尝试不修改uefipil.cfg文件,但是还是起不来,内存还是有冲突,可能还需要去修改一些文件。
我这边也会继续和qcom进行沟通,看是否还需要有其他的地方进行修改。

谢谢!

#11 Updated by IoT scm about 2 years ago

Gerrit Merge Information
ID Project Branch Uploader
208534 general/slpi_proc Pre_figure_turbox-c2130c-la1.1-vendor-dev
BSP:System: Increase the memory size of adsp
Increase the memory size of adsp
TC-RID: 1201-0101106
IssueID: TS-R-DF-118506
Change-Id: I6edacc5d2ff4347ef31b09980a00abb61cacedb8

#12 Updated by IoT scm about 2 years ago

Gerrit Merge Information
ID Project Branch Uploader
208533 general/adsp_proc Pre_figure_turbox-c2130c-la1.1-vendor-dev
BSP:System: Increase the memory size of adsp
Increase the memory size of adsp
TC-RID: 1201-0101106
IssueID: TS-R-DF-118506
Change-Id: I179c2b9563af52e9a2262c786369a6d199cc6efe

#13 Updated by IoT scm about 2 years ago

Gerrit Merge Information
ID Project Branch Uploader
208420 general/boot_images Pre_figure_turbox-c2130c-la1.1-vendor-dev
BSP:System: Increase the memory size of adsp
Increase the memory size of adsp
TC-RID: 1201-0101106
IssueID: TS-R-DF-118506
Change-Id: Ieb1b3d9986ac8d1257b0723c3c6aa14273d08ddd

#14 Updated by IoT scm about 2 years ago

Gerrit Merge Information
ID Project Branch Uploader
208416 general/trustzone_images Pre_figure_turbox-c2130c-la1.1-vendor-dev
BSP:System: Increase the memory size of adsp
Increase the memory size of adsp
TC-RID: 1201-0101106
IssueID: TS-R-DF-118506
Change-Id: Id62fa0e6a6316543059c0ec6df9ac8ad064ee673

#15 Updated by IoT scm about 2 years ago

Gerrit Merge Information
ID Project Branch Uploader
208415 general/LINUX/android/vendor/qcom/proprietary Pre_figure_turbox-c2130c-la1.1-vendor-dev
BSP:System: Increase the memory size of adsp
Increase the memory size of adsp
TC-RID: 1201-0101106
IssueID: TS-R-DF-118506
Change-Id: I12db72e94266162ced2fc1b62729efcb9f1213ce

#16 Updated by IoT scm about 2 years ago

Gerrit Merge Information
ID Project Branch Uploader
209689 general/LINUX/android/vendor/qcom/proprietary Pre_figure_turbox-c2130c-la1.1-vendor-release_sorp
BSP:System: Increase the memory size of adsp
Increase the memory size of adsp
TC-RID: 1201-0101106
IssueID: TS-R-DF-118506
Change-Id: I12db72e94266162ced2fc1b62729efcb9f1213ce
(cherry picked from commit 3ced994d5824d2444dd1042d6fcf5fc9417cb94d)

#17 Updated by IoT scm about 2 years ago

Gerrit Merge Information
ID Project Branch Uploader
209690 general/trustzone_images Pre_figure_turbox-c2130c-la1.1-vendor-release_sorp
BSP:System: Increase the memory size of adsp
Increase the memory size of adsp
TC-RID: 1201-0101106
IssueID: TS-R-DF-118506
Change-Id: Id62fa0e6a6316543059c0ec6df9ac8ad064ee673
(cherry picked from commit 0709b21168f20d9ebada2564619870c856bbac40)

#18 Updated by IoT scm about 2 years ago

Gerrit Merge Information
ID Project Branch Uploader
209691 general/slpi_proc Pre_figure_turbox-c2130c-la1.1-vendor-release_sorp
BSP:System: Increase the memory size of adsp
Increase the memory size of adsp
TC-RID: 1201-0101106
IssueID: TS-R-DF-118506
Change-Id: I6edacc5d2ff4347ef31b09980a00abb61cacedb8
(cherry picked from commit 5273fd4ff83d73266fab8e41bc2220003a40e6ab)

#19 Updated by IoT scm about 2 years ago

Gerrit Merge Information
ID Project Branch Uploader
209692 general/adsp_proc Pre_figure_turbox-c2130c-la1.1-vendor-release_sorp
BSP:System: Increase the memory size of adsp
Increase the memory size of adsp
TC-RID: 1201-0101106
IssueID: TS-R-DF-118506
Change-Id: I179c2b9563af52e9a2262c786369a6d199cc6efe
(cherry picked from commit cdaa33fbaf60a6002baa9a4333c1cf35616cf7d4)

#20 Updated by IoT scm about 2 years ago

Gerrit Merge Information
ID Project Branch Uploader
209693 general/boot_images Pre_figure_turbox-c2130c-la1.1-vendor-release_sorp
BSP:System: Increase the memory size of adsp
Increase the memory size of adsp
TC-RID: 1201-0101106
IssueID: TS-R-DF-118506
Change-Id: Ieb1b3d9986ac8d1257b0723c3c6aa14273d08ddd
(cherry picked from commit 2c1d968820eabe5d1a0725a1ad3468be074a9dea)

#21 Updated by CD BSP 叶鹏 about 2 years ago

  • Status changed from New to RESOLVED

#22 Updated by IoT scm almost 2 years ago

Gerrit Merge Information
ID Project Branch Uploader
215153 general/kernel/msm-4.19 Pre_figure_turbox-c2130c-la1.1-vendor-dev
BSP:System: Reset the adsp service while dasp crashes
Reset the adsp service while dasp crashes
TC-RID: 1201-0101106
IssueID: TS-R-DF-118506
Change-Id: Ia3eb521f101ed40114ff09b8bfe02cfb93b0884c

#23 Updated by IoT scm almost 2 years ago

Gerrit Merge Information
ID Project Branch Uploader
215172 general/platform/vendor/thundercomm/apps/SlimbusEvent Pre_figure_turbox-c2130c-la1.1-vendor-dev
BSP:System: Reset the adsp service while dasp crashes
Reset the adsp service while dasp crashes
TC-RID: 1201-0101106
IssueID: TS-R-DF-118506
Change-Id: I04ab49431accfb87095928213a79ed2d4ac1a7d3

#24 Updated by IoT scm almost 2 years ago

Gerrit Merge Information
ID Project Branch Uploader
215775 general/vendor/thundercomm/products/sepolicy_vndr Pre_figure_turbox-c2130c-la1.1-vendor-dev
BSP:System: Reset the adsp service while dasp crashes
Add permissions for slimbus_event
TC-RID: 1201-0101106
IssueID: TS-R-DF-118506
Change-Id: I51e9ca6905bf8e62ad4790891bf2bc2235a8854c

#25 Updated by IoT scm almost 2 years ago

Gerrit Merge Information
ID Project Branch Uploader
215777 general/vendor/thundercomm/products/figure Pre_figure_turbox-c2130c-la1.1-vendor-dev
BSP:System: Reset the adsp service while dasp crashes
Reset the adsp service while dasp crashes
TC-RID: 1201-0101106
IssueID: TS-R-DF-118506
Change-Id: If2047665b061420fc5dc0b4d7e0649a80a8e65b6

#26 Updated by IoT scm almost 2 years ago

Gerrit Merge Information
ID Project Branch Uploader
216260 general/kernel/msm-4.19 Pre_figure_turbox-c2130c-la1.1-vendor-release_sorp
BSP:System: Reset the adsp service while dasp crashes
Reset the adsp service while dasp crashes
TC-RID: 1201-0101106
IssueID: TS-R-DF-118506
Change-Id: Ia3eb521f101ed40114ff09b8bfe02cfb93b0884c
(cherry picked from commit ffdcc33d17c331ff9b9ba236abf4911651c62992)

#27 Updated by IoT scm almost 2 years ago

Gerrit Merge Information
ID Project Branch Uploader
216258 general/vendor/thundercomm/products/sepolicy_vndr Pre_figure_turbox-c2130c-la1.1-vendor-release_sorp
BSP:System: Reset the adsp service while dasp crashes
Add permissions for slimbus_event
TC-RID: 1201-0101106
IssueID: TS-R-DF-118506
Change-Id: I51e9ca6905bf8e62ad4790891bf2bc2235a8854c
(cherry picked from commit 8acc9702ac125b2af6273eb63be378af80d71a74)

#28 Updated by IoT scm almost 2 years ago

Gerrit Merge Information
ID Project Branch Uploader
216257 general/vendor/thundercomm/products/figure Pre_figure_turbox-c2130c-la1.1-vendor-release_sorp
BSP:System: Reset the adsp service while dasp crashes
Reset the adsp service while dasp crashes
TC-RID: 1201-0101106
IssueID: TS-R-DF-118506
Change-Id: If2047665b061420fc5dc0b4d7e0649a80a8e65b6
(cherry picked from commit ee4d3a875f113df3ad5b0623de3d507755d0384c)

#29 Updated by CDTS_TEST 王成 almost 2 years ago

  • Status changed from RESOLVED to ASSIGNED
  • Assignee changed from CD BSP 叶鹏 to CD TPM-王祥林

需要跟进处理

Also available in: Atom PDF