Bug #118126

【IOVDEV-29542】【自动化测试】【稳定性测试】【VC1】【副屏】副屏重启

Added by CDTS_TEST 王成 about 2 years ago. Updated about 2 years ago.

Status:CLOSEDStart date:2023-05-23
Priority:UrgentDue date:2023-05-26
Assignee:CDTS_TEST 王成% Done:

100%

Category:重启
Target version:-
Need_Info:-- Found Version:01.00.0095.c103
Resolution:-- Degrated:--
Severity:Critical Verified Version:
Reproducibility:Occasionally Fixed Version:2023-05-26
Test Type:ST Root cause:

Description

【重启时间】：2023/5/19 18:00 ~ 5/20 09:00之间
【前提条件】：副屏上电
【操作步骤】

【前提条件】.副屏正常可以操作

1.执行monkey稳定性测试
【实际结果】
1.monkey停止运行，安卓系统重启
【期望结果】
1.副屏正常再跑monkey命令

【备注】：

adb shell

ps -ef | grep system_server

发现system_server PID 8000多，怀疑system_srever被kill掉，安卓重启

日志地址：https://ofs.human-horizons.com/#/download/index/Bc4triLf%2BV8%3D
【硬件版本号】：

【中控版本号】：03.02.0080.c101

【副屏版本号】：01.00.0095.c103

Subtasks

Bug #118129: 【IOVDEV-29629】【自动化测试】【稳定性测试】【VC1】【副屏】monkey稳定性测试，副屏重启（实验室...CLOSEDCD SYSTEM-夏旭

Bug #118196: 【IOVDEV-29629】【自动化测试】【稳定性测试】【VC1】【副屏】monkey稳定性测试，副屏重启（实验室...CLOSEDCDTS-TEST 周婷

Bug #118200: 请确认一下显示驱动 DRM模块下面的报错是否异常CLOSEDCDTS_TEST 王成

Bug #118264: 【monkey】4个预置应用+12V10A电源 monkey异常终止CLOSEDCD SYSTEM-夏旭

Related issues

Related to Figure - Bug #118416: 【AF】【EVT3】【Monkey】Monkey测试12小时，出现CRASH:com.android.docume... CLOSED 2023-06-01

History

#2 Updated by CD SYSTEM-夏旭 about 2 years ago

  • Status changed from New to ASSIGNED
  • % Done changed from 0 to 20

1.从log看system_server被杀。原因out_of_memory.
从system_server 这变化的PID看：

 E  : Out of memory: Kill process *1517* (system_server) score 0 or sacrifice child
     9697  9697 I system_server_start: [2,60308609,60308609]
     000041_persist_00005_700101_080011.log:152:01-01 08:00:11.218  1528  1528 I system_server_start: [1,8581,8581]（1次）
     000042_persist_00055_700101_080009.log:543:01-01 08:00:09.876  1520  1520 I system_server_start: [1,7026,7026]（2次）
     000043_persist_00127_700101_080009.log:616:01-01 08:00:10.310  1548  1548 I system_server_start: [1,7647,7647]（3次）
     000044_persist_00165_700101_080009.log:1051:01-01 08:00:10.069  1526  1526 I system_server_start: [1,7388,7388]（4次）
     000045_persist_00179_700101_080010.log:1706:01-01 08:00:09.801  1206  1206 I system_server_start: [1,6666,6666](5次)
     000045_persist_00022_230519_005507.log:1146:05-19 00:55:06.405  4449  4449 I system_server_start: [2,37099124,37099124](6次)
   
05-19 00:55:02.139  1206  1936 I ThunderPerfManager: PerfAcq error : state=true pkgName=com.android.music
   05-19 00:55:02.177  1226  1226 I perfetto: ory_arbiter_impl.cc:167 Recovered from stall after 12 iterations
   01-01 18:18:17.898     0     0 E         : Killed process 4186 (fsync) total-vm:64200kB, anon-rss:376kB, file-rss:1000kB, shmem-rss:112kB oom_score_adj=-900
   01-01 18:18:17.899     0     0 I oom_reaper: reaped process 4186 (fsync), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
   01-01 18:18:17.966     0     0 E         : Out of memory: Kill process 1206 (system_server) score 0 or sacrifice child
   01-01 18:18:17.968     0     0 E         : Killed process 1206 (system_server) total-vm:21132092kB, anon-rss:0kB, file-rss:201080kB, shmem-rss:9664kB oom_score_adj=-900
   zygote 重启：
   01-01 08:00:08.297   677   677 D AndroidRuntime: >>>>>> START com.android.internal.os.ZygoteInit uid 0 <<<<<<

#3 Updated by CD SYSTEM-夏旭 about 2 years ago

  • Assignee changed from CD SYSTEM-夏旭 to CD FW 曹覃刚

#4 Updated by CD FW 曹覃刚 about 2 years ago

  • Category changed from BSP to SYSTEM
  • Assignee changed from CD FW 曹覃刚 to CD SYSTEM-夏旭

Hi 宗惠
如昨晚线下沟通
1.FW检查下之前system_server_start的原因
000041_persist_00005_700101_080011.log:152:01-01 08:00:11.218 1528 1528 I system_server_start: [1,8581,8581]（1次）
000042_persist_00055_700101_080009.log:543:01-01 08:00:09.876 1520 1520 I system_server_start: [1,7026,7026]（2次）
000043_persist_00127_700101_080009.log:616:01-01 08:00:10.310 1548 1548 I system_server_start: [1,7647,7647]（3次）
000044_persist_00165_700101_080009.log:1051:01-01 08:00:10.069 1526 1526 I system_server_start: [1,7388,7388]（4次）
000045_persist_00179_700101_080010.log:1706:01-01 08:00:09.801 1206 1206 I system_server_start: [1,6666,6666](5次)
FSEM_000041_1155_0510180133/logcat/000041_persist_00005_700101_080011.log:152:01-01 08:00:11.218 1528 1528 I system_server_start: [1,8581,8581]
FSEM_000042_1158_0510190743/logcat/000042_persist_00055_700101_080009.log:543:01-01 08:00:09.876 1520 1520 I system_server_start: [1,7026,7026]
FSEM_000043_1163_0510211112/logcat/000043_persist_00127_700101_080009.log:616:01-01 08:00:10.310 1548 1548 I system_server_start: [1,7647,7647]
FSEM_000044_1164_0510213123/logcat/000044_persist_00165_700101_080009.log:1051:01-01 08:00:10.069 1526 1526 I system_server_start: [1,7388,7388]
FSEM_000045_1165_0510213417/logcat/000045_persist_00179_700101_080010.log:1706:01-01 08:00:09.801 1206 1206 I system_server_start: [1,6666,6666]
这部分日志是0510生成的，日志包中还有0406和0427的日志, 这与日志包中最新时间0518和0519时间差异较大
①假定日志打包工具时间是正常的，那么日期为0510的日志由于与问题发生时间差距较大，且前后有日志缺失，无法根据其分析
如果日志打包工具有异常，需要夏旭同学帮忙看下，日志工具的问题
②在日志中有看到下列shutdown的日志，可以判定之前有通过其他方式关机过，重启之后，system_server_start就是正常的
FSEM_000041_1155_0510180133/persist/000040_persist_00674_230411_061607.log.ing:110:04-11 06:16:12.215 1138 1213 D ShutdownThread: Notifying thread to start shutdown longPressBehavior=5
结合上述两点，之前system_server_start是有通过正常开关机重新启动的，是正常现象，另外由于与问题发生时间距离较久，没有太多参考意义

2.SYSTEM的同学分析下oom_reaper的机制, 这是系统内核中的机制
为什么在system_server内存占用正常以及优先级较高的情况下，会被kill掉，正常情况应该是杀其他不重要的进程
问题发生时system_server物理内存占了224M
logcat/000045_persist_00007_230519_005507.log:2277:05-19 00:54:31.428 1206 1560 I am_pss : [1206,1000,system,224908288,139141120,7460864,479518720,1,0,20]
我在我正常的跑的刚重启的设备中检查system_server的物理内存为205M
05-20 23:15:41.230 1025 1190 I am_pss : [1025,1000,system,208441344,112046080,0,521940992,0,0,29]

请帮忙分析下第2点，谢谢

#5 Updated by CD SYSTEM-夏旭 about 2 years ago

  • Assignee changed from CD SYSTEM-夏旭 to CD FW 曹覃刚

Hi 覃刚:

1. 从下面的在日志看，在第45次启动中就出现了两次system_server重启了，如果log无法分析，请测试复现，
000045_persist_00179_700101_080010.log:1706:01-01 08:00:09.801 1206 1206 I system_server_start: [1,6666,6666](5次)
000045_persist_00022_230519_005507.log:1146:05-19 00:55:06.405 4449 4449 I system_server_start: [2,37099124,37099124](6次)

2. oom_kill.c 中打印下面的信息，这个是内核默认打开的，
可查看 android/kernel/msm-4.19/mm/Makefile
android/kernel/msm-4.19/mm/oom_kill.c
obj-y := oom_kill.o
且打印此内容的函数说明，此函数只是回收地址空间，并不是由它杀死进程的

I oom_reaper: reaped process 4186 (fsync), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
   01-01 18:18:17.966     0     0 E         : Out of memory: Kill process 1206 (system_server) score 0 or sacrifice child

#6 Updated by 物联网项目组-RD3_CDTS 周飞 about 2 years ago

测试反馈未连接中控和amp
研发分析到是内存占用过高

#7 Updated by CDTS_TEST 王成 about 2 years ago

  • Fixed Version set to 2023-05-26

#8 Updated by CDTS_TEST 王成 about 2 years ago

  • Due date set to 2023-05-26

#9 Updated by CDTS-TEST 周婷 about 2 years ago

  • Category changed from SYSTEM to 重启

#10 Updated by CD FW 曹覃刚 about 2 years ago

  • Assignee changed from CD FW 曹覃刚 to CD SYSTEM-夏旭

Hi 夏旭
与 #118122 类似，如飞书语音沟通，请帮忙调查外部杀system_server的原因
感谢

#11 Updated by CD SYSTEM-夏旭 about 2 years ago

  • Status changed from ASSIGNED to RESOLVED

#12 Updated by CD SYSTEM-夏旭 about 2 years ago

  • Status changed from RESOLVED to ASSIGNED

#13 Updated by CD SYSTEM-夏旭 about 2 years ago

1. 打开内核kasan机制，查看是否有内存泄露

 
+CONFIG_KASAN=y
+CONFIG_KASAN_GENERIC=y
+CONFIG_KASAN_OUTLINE=y
+CONFIG_KCOV=y
+CONFIG_KCOV_ENABLE_COMPARISONS=y
+CONFIG_FRAME_WARN=0
+CONFIG_KASAN_STACK_ENABLE=y

2. 打开用记空间的malloc debug查看是否有内存泄露
3. 优化 android系统水线

#14 Updated by CD SYSTEM-夏旭 about 2 years ago

1.收集的信息确认，tlog内存泄露 USS空间占用的内存越来越大，

PID Vss Rss Pss Uss Swap PSwap USwap ZSwap cmdline
1289 13009224K 12452K 4619K 4284K 0K 0K 0K 0K /system/bin/tlog
1289 13009224K 12464K 4626K 4296K 0K 0K 0K 0K /system/bin/tlog
1289 13010772K 13120K 5305K 4976K 0K 0K 0K 0K /system/bin/tlog
1289 13010772K 13148K 5333K 5004K 0K 0K 0K 0K /system/bin/tlog
1289 13010772K 13012K 5376K 5076K 0K 0K 0K 0K /system/bin/tlog
1289 13010772K 12524K 4887K 4588K 568K 568K 568K 142K /system/bin/tlog
1289 13010772K 11240K 4823K 4656K 568K 568K 568K 148K /system/bin/tlog
1289 13010772K 11252K 4882K 4720K 568K 568K 568K 164K /system/bin/tlog
1289 13010772K 11336K 4984K 4824K 568K 568K 568K 162K /system/bin/tlog
1289 13010772K 11132K 5046K 4964K 568K 568K 568K 165K /system/bin/tlog
1289 13010772K 11220K 5205K 5124K 568K 568K 568K 167K /system/bin/tlog
1289 13013856K 12640K 6666K 6588K 568K 568K 568K 163K /system/bin/tlog
1289 13013856K 12736K 6818K 6740K 568K 568K 568K 168K /system/bin/tlog
1289 13013856K 12860K 6960K 6884K 568K 568K 568K 165K /system/bin/tlog
1289 13013856K 13020K 7132K 7056K 568K 568K 568K 172K /system/bin/tlog
1289 13013856K 13208K 7333K 7260K 568K 568K 568K 175K /system/bin/tlog
1289 13013856K 13468K 7593K 7520K 568K 568K 568K 173K /system/bin/tlog
1289 13013856K 13540K 7814K 7748K 568K 568K 568K 171K /system/bin/tlog
1289 13013856K 13716K 8035K 7972K 568K 568K 568K 174K /system/bin/tlog
1289 13016928K 16628K 10964K 10900K 568K 568K 568K 174K /system/bin/tlog
1289 13016928K 16728K 11099K 11036K 568K 568K 568K 176K /system/bin/tlog
1289 13016928K 17080K 11471K 11408K 568K 568K 568K 175K /system/bin/tlog
1289 13016928K 17280K 11671K 11608K 568K 568K 568K 187K /system/bin/tlog
1289 13016928K 17504K 11895K 11832K 568K 568K 568K 174K /system/bin/tlog
1289 13016928K 17820K 12211K 12148K 568K 568K 568K 173K /system/bin/tlog
1289 13016928K 18204K 12594K 12532K 568K 568K 568K 175K /system/bin/tlog
1289 13016928K 18488K 12878K 12816K 568K 568K 568K 171K /system/bin/tlog
1289 13016928K 18756K 13146K 13084K 568K 568K 568K 173K /system/bin/tlog
1289 13016928K 18936K 13326K 13264K 568K 568K 568K 176K /system/bin/tlog
1289 13016928K 19088K 13478K 13416K 568K 568K 568K 175K /system/bin/tlog
1289 13023072K 24748K 19142K 19080K 568K 568K 568K 173K /system/bin/tlog
1289 13023072K 25000K 19402K 19340K 568K 568K 568K 175K /system/bin/tlog
1289 13023072K 25432K 19838K 19776K 568K 568K 568K 173K /system/bin/tlog
1289 13023072K 25740K 20146K 20084K 568K 568K 568K 177K /system/bin/tlog
1289 13023072K 26160K 20570K 20508K 568K 568K 568K 176K /system/bin/tlog
1289 13023072K 26564K 20974K 20912K 568K 568K 568K 181K /system/bin/tlog
1289 13023072K 27100K 21760K 21704K 568K 568K 568K 170K /system/bin/tlog
1289 13023072K 27936K 22600K 22544K 568K 568K 568K 175K /system/bin/tlog
1289 13023072K 28692K 23356K 23300K 568K 568K 568K 170K /system/bin/tlog
1289 13023072K 29652K 24316K 24260K 568K 568K 568K 170K /system/bin/tlog
1289 13023072K 30468K 25133K 25076K 568K 568K 568K 170K /system/bin/tlog
1289 13035360K 41932K 36596K 36540K 568K 568K 568K 172K /system/bin/tlog
1289 13035360K 42908K 37572K 37516K 568K 568K 568K 171K /system/bin/tlog
1289 13035360K 43720K 38384K 38328K 568K 568K 568K 169K /system/bin/tlog
1289 13035360K 44556K 39220K 39164K 568K 568K 568K 166K /system/bin/tlog
1289 13035360K 45416K 40080K 40024K 568K 568K 568K 167K /system/bin/tlog
1289 13035360K 46080K 40744K 40688K 568K 568K 568K 169K /system/bin/tlog
1289 13035360K 47096K 41760K 41704K 568K 568K 568K 167K /system/bin/tlog
1289 13035360K 47908K 42572K 42516K 568K 568K 568K 167K /system/bin/tlog
1289 13035360K 48616K 43280K 43224K 568K 568K 568K 180K /system/bin/tlog
1289 13035360K 49160K 43824K 43768K 568K 568K 568K 134K /system/bin/tlog
1289 13035360K 49892K 44557K 44500K 568K 568K 568K 137K /system/bin/tlog
1289 13035360K 50388K 45053K 44996K 568K 568K 568K 139K /system/bin/tlog
1289 13035360K 51084K 45748K 45692K 568K 568K 568K 137K /system/bin/tlog
1289 13035360K 51504K 46169K 46112K 568K 568K 568K 140K /system/bin/tlog
1289 13035360K 51544K 46209K 46152K 568K 568K 568K 134K /system/bin/tlog
1289 13035360K 51616K 46280K 46224K 568K 568K 568K 134K /system/bin/tlog
1289 13035360K 51748K 46413K 46356K 568K 568K 568K 142K /system/bin/tlog
1289 13035360K 51864K 46529K 46472K 568K 568K 568K 141K /system/bin/tlog
1289 13035360K 52008K 46673K 46616K 568K 568K 568K 134K /system/bin/tlog
1289 13035360K 52132K 46798K 46740K 568K 568K 568K 138K /system/bin/tlog
1289 13035360K 52332K 46997K 46940K 568K 568K 568K 139K /system/bin/tlog
1289 13035360K 52448K 47112K 47056K 568K 568K 568K 140K /system/bin/tlog
1289 13035360K 52732K 47397K 47340K 568K 568K 568K 135K /system/bin/tlog
1289 13035360K 53040K 47704K 47648K 568K 568K 568K 134K /system/bin/tlog
1289 13035360K 53360K 48025K 47968K 568K 568K 568K 141K /system/bin/tlog
1289 13059936K 74900K 69564K 69508K 568K 568K 568K 142K /system/bin/tlog
1289 13059936K 75288K 69956K 69900K 568K 568K 568K 141K /system/bin/tlog
1289 13059936K 75804K 70472K 70416K 568K 568K 568K 145K /system/bin/tlog
1289 13059936K 76396K 71064K 71008K 568K 568K 568K 148K /system/bin/tlog
1289 13059936K 76864K 71532K 71476K 568K 568K 568K 150K /system/bin/tlog
1289 13059936K 77512K 72180K 72124K 568K 568K 568K 154K /system/bin/tlog
1289 13059936K 78012K 72680K 72624K 568K 568K 568K 155K /system/bin/tlog
1289 13059936K 78500K 73168K 73112K 568K 568K 568K 158K /system/bin/tlog
1289 13059936K 79220K 73888K 73832K 568K 568K 568K 157K /system/bin/tlog
1289 13059936K 79652K 74320K 74264K 568K 568K 568K 160K /system/bin/tlog
1289 13059936K 80360K 75028K 74972K 568K 568K 568K 159K /system/bin/tlog
1289 13059936K 81344K 76012K 75956K 568K 568K 568K 166K /system/bin/tlog
1289 13059936K 81960K 76633K 76576K 568K 568K 568K 152K /system/bin/tlog
1289 13059936K 82556K 77229K 77172K 568K 568K 568K 152K /system/bin/tlog
1289 13059936K 83396K 78069K 78012K 568K 568K 568K 138K /system/bin/tlog
1289 13059936K 84312K 78984K 78928K 568K 568K 568K 144K /system/bin/tlog
1289 13059936K 85612K 80284K 80228K 568K 568K 568K 142K /system/bin/tlog
1289 13059936K 86760K 81432K 81376K 568K 568K 568K 144K /system/bin/tlog
1289 13059936K 88060K 82732K 82676K 568K 568K 568K 142K /system/bin/tlog
1289 13059936K 89252K 83924K 83868K 568K 568K 568K 161K /system/bin/tlog
1289 13059936K 90080K 84755K 84700K 568K 568K 568K 161K /system/bin/tlog
1289 13059936K 91396K 86071K 86016K 568K 568K 568K 145K /system/bin/tlog
1289 13059936K 92688K 87364K 87308K 568K 568K 568K 145K /system/bin/tlog
1289 13059936K 94020K 88696K 88640K 568K 568K 568K 152K /system/bin/tlog
1289 13059936K 95204K 89880K 89824K 568K 568K 568K 148K /system/bin/tlog
1289 13059936K 96224K 90900K 90844K 568K 568K 568K 153K /system/bin/tlog
1289 13059936K 97200K 91876K 91820K 568K 568K 568K 147K /system/bin/tlog
1289 13059936K 97928K 92604K 92548K 568K 568K 568K 148K /system/bin/tlog
1289 13059936K 98936K 93612K 93556K 568K 568K 568K 155K /system/bin/tlog
1289 13109088K 142856K 137531K 137476K 568K 568K 568K 146K /system/bin/tlog
1289 13109088K 144344K 139019K 138964K 568K 568K 568K 147K /system/bin/tlog
1289 13109088K 145284K 139959K 139904K 568K 568K 568K 152K /system/bin/tlog
1289 13109088K 146536K 141211K 141156K 568K 568K 568K 144K /system/bin/tlog
1289 13109088K 148036K 142711K 142656K 568K 568K 568K 153K /system/bin/tlog
1289 13109088K 149804K 144479K 144424K 568K 568K 568K 153K /system/bin/tlog
1289 13109088K 151696K 146371K 146316K 568K 568K 568K 153K /system/bin/tlog
1289 13109088K 153884K 148559K 148504K 568K 568K 568K 152K /system/bin/tlog
1289 13109088K 155124K 149799K 149744K 568K 568K 568K 159K /system/bin/tlog
1289 13109088K 156216K 150891K 150836K 568K 568K 568K 149K /system/bin/tlog
1289 13109088K 158440K 153115K 153060K 568K 568K 568K 151K /system/bin/tlog
1289 13109088K 160184K 154859K 154804K 568K 568K 568K 153K /system/bin/tlog
1289 13109088K 160892K 155567K 155512K 568K 568K 568K 162K /system/bin/tlog
1289 13109088K 160908K 155583K 155528K 568K 568K 568K 151K /system/bin/tlog
1289 13109088K 160940K 155615K 155560K 568K 568K 568K 153K /system/bin/tlog
1289 13109088K 161008K 155683K 155628K 568K 568K 568K 153K /system/bin/tlog
1289 13109088K 161116K 155791K 155736K 568K 568K 568K 152K /system/bin/tlog
1289 13109088K 161188K 155863K 155808K 568K 568K 568K 154K /system/bin/tlog
1289 13109088K 161368K 156043K 155988K 568K 568K 568K 156K /system/bin/tlog
1289 13109088K 161564K 156239K 156184K 568K 568K 568K 158K /system/bin/tlog
1289 13109088K 161796K 156471K 156416K 568K 568K 568K 157K /system/bin/tlog
1289 13109088K 161988K 156663K 156608K 568K 568K 568K 158K /system/bin/tlog
1289 13109088K 162200K 156875K 156820K 568K 568K 568K 157K /system/bin/tlog
1289 13109088K 162660K 157335K 157280K 568K 568K 568K 154K /system/bin/tlog
1289 13109088K 163144K 157819K 157764K 568K 568K 568K 158K /system/bin/tlog
1289 13109088K 163596K 158271K 158216K 568K 568K 568K 154K /system/bin/tlog
1289 13109088K 164240K 158916K 158860K 568K 568K 568K 158K /system/bin/tlog
1289 13109088K 164572K 159247K 159192K 568K 568K 568K 163K /system/bin/tlog
1289 13109088K 164980K 159655K 159600K 568K 568K 568K 170K /system/bin/tlog
1289 13109088K 165700K 160376K 160320K 568K 568K 568K 170K /system/bin/tlog
1289 13109088K 166088K 160929K 160876K 568K 568K 568K 171K /system/bin/tlog
1289 13109088K 166740K 161581K 161528K 568K 568K 568K 174K /system/bin/tlog
1289 13109088K 167504K 162346K 162292K 568K 568K 568K 189K /system/bin/tlog
1289 13109088K 168124K 162966K 162912K 568K 568K 568K 192K /system/bin/tlog
1289 13109088K 168928K 163770K 163716K 568K 568K 568K 192K /system/bin/tlog
1289 13109088K 169648K 164490K 164436K 568K 568K 568K 195K /system/bin/tlog
1289 13109088K 170400K 165242K 165188K 568K 568K 568K 194K /system/bin/tlog
1289 13109088K 171496K 166338K 166284K 568K 568K 568K 193K /system/bin/tlog
1289 13109088K 172320K 167162K 167108K 568K 568K 568K 194K /system/bin/tlog
1289 13109088K 173368K 168210K 168156K 568K 568K 568K 198K /system/bin/tlog
1289 13109088K 174260K 169101K 169048K 568K 568K 568K 194K /system/bin/tlog
1289 13109088K 175180K 170022K 169968K 568K 568K 568K 201K /system/bin/tlog
1289 13109088K 176136K 170978K 170924K 568K 568K 568K 193K /system/bin/tlog
1289 13109088K 177284K 172126K 172072K 568K 568K 568K 192K /system/bin/tlog
1289 13109088K 178304K 173146K 173092K 568K 568K 568K 193K /system/bin/tlog
1289 13109088K 179176K 174018K 173964K 568K 568K 568K 191K /system/bin/tlog
1289 13109088K 180432K 175274K 175220K 568K 568K 568K 188K /system/bin/tlog
1289 13109088K 181536K 176382K 176328K 568K 568K 568K 190K /system/bin/tlog
1289 13109088K 183180K 178037K 177984K 568K 568K 568K 189K /system/bin/tlog
1289 13109088K 184164K 179021K 178968K 568K 568K 568K 190K /system/bin/tlog
1289 13109088K 185988K 180845K 180792K 568K 568K 568K 191K /system/bin/tlog
1289 13109088K 168648K 163505K 163452K 19304K 19304K 19304K 6433K /system/bin/tlog
1289 13109088K 124860K 119717K 119664K 64456K 64456K 64456K 20914K /system/bin/tlog
1289 13109088K 126120K 120978K 120924K 64588K 60740K 56892K 19623K /system/bin/tlog
1289 13207392K 275708K 270724K 270672K 568K 568K 568K 189K /system/bin/tlog
1289 13207392K 277288K 272340K 272288K 568K 568K 568K 187K /system/bin/tlog
1289 13207392K 278532K 273600K 273548K 568K 568K 568K 186K /system/bin/tlog
1289 13207392K 280376K 275448K 275396K 568K 568K 568K 185K /system/bin/tlog
1289 13207392K 282024K 277095K 277044K 568K 568K 568K 185K /system/bin/tlog
1289 13207392K 283800K 278872K 278820K 568K 568K 568K 184K /system/bin/tlog
1289 13207392K 285172K 280244K 280192K 568K 568K 568K 188K /system/bin/tlog
1289 13207392K 286928K 282000K 281948K 568K 568K 568K 189K /system/bin/tlog
1289 13207392K 288720K 283792K 283740K 568K 568K 568K 187K /system/bin/tlog
1289 13207392K 290180K 285252K 285200K 568K 568K 568K 187K /system/bin/tlog
1289 13207392K 290196K 285268K 285216K 568K 568K 568K 186K /system/bin/tlog
1289 13207392K 290236K 285308K 285256K 568K 568K 568K 186K /system/bin/tlog
1289 13207392K 290332K 285404K 285352K 568K 568K 568K 187K /system/bin/tlog
1289 13207392K 290464K 285536K 285484K 568K 568K 568K 183K /system/bin/tlog
1289 13207392K 290628K 285701K 285648K 568K 568K 568K 185K /system/bin/tlog
1289 13207392K 290852K 285924K 285872K 568K 568K 568K 186K /system/bin/tlog
1289 13207392K 291016K 286089K 286036K 568K 568K 568K 186K /system/bin/tlog
1289 13207392K 291188K 286261K 286208K 568K 568K 568K 185K /system/bin/tlog
1289 13207392K 291404K 286477K 286424K 568K 568K 568K 186K /system/bin/tlog
1289 13207392K 291544K 286617K 286564K 568K 568K 568K 185K /system/bin/tlog
1289 13207392K 291848K 286920K 286868K 568K 568K 568K 185K /system/bin/tlog
1289 13207392K 292160K 287232K 287180K 568K 568K 568K 183K /system/bin/tlog
1289 13207392K 292628K 287700K 287648K 568K 568K 568K 183K /system/bin/tlog
1289 13207392K 293100K 288172K 288120K 568K 568K 568K 184K /system/bin/tlog
1289 13207392K 293612K 288684K 288632K 568K 568K 568K 185K /system/bin/tlog
1289 13207392K 294248K 289320K 289268K 568K 568K 568K 183K /system/bin/tlog
1289 13207392K 294848K 289920K 289868K 568K 568K 568K 184K /system/bin/tlog
1289 13207392K 295240K 290312K 290260K 568K 568K 568K 183K /system/bin/tlog
1289 13207392K 295136K 290208K 290156K 1208K 1208K 1208K 387K /system/bin/tlog
1289 13207392K 295404K 290477K 290424K 1372K 1372K 1372K 437K /system/bin/tlog
1289 13207392K 295984K 291057K 291004K 1236K 1236K 1236K 409K /system/bin/tlog
1289 13207392K 296700K 291773K 291720K 1052K 1052K 1052K 342K /system/bin/tlog
1289 13207392K 297356K 292429K 292376K 764K 764K 764K 238K /system/bin/tlog
1289 13207392K 297640K 292713K 292660K 668K 668K 668K 209K /system/bin/tlog
1289 13207392K 297908K 292981K 292928K 664K 664K 664K 207K /system/bin/tlog
1289 13207392K 298028K 293100K 293048K 660K 660K 660K 206K /system/bin/tlog
1289 13207392K 298188K 293260K 293208K 656K 656K 656K 204K /system/bin/tlog
1289 13207392K 298384K 293456K 293404K 656K 656K 656K 204K /system/bin/tlog
1289 13207392K 298500K 293572K 293520K 656K 656K 656K 204K /system/bin/tlog
1289 13207392K 298660K 293733K 293680K 656K 656K 656K 204K /system/bin/tlog
1289 13207392K 298820K 293893K 293840K 656K 656K 656K 205K /system/bin/tlog
1289 13207392K 298944K 294017K 293964K 656K 656K 656K 206K /system/bin/tlog
1289 13207392K 299152K 294225K 294172K 652K 652K 652K 204K /system/bin/tlog
1289 13207392K 299316K 294389K 294336K 652K 652K 652K 204K /system/bin/tlog
1289 13207392K 299532K 294605K 294552K 648K 648K 648K 202K /system/bin/tlog
1289 13207392K 299700K 294773K 294720K 648K 648K 648K 205K /system/bin/tlog
1289 13207392K 299872K 294945K 294892K 648K 648K 648K 201K /system/bin/tlog
1289 13207392K 300048K 295121K 295068K 644K 644K 644K 185K /system/bin/tlog
1289 13207392K 300264K 295338K 295284K 644K 644K 644K 194K /system/bin/tlog
1289 13207392K 300448K 295522K 295468K 636K 636K 636K 166K /system/bin/tlog
1289 13207392K 300624K 295699K 295644K 636K 636K 636K 164K /system/bin/tlog
1289 13207392K 300804K 295879K 295824K 632K 632K 632K 166K /system/bin/tlog
1289 13207392K 300984K 296059K 296004K 632K 632K 632K 166K /system/bin/tlog
1289 13207392K 301120K 296195K 296140K 632K 632K 632K 162K /system/bin/tlog
1289 13207392K 301304K 296379K 296324K 632K 632K 632K 163K /system/bin/tlog
1289 13207392K 301440K 296515K 296460K 632K 632K 632K 163K /system/bin/tlog
1289 13207392K 301580K 296655K 296600K 632K 632K 632K 163K /system/bin/tlog
1289 13207392K 301720K 296795K 296740K 632K 632K 632K 163K /system/bin/tlog
1289 13207392K 301912K 296987K 296932K 628K 628K 628K 162K /system/bin/tlog
1289 13207392K 302064K 297139K 297084K 616K 616K 616K 159K /system/bin/tlog
1289 13207392K 302304K 297379K 297324K 616K 616K 616K 160K /system/bin/tlog
1289 13207392K 302444K 297519K 297464K 616K 616K 616K 160K /system/bin/tlog

2. 下一步，确认内存泄露点

#15 Updated by CD SYSTEM-夏旭 about 2 years ago

1.已经能tlog重构，现在能抓取到tlog程序运行时，实时内存分配的情况


        4096 bytes in 1 allocations from stack
                __smakebuf+0xa4 [libc.so]
                __swsetup+0xb0 [libc.so]
                __sfvwrite+0x44 [libc.so]
                fwrite+0xc4 [libc.so]
                tlog::ZipEncryptWriter::encryptWrite(tlog::LogFileEntry*, unsigned char*, unsigned int)+0x224 [tlog]
                tlog::ZipEncryptWriter::deflatEncryptWrite(tlog::LogFileEntry*, char*, unsigned long)+0x118 [tlog]
                tlog::TlogWriter::writeToFile(LogData*)+0x13c0 [tlog]
                tlog::PersistLogger::logCollectBl()+0x26c [tlog]
                tlog::_handlerThread(void*) (.cfi)+0x4c [tlog]
                __pthread_start(void*)+0x10c [libc.so]
                __start_thread+0x48 [libc.so]
        15872 bytes in 248 allocations from stack
                operator new(unsigned long)+0x20 [libc++.so]
                std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::basic_string(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)+0x74 [libc++.so]
                tlog::TlogWriter::writeToFile(LogData*)+0xa14 [tlog]
                tlog::PersistLogger::logCollectBl()+0x26c [tlog]
                tlog::_handlerThread(void*) (.cfi)+0x4c [tlog]
                __pthread_start(void*)+0x10c [libc.so]
                __start_thread+0x48 [libc.so]
        262144 bytes in 4096 allocations from stack
                operator new(unsigned long)+0x20 [libc++.so]
                std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::basic_string(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&)+0x74 [libc++.so]
                tlog::TlogWriter::writeToFile(LogData*)+0xcc0 [tlog]
                tlog::PersistLogger::logCollectBl()+0x26c [tlog]
                tlog::_handlerThread(void*) (.cfi)+0x4c [tlog]
                __pthread_start(void*)+0x10c [libc.so]
                __start_thread+0x48 [libc.so]
        786432 bytes in 1 allocations from stack
                operator new(unsigned long)+0x20 [libc++.so]
                tlog::TlogWriter::writeToFile(LogData*)+0xc2c [tlog]
                tlog::PersistLogger::logCollectBl()+0x26c [tlog]
                tlog::_handlerThread(void*) (.cfi)+0x4c [tlog]
                __pthread_start(void*)+0x10c [libc.so]
                __start_thread+0x48 [libc.so]

2. 下一步，检查在内存分配的路径上可能存在的内存泄露问题

#16 Updated by IoT scm about 2 years ago

Gerrit Merge Information：
ID Project Branch Uploader
203005 general/kernel/msm-4.19 Pre_figure_turbox-c2130c-la1.1-vendor-dev
SYSTEM :Tlog: Reset after detecting Tlog memory anomaly
When use __alloc_pages_slowpath path, detect tlog memory usage.
If RSS Size exceeds 576MB, kill tlog frees memory
Tlog automatically restarts to continue capturing logs
TC-RID: 1200-0800102
IssueID: TS-R-DF-118126
Change-Id: Ie801e942bf364f378b2ef05b804241d41a526f26

#17 Updated by IoT scm about 2 years ago

Gerrit Merge Information：
ID Project Branch Uploader
203013 general/vendor/thundercomm/tsnv Pre_figure_turbox-c2130c-la1.1-vendor-dev
BSP:TSNV: update TSNV for Tlog workaround
update TSNV for Tlog workaround
TC-RID: 1200-0102003
IssueID: TS-R-DF-118126
Change-Id: Ifd64736de6585e2766a33d9a6936cc7779f676e8

#18 Updated by CD SYSTEM-夏旭 about 2 years ago

1. 现在初步定位到代码中的模块
是因为在写log文件中，当写完一个文件后，再去新建一个log文件的，在这个过程中会造成内存泄露
2. 我们做过验证，只写一人文件，去查看相关的内存参数，并没有出现内存泄露的问题

#19 Updated by CD SYSTEM-胡兵 about 2 years ago

当前状态：
已定位到具体代码段问题，容器存储对象未释放导致，修改代码逻辑后，本地压力测试24小时(3s生成一个256k日志文件)，Tlog内存几乎没有变化，同样压测手法，修改前内存有明显增长。

下一步计划：
优化文件写入逻辑，本周合入优化方案

#20 Updated by IoT scm about 2 years ago

Gerrit Merge Information：
ID Project Branch Uploader
203640 general/platform/vendor/thundercomm/apps/LogSystem Pre_figure_turbox-c2130c-la1.1-qssi12-dev
SYSTEM :Tlog: tlog memory leak
Continuously adding file names to the rotatedLogFileList without releasing them, resulting in memory leakage and no actual calls to the rotatedLogFileList. Therefore, the rotatedLogFileList was removed
TC-RID: 1200-0800102
IssueID: TS-R-BUG-118126
Change-Id: Id6f8584c49e4e6ab0872a61a514d2e1770fb50b7

#21 Updated by IoT scm about 2 years ago

Gerrit Merge Information：
ID Project Branch Uploader
204151 general/kernel/msm-4.19 Pre_figure_turbox-c2130c-la1.1-vendor-dev
Revert SYSTEM :Tlog: Reset after detecting Tlog memory anomaly
This reverts commit 692bbc7cc36ac7bdc01b1d26cef6eb6e25783769.
TC-RID: 1200-0102003
IssueID: TS-R-DF-118126
Change-Id: I286fa228b159b252a5cd7ff35291a184a518be8e

#22 Updated by IoT scm about 2 years ago

Gerrit Merge Information：
ID Project Branch Uploader
204150 general/vendor/thundercomm/tsnv Pre_figure_turbox-c2130c-la1.1-vendor-dev
Revert BSP:TSNV: update TSNV for Tlog workaround
This reverts commit c106a3e1109eb6ca5e418aab26a370d180cef6fa.
TC-RID: 1200-0102003
IssueID: TS-R-DF-118126
Change-Id: I01f9eb7dbcd623ebacea13b17661d27746f6eee5

#23 Updated by CD SYSTEM-胡兵 about 2 years ago

  • Status changed from ASSIGNED to RESOLVED
  • Assignee changed from CD SYSTEM-夏旭 to CDTS_TEST 王成

正式方案已合入，请测试同学使用DB跑monkey验证

bug：tlog内存泄露导致副屏重启
提交：https://dev.thundercomm.com/gerrit/c/general/platform/vendor/thundercomm/apps/LogSystem/+/203640
验证：其他的功能正常
user版本的VB：安装apk，monkey跑了5个小时，tlog的RSS内存一直是10M以内，tlog进程号没有改变也没有被杀掉
userdebug版本的VB：修改文件大小为10K，tlog的RSS内存一直是10M以内，tlog进程号没有改变也没有被杀掉

#24 Updated by CDTS_TEST 王成 about 2 years ago

客户反馈此BUG对应的测试环境没有连接AMP和中控，从日志分析结果看也没禁用音频，已关闭此BUG

#25 Updated by CDTS_TEST 王成 about 2 years ago

  • Status changed from RESOLVED to VERIFIED

#26 Updated by CDTS_TEST 王成 about 2 years ago

  • Status changed from VERIFIED to CLOSED

Also available in: Atom PDF