Kernel cpu leak
2007-12-24 18:54:00
let me add more info on that..
Hello Lorne,
I still cannot fix the problem.
You can try to call sun to help us out.
send them the info below:
-------------------------------------------------------------------------------------------
(610)root at machine@/etc>ps -elo pcpu,user,args|sort -n -r
1.6 root fsflush
0.2 root sshd -i -V SSH-1.5-1.2.30
0.1 root sshd -i -V SSH-1.5-1.2.30
0.1 root ps -elo pcpu,user,args
0.1 root bash
0.0 gaziz bash
0.0 brother /usr/local/apache.brother/bin/httpd
0.0 brother /usr/local/apache.brother/bin/httpd
0.0 root sched
0.0 root pageout
0.0 root /usr/sbin/vold
0.0 root /usr/sbin/syslogd
0.0 root /usr/sbin/nscd
0.0 root /usr/sbin/inetd -t -s
0.0 root /usr/sbin/cron
0.0 root /usr/local/sbin/sshd
0.0 root /usr/local/apache.brother/bin/httpd
0.0 root /usr/lib/utmpd
0.0 root /usr/lib/saf/ttymon
0.0 root /usr/lib/saf/sac -t 300
0.0 root /usr/lib/devfsadm/devfseventd
0.0 root /usr/lib/devfsadm/devfsadmd
0.0 root /etc/init -
0.0 root -sh
(502)gaziz at machine@/export/home/gaziz>vmstat 1 33
procs memory page disk faults cpu
r b w swap free re mf pi po fr de sr s0 s1 s6 -- in sy cs us sy id
2 0 0 583672 334928 0 364 66 1 1 0 0 2 1 0 0 191 574 114 8 69 23
2 0 0 583816 318192 0 5 0 0 0 0 0 0 0 0 0 147 192 112 2 66 32
1 0 0 583816 318072 0 0 0 0 0 0 0 0 0 0 0 166 188 112 1 68 31
1 0 0 583792 317944 0 0 0 0 0 0 0 0 1 0 0 186 166 111 2 66 32
1 0 0 583792 317840 0 0 0 0 0 0 0 2 1 0 0 232 184 118 4 66 30
1 0 0 583792 317728 0 0 0 0 0 0 0 0 0 0 0 168 185 117 4 69 27
1 0 0 583792 317632 0 0 0 0 0 0 0 0 0 0 0 138 178 107 3 67 30
2 0 0 583792 317536 0 0 0 0 0 0 0 0 0 0 0 144 166 94 4 67 29
2 0 0 583792 317424 0 0 0 0 0 0 0 0 1 0 0 192 161 112 3 70 27
2 0 0 583368 317144 0 284 0 32 32 0 0 0 2 0 0 202 409 128 7 70 23
1 0 0 583792 317272 0 0 0 0 0 0 0 0 0 0 0 159 161 92 2 67 31
1 0 0 583792 317184 0 0 0 0 0 0 0 0 0 0 0 163 142 94 2 68 30
1 0 0 583792 317080 0 0 0 0 0 0 0 1 2 0 0 229 170 103 4 69 27
1 0 0 583792 316976 0 0 0 0 0 0 0 0 0 0 0 148 165 98 1 66 33
2 0 0 583544 316792 0 284 0 32 32 0 0 0 2 0 0 220 421 137 7 71 23
1 0 0 583792 316800 0 0 0 0 0 0 0 0 0 0 0 140 174 105 1 66 33
1 0 0 583792 316704 0 0 0 0 0 0 0 1 1 0 0 211 167 112 5 67 28
2 0 0 583792 316600 0 0 0 0 0 0 0 0 1 0 0 181 186 109 3 68 29
1 0 0 583792 316488 0 0 0 0 0 0 0 0 0 0 0 152 163 95 3 66 31
procs memory page disk faults cpu
r b w swap free re mf pi po fr de sr s0 s1 s6 -- in sy cs us sy id
2 0 0 583792 316392 0 0 0 0 0 0 0 0 0 0 0 148 172 104 2 66 32
1 0 0 583792 316288 0 0 0 0 0 0 0 0 0 0 0 171 181 115 0 70 30
2 0 0 583664 316104 0 548 0 0 0 0 0 0 0 0 0 151 962 136 5 74 21
1 0 0 583528 315880 0 509 0 0 0 0 0 0 1 0 0 158 784 135 7 71 22
3 0 0 583432 315688 22 1605 0 0 0 0 0 1 1 0 0 198 2209 189 8 90 2
3 0 0 583360 315600 0 1669 0 0 0 0 0 1 1 0 0 225 1977 212 10 90 0
^C
(503)gaziz at machine@/export/home/gaziz>vmstat -S 1 33
procs memory page disk faults cpu
r b w swap free si so pi po fr de sr s0 s1 s6 -- in sy cs us sy id
2 0 0 583680 333648 0 0 61 1 1 0 0 2 1 0 0 190 570 115 8 70 23
3 0 0 583400 315416 0 0 0 8 8 0 0 0 0 0 0 145 2275 197 12 88 0
3 0 0 583320 315288 0 0 0 0 0 0 0 0 0 0 0 158 2163 199 12 88 0
2 0 0 583480 315248 0 0 0 8 8 0 0 3 2 0 0 308 1918 192 11 85 4
3 0 0 583432 315104 0 0 0 0 0 0 0 0 1 0 0 181 2136 186 17 77 6
4 0 0 583392 314992 0 0 0 16 16 0 0 0 1 0 0 181 2260 156 17 81 2
3 0 0 583336 314808 0 0 0 0 0 0 0 0 0 0 0 161 2076 162 19 80 1
3 0 0 583216 314616 0 0 0 8 8 0 0 1 1 0 0 230 2207 151 21 78 1
3 0 0 583264 314544 0 0 0 0 0 0 0 1 2 0 0 269 2299 153 10 90 0
3 0 0 583720 314792 0 0 0 8 8 0 0 0 1 0 0 171 820 141 13 70 17
1 0 0 583792 314736 0 0 0 0 0 0 0 0 0 0 0 148 163 105 2 66 32
2 0 0 583792 314640 0 0 0 0 0 0 0 0 0 0 0 153 151 95 0 68 32
^C
(504)gaziz at machine@/export/home/gaziz>mpstat
CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
0 367 5 0 289 88 115 15 0 0 0 585 8 70 4 19
(505)gaziz at machine@/export/home/gaziz>mpstat 1 33
CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl
0 366 5 0 288 88 115 15 0 0 0 584 8 70 4 19
0 5 0 0 314 114 110 11 0 2 0 194 5 69 1 25
0 0 0 0 282 82 102 3 0 0 0 169 2 68 0 30
0 284 0 0 298 98 121 13 0 0 0 372 6 70 2 22
0 0 0 0 262 62 103 3 0 0 0 179 3 66 0 31
0 0 0 0 246 46 100 13 0 0 0 162 4 69 0 27
0 0 0 0 333 133 116 13 0 0 0 173 6 67 2 25
0 0 0 0 246 46 109 7 0 0 0 182 2 67 0 31
0 0 0 0 283 83 96 6 0 0 0 148 4 68 0 28
0 0 0 0 244 44 110 16 0 0 0 172 4 67 0 29
(539)root at machine@/tmp>sar -c 1 33
SunOS machine 5.7 Generic_106541-11 sun4u 12/13/00
20:50:34 scall/s sread/s swrit/s fork/s exec/s rchar/s wchar/s
20:50:42 225 17 18 0.00 0.00 4574 3624
20:50:49 189 14 15 0.00 0.00 3450 3533
20:50:56 189 14 15 0.00 0.00 3450 3533
20:51:03 193 14 16 0.00 0.00 3330 3502
20:51:10 189 14 15 0.00 0.00 3450 3533
20:51:17 341 24 19 0.00 0.00 7916 4491
20:51:25 832 75 75 1.89 2.83 8647 5985
20:51:32 408 82 74 0.00 0.00 626089 547872
20:51:39 349 57 49 0.00 0.00 426362 361325
20:51:46 334 58 55 0.00 0.00 439160 441481
20:51:53 332 59 54 0.00 0.00 445818 447709
20:52:00 330 59 53 0.00 0.00 445818 439021
20:52:08 252 31 32 0.00 0.00 170455 178672
(553)root at machine@/etc>more /etc/vfstab
#device device mount FS fsck mount mount
#to mount to fsck point type pass at boot options
#
#/dev/dsk/c1d0s2 /dev/rdsk/c1d0s2 /usr ufs 1 yes -
fd - /dev/fd fd - no -
/proc - /proc proc - no -
/dev/dsk/c0t0d0s3 - - swap - no -
/dev/dsk/c0t0d0s0 /dev/rdsk/c0t0d0s0 / ufs 1 no -
/dev/dsk/c0t0d0s6 /dev/rdsk/c0t0d0s6 /usr ufs 1 no -
/dev/dsk/c0t0d0s1 /dev/rdsk/c0t0d0s1 /var ufs 1 no -
/dev/dsk/c0t1d0s7 /dev/rdsk/c0t1d0s7 /export ufs 2 yes -
/dev/dsk/c0t0d0s7 /dev/rdsk/c0t0d0s7 /usr/local ufs 2 yes -
swap - /tmp tmpfs - yes -
(554)root at machine@/etc>swap -d /dev/dsk/c0t0d0s3
/dev/dsk/c0t0d0s3 was dump device --
invoking dumpadm(1M) -d swap to select new dump device
dumpadm: no swap devices are available
(555)root at machine@/etc>w
9:05pm up 11 day(s), 22:32, 2 users, load average: 3.48, 3.84, 3.88
User tty login@ idle JCPU PCPU what
gaziz pts/0 1Dec00 w
gaziz pts/1 8:21pm 2 bash
(556)root at machine@/etc>w
9:05pm up 11 day(s), 22:32, 2 users, load average: 3.48, 3.84, 3.88
User tty login@ idle JCPU PCPU what
gaziz pts/0 1Dec00 w
gaziz pts/1 8:21pm 2 bash
(557)root at machine@/etc>swap -l
No swap devices configured
(558)root at machine@/etc>sar -p 1 33
SunOS machine 5.7 Generic_106541-11 sun4u 12/13/00
21:05:13 atch/s pgin/s ppgin/s pflt/s vflt/s slock/s
21:05:21 0.00 1.89 16.98 15.09 7.55 0.00
21:05:28 0.00 2.00 31.00 0.00 3.00 0.00
21:05:35 0.00 2.00 23.00 0.00 0.00 0.00
21:05:42 0.00 2.83 35.85 0.00 0.00 0.00
^C
(552)root at machine@/etc>prtconf
System Configuration: Sun Microsystems sun4u
Memory size: 384 Megabytes
System Peripherals (Software Nodes):
SUNW,UltraSPARC-IIi-Engine
packages (driver not attached)
terminal-emulator (driver not attached)
deblocker (driver not attached)
obp-tftp (driver not attached)
disk-label (driver not attached)
ufs-file-system (driver not attached)
cdfs (driver not attached)
SUNW,builtin-drivers (driver not attached)
sun-keyboard (driver not attached)
ufs-file-system (driver not attached)
chosen (driver not attached)
openprom (driver not attached)
client-services (driver not attached)
options, instance #0
aliases (driver not attached)
memory (driver not attached)
virtual-memory (driver not attached)
pci, instance #0
pci, instance #0
ebus, instance #0
auxio (driver not attached)
power (driver not attached)
SUNW,pll (driver not attached)
se, instance #0
su_pnp, instance #0
su_pnp, instance #1
ecpp, instance #0
fdthree, instance #0 (driver not attached)
eeprom (driver not attached)
flashprom (driver not attached)
beeper (driver not attached)
network, instance #0
ATY,3DCHARGER, instance #0
pci, instance #1
scsi, instance #0
disk (driver not attached)
tape (driver not attached)
sd, instance #0
sd, instance #1
sd, instance #2 (driver not attached)
sd, instance #3 (driver not attached)
sd, instance #4 (driver not attached)
sd, instance #5 (driver not attached)
sd, instance #6
sd, instance #7 (driver not attached)
sd, instance #8 (driver not attached)
sd, instance #9 (driver not attached)
sd, instance #10 (driver not attached)
sd, instance #11 (driver not attached)
sd, instance #12 (driver not attached)
sd, instance #13 (driver not attached)
sd, instance #14 (driver not attached)
scsi, instance #1
disk (driver not attached)
tape (driver not attached)
sd, instance #15 (driver not attached)
sd, instance #16 (driver not attached)
sd, instance #17 (driver not attached)
sd, instance #18 (driver not attached)
sd, instance #19 (driver not attached)
sd, instance #20 (driver not attached)
sd, instance #21 (driver not attached)
sd, instance #22 (driver not attached)
sd, instance #23 (driver not attached)
sd, instance #24 (driver not attached)
sd, instance #25 (driver not attached)
sd, instance #26 (driver not attached)
sd, instance #27 (driver not attached)
sd, instance #28 (driver not attached)
sd, instance #29 (driver not attached)
SUNW,UltraSPARC-IIi (driver not attached)
pseudo, instance #0
(555)root at fusion@/etc>/usr/platform/sun4u/sbin/prtdiag
System Configuration: Sun Microsystems sun4u SPARCengine(tm)Ultra(tm) AXi (UltraSPARC-IIi 333MHz)
System clock frequency: 111 MHz
Memory size: 384 Megabytes
========================= CPUs =========================
Run Ecache CPU CPU
Brd CPU Module MHz MB Impl. Mask
--- --- ------- ----- ------ ------ ----
0 0 0 333 2.0 12 1.3
========================= IO Cards =========================
Bus Freq
Brd Type MHz Slot Name Model
--- ---- ---- ---- -------------------------------- ----------------------
0 PCI 66 1 network-SUNW,hme
0 PCI 66 1 scsi-glm/disk (block) Symbios,53C875
0 PCI 66 1 scsi-glm/disk (block) Symbios,53C875
0 PCI 66 4 ATY,3DCHARGER-SUNW,m64B
No failures found in System
===========================
(556)root at fusion@/etc
--
Best regards,
Gaziz Nugmanov,
gaziz007 at softhome.net
Thursday, December 14, 2000, 9:44:38 AM, you wrote:
KB> Gaziz,
KB> Well, this isn't necessarily a problem with the kernel. First of all,
KB> don't rely on top. I know everybody likes it, but it's not supported by
KB> Sun and it's not always 100% accurate. Secondly, you say the "kernel" is
KB> eating up 80% of the CPU; what specific process?
KB> Are you using any other tools to monitor the system in question? For
KB> example, are you using vmstat and / or mpstat? If so, what are the other
KB> stats looking like? The system could be spending a lot of time in kernel
KB> mode because an application is trying to use way more memory than is
KB> available and therefore the system has to do a lot of paging.
KB> Another possibility is that if this is a uni-processor system, the
KB> system could be trying to context switch between process so frequently that
KB> it's spending more time switching between processes than actually running
KB> them. Since your CPU queue has gone up, what processes are waiting on the
KB> CPU?
KB> Yet another possibility is that a process is requesting lots of disk
KB> I/O. If you've got a bottleneck there, that could cause your system to
KB> spend a lot of time trying to do the I/O.
KB> It is possible that you've got a problem with the kernel. However,
KB> I'd sure try to rule everything else out first. Hope this helps...
KB> Kevin Buterbaugh
KB> LifeWay
KB> "Anyone can build a fast CPU. The trick is to build a fast system." -
KB> Seymour Cray
KB> P.S. After watching TV from 8 - 9:30 PM CST last night, I think December
KB> 13th is one of the best days in American History...
KB> Please respond to Gaziz Nugmanov <lists at lists.wysdom.com>
KB> Sent by: codeprof-admin at codeprof.com
KB> To: codeprof at codeprof.com
KB> cc:
KB> Subject: Kernel cpu leak
KB> Hello gurus,
KB> All of a sudden the kernel of Solaris 7 on SPARC started to work only
KB> for itself. cpu queue went up to 2.5-5.5
KB> In output of ps and top I see that no other processes are taking cpu
KB> time. Top says that kernel eats up 80% cpu time...
KB> Is there any way to find out what went wrong. Somewhere on a kernel
KB> module?
KB> I am patching it right now but I hardly hope that will help.
KB> PS. That was a Dec 13 :(
KB> --
KB> Best regards,
KB> Gaziz
KB> _______________________________________________
KB> codeprof mailing list
KB> codeprof at codeprof.com
KB> http://www.codeprof.com/execute/ask/?codeinfoid=43
--
Best regards,
lists mailto:lists at lists.wysdom.com
Comments
Got something to say?
You must be logged in to post a comment.
