Subject: [TAG] Diagnosing a Linux crash

From: Tom Brown <[email-masked]>
Date: Tue, 16 Mar 2004 16:43:55 -0500

OK guys, here's a n00b question for you that probably crosses over into 
Sys Admin territory.

What steps should someone follow after Linux crashes to figure out what 
went wrong?
Where do I start, and where do I look for clues?
Are all the logs found in /var/log, or are there others?
In what order should I look at the logs, and what should I look for?
Any pro-active steps I should be taking to get more info, should it 
happen again?

The specifics of my case: my file server (a 750 Mhz Athlon running Suse 
9) simply locked up, and I couldn't get anything to display (GUI or 
command line). I knew the machine was in trouble, when it didn't respond 
to pings. I had to hit the reset button to get it back (and deal with 
fsck, naturally). Funny thing is, the system clock reset itself to 28 
minutes after midnight (when it should have read the middle of the 
afternoon), but didn't loose the date. Odd, that. The machine's been 
running 24/7 for about three weeks now (I set it up around then), and no 
sign of problems until now.

I searched the archives first for relevant articles, but didn't see any.

Thanks, as always.

Tom Brown

P.S., You'll never guess what Mozilla Thunderbird's spell checker 
recommends for the word "fsck". LOL!
_______________________________________________
TAG mailing list
[email-masked]
http://linuxgazette.net/mailman/listinfo/tag

---

From: Thomas Adam <[email-masked]>
Date: Tue, 16 Mar 2004 21:53:51 +0000 (GMT)

> Any pro-active steps I should be taking to get more info, should it 

It depends what you think went wrong. Essentially:

/var/log/messages

is where syslogd will dump all its data and so is the best place to look.
But there may well be application specific data in /var/log
(XFree86.0.log) is one such example.
 
> command line). I knew the machine was in trouble, when it didn't respon

This might be framebuffer related. At the lilo/grub prompt, type:

linux video=vga16:off

to see if that has any effect.

There have been snippets of these effects metioned in the past. The one
that springs to mind is:

http://linuxgazette.net/issue74/tag/9.html

-- Thomas Adam

=====
"The Linux Weekend Mechanic" -- http://linuxgazette.net
"TAG Editor"                 -- http://linuxgazette.net

"<shrug> We'll just save up your sins, Thomas, and punish 
you for all of them at once when you get better. The 
experience will probably kill you. :)"

 -- Benjamin A. Okopnik (Linux Gazette Technical Editor)

___________________________________________________________
Yahoo! Messenger - Communicate instantly..."Ping" 
your friends today! Download Messenger Now 
http://uk.messenger.yahoo.com/download/index.html
_______________________________________________
TAG mailing list
[email-masked]
http://linuxgazette.net/mailman/listinfo/tag

---

From: Neil Youngman <[email-masked]>
Date: Wed, 17 Mar 2004 04:21:54 +0000

> sign of problems until now.

The only time I've had Linux lock up this badly, It was a memory issue. 
Nothing showed in the logs and memtest86 didn't show anything, but replacing 
the memory changed the situation from lock ups twice a day to total 
reliability.

Neil Youngman

---

From: Karl-Heinz Herrmann <[email-masked]>
Date: Wed, 17 Mar 2004 09:38:26 +0100

Hi,

On Tue, 16 Mar 2004 16:43:55 -0500

> Any pro-active steps I should be taking to get more info, should it 

Well, generally what I would do is:
hunt down all info of the last crash (logfiles, register dump,...)
 
Usually I would also suggest: /usr/src/linux/Documentation/BUG-HUNTING
but in your case it won't help much. Somewhere is a txt file decribing
what to do with kernerl register dumps (finding in what routine it
actually crashed).

> command line). I knew the machine was in trouble, when it didn't respond 

That's bad -- complete lockup and no register dump on screen leaves
pretty few clues around for diagnostics. Any drive LED's on?

There are ways of still getting kernel info (pro active steps): 

* plug an old printer into the lpX port and declare it the system console
  (kernel kompile parameter, and I don't know how exactly you activate it
  -- maybe inittab).

* When running switch to system console (Alt-Ctrl-F10 on SuSE) and leave
  it there. It *might* show a kernel oops/panic there next crash.

* search SuSE config for Magic SysRequest keys -- the function should be
  compiled in the kernel but has to be activated. Then you can press 
  weird key-combinations like Alt-Ctrl-Sysreq-R for register dump, ...S
  for disk sync,... see /usr/src/linux/Documentation for details.

* File server? What hardware? I had SCSI disks locking my system for 
  various reasons (Tagged queuing incompatibilites of indiv. drives, too
  long cables,..)

K.-H.