Subject: Re: [TAG] Diagnosing a Linux crash

From: Tom Brown <[email-masked]>
Date: Thu, 18 Mar 2004 12:40:24 -0500

Karl,

    Thanks for the input, but I think you're giving me a lot more credit 
than I deserve. A lot of your suggestions went over my head. Learning to 
install and configure Linux is one thing. Learning how to do an autopsy 
seems to be quite another!

| hunt down all info of the last crash (logfiles, register dump,...)

Yes, well, I looked at the messages log, but saw only a gap time-wise between cron processing around 4 in the morning, and the time of the crash. I'm not sure which of the other logs are important in that case. Where do I find the register dump (although I suspect it won't make much sense to me, rather like those register dumps you get in Windows XP)? 

| File server? What hardware?

Two regular IDE drives: one 20GB and one 250GB, so SCSI shouldn't be an 
issue.

I'm going to keep your response handy -- several things to try. 
Meantime, I realized I was booting the thing into runlevel 5 (rather 
stupid, actually), so I've since changed it to 3. If it is, as someone 
suggested, a framebuffer problem, maybe that will solve it for now. I'm 
using a real old Voodoo 3 card I scrounged from my parts bin. If it 
happens again, I'll have to tear the machine apart and start playing 
with the memory, as someone else here suggested.

Thanks.

Tom
_______________________________________________
TAG mailing list
[email-masked]
http://linuxgazette.net/mailman/listinfo/tag

---

From: Thomas Adam <[email-masked]>
Date: Thu, 18 Mar 2004 18:20:50 +0000 (GMT)

> seems to be quite another!

That's because generally one doesn't do it quite like that. Problem
diagnosis is situation dependant. In any givem situation there is often a
small set of files and related information that you can analyse without
having to worry about the rest of the system.

Granted, this is related to how much information one is told at the time
(if you've been on this list for as long as I have, you'll come to realise
that usually we don't get any), and whether or not the person has tried to
remedy it.
 
In general though, poking around, taking an aspect of your system, looking
at what it does, and how is all related and helful to you when you have to
come to diagnose anything.

> Yes, well, I looked at the messages log, but saw only a gap time-wise
> between cron processing around 4 in the morning, and the time of the
> crash. I'm not sure which of the other logs are important in that case.
> Where do I find the register dump (although I suspect it won't make much
> sense to me, rather like those register dumps you get in Windows XP)? 

Syslogd would have logged it, if the problem was software related, and
indeed if the said program produced any errors.

> suggested, a framebuffer problem, maybe that will solve it for now. I'm 

That was I who suggested a framebuffer problem. I'll re-iterate that:

linux video=vga16:off

is what you can try at the LILO/GRUB prompt.

> with the memory, as someone else here suggested.

It *might* be memory, but as the link I have you last time around said,
memory problems tend to be more 'visible' in the sense that you get a lot
of applications SEGFAULTing and SEGABRTing for no apparant reason. In such
instances, installing and running 'memtest86' is usually of help.

-- Thomas Adam

=====
"The Linux Weekend Mechanic" -- http://linuxgazette.net
"TAG Editor"                 -- http://linuxgazette.net

"<shrug> We'll just save up your sins, Thomas, and punish 
you for all of them at once when you get better. The 
experience will probably kill you. :)"

 -- Benjamin A. Okopnik (Linux Gazette Technical Editor)

___________________________________________________________
Yahoo! Messenger - Communicate instantly..."Ping" 
your friends today! Download Messenger Now 
http://uk.messenger.yahoo.com/download/index.html
_______________________________________________
TAG mailing list
[email-masked]
http://linuxgazette.net/mailman/listinfo/tag

---

From: Karl-Heinz Herrmann <[email-masked]>
Date: Thu, 18 Mar 2004 21:17:04 +0100

On Thu, 18 Mar 2004 18:20:50 +0000 (GMT)
> indeed if the said program produced any errors.

Not necessarily -- Most of the time I had the great luck of oopes and
kernel crashes occuring in the scsi layer, often hardware problems. If
the scsi layer is in trouble nothing will get written to disk. What's
software related regarding the kernel? The kernel deals with hardware,
and it's supposed to handle error conditions gracefully, i.e. not just
freeze without a hints whats gone wrong. But there are situations where
the kernel doesn't have a chance of leaving hints on harddrive. 

Then a few thing might be useful: (to Tom B) 

- run the box without X as you suggested yourself 

- switch to konsole 10 (sys messages). Even if the kernel might not be
  able to leave a trace on the HD it might give a hint here.

- for any reg dump on konsole 10 or syslog you *need* to run it through 
  ksymoops to make it useful. That's something nobody can take over
  because it has to be done on your system, with your kernel and kernel
  symbols. I hope SuSE set everything needed up correctly. 
  As you mentioned WinXX reg dumps -- in Linux they are about as useful
  as in WinXX, but Linux has the tools to decode them (ksymoops) to make
  them useful.

- If you gain any information (and yes you will have to note it down on
  paper and give it to ksymoops after reboot) you can try here oo with the
  kernel people. 

- this is an *option* to follow if you are interested *why* your system 
  crashes. As it's crashing very irregular this is a rather difficult   
  situation and a very slow process.  But "the machine was dead on the  
  next morning" wont help you next time it happens. Above mentioned things
  (along with running a  printer or a serial line console) would also help
  in getting the syslog right up to the crash. 

suggested reading:

/usr/src/linux/Documentation/oops-tracing.txt

ksymoops man page

But I have to say that often enough I also do not try to hunt spurious
crashes which *do* occasionally happen. Either hardware causes or whatever. 
You always can try a different kernel or simply hope for the best. 

Still -- keeping the system on konsole 10 is not a difficult thing to do
and it just might give you something useful next time (note it down for
ksymoops if it's a oops or panic).

SuSE has memtest as bootoption -- run it if you suspect the RAM, run it
long (several passes) and the full test suite if you don't find any
errors on the first go.

K.-H.

---

From: Tom Brown <[email-masked]>
Date: Thu, 18 Mar 2004 18:08:20 -0500

Thanks Karl and Thomas. This is the starting point I needed. (For one 
thing, I didn't even know about konsole10: looks helpful).  I just wish 
I had more from the crash than just a black screen, but that's what I 
get for running X on bootup for a file server. Between the two of you, I 
think I have the answers I was looking for when I started this thread: 
not what went wrong exactly, but how to dig in, and try to figure it out 
for myself.

Oh, Thomas, when I rebooted to runlevel 3, I entered that video setting 
you suggested as well.

I just know I'll be back with more questions, though. One way or 
another, I'll figure this Linux thing out.

Thanks again, guys. Your help, as always, is much appreciated.

Tom Brown