System panics on reboot on Solaris 10 with root and /var mirrored
I've encountered this issue while installing Solaris 10 11/06 sparc on the following systems:
Netra t1 100
Netra 1400
SunFire V880
SunFire 280R
I get a a panic on reboot after editing the /etc/vfstab to boot as mirrored devices. Havte noted the panic below. Did find this bug:
http://sunsolve.sun.com/search/document.do?assetkey=1-1-6376368-1
.. which seems to point to exactly the problem I am experiencing. Although I do use a lockfs -fa in my mirroring setup, so not sure why I would still get the panic. My mirror setup is as follows:
Install Solaris 10
Patch Solaris 10
prtvtoc /dev/rdsk/c0t0d0s2 > pt.c0t0d0
fmthard -s pt.c0t0d0 /dev/rdsk/c0t1d0s2
newfs /dev/rdsk/c0t1d0s0
newfs /dev/rdsk/c0t1d0s3
newfs /dev/rdsk/c0t1d0s4
newfs /dev/rdsk/c0t1d0s5
prtvtoc /dev/rdsk/c0t2d0s2 > pt.c0t2d0
fmthard -s pt.c0t2d0 /dev/rdsk/c0t3d0s2
newfs /dev/rdsk/c0t3d0s0
metadb -a -f c0t0d0s6 c0t0d0s7
metadb -a -f c0t1d0s6 c0t1d0s7
metadb -a -f c0t2d0s6 c0t2d0s7
metadb -a -f c0t3d0s6 c0t3d0s7
metadb
metainit -f d11 1 1 c0t0d0s0
metainit -f d12 1 1 c0t1d0s0
metainit d10 -m d11
metaroot d10
metainit -f d21 1 1 c0t0d0s1
metainit -f d22 1 1 c0t1d0s1
metainit d20 -m d21
metainit -f d31 1 1 c0t0d0s3
metainit -f d32 1 1 c0t1d0s3
metainit d30 -m d31
metainit -f d41 1 1 c0t0d0s4
metainit -f d42 1 1 c0t1d0s4
metainit d40 -m d41
metainit -f d51 1 1 c0t0d0s5
metainit -f d52 1 1 c0t1d0s5
metainit d50 -m d51
metainit -f d61 1 1 c0t2d0s0
metainit -f d62 1 1 c0t3d0s0
metainit d60 -m d61
lockfs -fa
/sbin/init 6
... after reboot I attach the mirrors ...
metattach d10 d12
metattach d20 d22
metattach d30 d32
metattach d40 d42
metattach d50 d52
metattach d60 d62
... allow the mirrors to sync then update /etc/vfstab and reboot, then I get the panic.
panic[cpu2]/thread=300014b5940: free: freeing free frag, dev:0x5500000028, blk:24023, cg:0, ino:273, fs:/var
000002a10076d1a0 ufs:real_panic_v+60 (0, 18bb690, 2a10076d440, 30000f38000, 0, 60000de8880)
%l0-3: 00000000018e4800 00000000018e4888 0000030000f38310 000000000130f500
%l4-7: 00000000018eafa8 0000060000cc3dc8 0000000000000064 0000000001814000
000002a10076d250 ufs:ufs_fault_v+c8 (600014cf4c0, 18bb690, 2a10076d440, 0, 300003b57a8, 0)
%l0-3: 0000000000001fff 0000000000000000 0000060000e2cb70 0000000000000400
%l4-7: 0000000000000001 0000000000000000 00000300003b5700 0000000000000001
000002a10076d300 ufs:ufs_fault+1c (600014cf4c0, 18bb690, 5500000028, 5dd7, 0, 111)
%l0-3: 0000000000000000 00000000000303c0 0000000001863400 0000000000000000
%l4-7: 0000030000f38000 0000000000000000 000000000183d400 00000000018aa758
000002a10076d3b0 ufs:free+510 (600015920d4, 0, 5dd0, 5dd7, 0, 600015a04c0)
%l0-3: 0000000000000000 0000000000090000 00000600014d12b0 0000000000000000
%l4-7: 00000600015a0000 00000000018bb400 0000000000000000 0000000000000001
000002a10076d490 ufs:bmap_write+7ec (844, 0, 0, 6cf, 5dd7, 600014d12b0)
%l0-3: 0000000000000000 0000000000000400 0000000000005bb0 00000600014cf4c0
%l4-7: 000000007ffffc00 0000000000000002 0000000000000000 0000000000000400
000002a10076d6a0 ufs:wrip+448 (2d0, 2a10076da98, ffffffffff, 174, 600014d12b0, 8000)
%l0-3: 00000000000062d0 0000000000000000 00000000000002d0 00000000000002d0
%l4-7: 00000300003b57a8 0000000000006444 fffffffffffffd30 0000000000000174
000002a10076d810 ufs:ufs_write+580 (600014cf4c0, 2a10076da98, 8, 300003b5748, 1, 600014d12b0)
%l0-3: 00000600014d12d0 00000600014d1390 00000600014d1398 0000000000000001
%l4-7: 00000000018ba4dc 00000600014d13f0 00000300003b5700 0000000000000000
000002a10076d930 genunix:fop_write+20 (600014cf4c0, 2a10076da98, 8, 60000c03d98, 0, 121cc60)
%l0-3: 0000000000000174 0000000000002420 0000000000002000 0000000000000001
%l4-7: 0000060000c48040 0000060000deea80 0000060000deef68 0000000000000004
000002a10076d9e0 genunix:write+268 (19, 8058, 60000db33b8, 174, a, 1)
%l0-3: 0000000000000000 00000600014cf4c0 0000000000000000 000000000000704a
%l4-7: 0000000000000174 00000000000062d0 0000000000000008 000000000000000a
After the panic it looks to fsck /var but I can never come up with a clean system. I can get past the panic on some instances with an fsck but always end up with having to fsck /var on reboot ... with the folllowing output.
# fsck -o f /dev/md/rdsk/d40
/dev/md/dsk/d40 IS CURRENTLY MOUNTED READ/WRITE.
CONTINUE? y
** /dev/md/dsk/d40
** Currently Mounted on /var
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3a - Check Connectivity
** Phase 3b - Verify Shadows/ACLs
** Phase 4 - Check Reference Counts
UNREF FILE I=15866 OWNER=root MODE=100600
SIZE=324 MTIME=Feb 5 09:53 2007
RECONNECT? y
** Phase 5 - Check Cylinder Groups
CORRECT BAD CG SUMMARIES? y
CORRECTED SUMMARY FOR CG 0
FILE BITMAP WRONG
FIX? y
CORRECTED SUMMARY FOR CG 2
FILE BITMAP WRONG (CORRECTED)
FRAG BITMAP WRONG (CORRECTED)
CORRECT GLOBAL SUMMARY
SALVAGE? y
FILESYSTEM MAY STILL BE INCONSISTENT.
15644 files, 135415 used, 2872019 free (33603 frags, 354802 blocks, 1.1% fragmentation)
***** FILE SYSTEM WAS MODIFIED *****
***** FILE SYSTEM IS BAD *****
***** PLEASE RERUN FSCK *****
***** REBOOT NOW *****
Has anyone else experienced this? Any "other" workarounds that I have not found?
Thanks,
Lauri-
