FAQ: How to fix a degraded filesystem? PDF Print E-mail
User Rating: / 1
PoorBest 
Written by Michael Felt   

I ran into this by chance, and am writing it up before they all go away - forever.

A degraded filesystem seems to be one that does not have all the meta data agreeing. To protect the data that is there the filesystem is mounted, but in ro,degraded status (read-only, degraded).

michael@x054:[/home/michael]mount | grep export
        /dev/lvExport    /export          jfs2   Mar 12 21:16 ro,degraded
        /dev/lvLpp       /export/lpp      jfs2   Mar 12 21:16 ro,degraded
        /dev/lvSpot      /export/spot     jfs2   Mar 12 21:16 ro,degraded

p.s. the problem seems to be an iscsi "delivery" issue - some/all communication with the iscsi (read NAS) server has been lost. See more below.

Now that this has come back again (borrowed an old NAS device :wink:) I can verify that the solution path is fairly simple. Just unmount and mount again. (because the jfs2log, VGDA, etc gets read again, and the data "loss" is resolved). The hardest part may be getting the filesystem to unmount.

 

michael@x054:[/home/michael]unmount -f /export/spot 
michael@x054:[/home/michael]fsck -n /export/spot

The current volume is: /dev/lvSpot

Primary superblock is valid.
*** Phase 1 - Initial inode scan
*** Phase 2 - Process remaining directories
*** Phase 3 - Process remaining files
*** Phase 4 - Check inode allocation map
*** Phase 5 - Check block allocation map
File system is clean.
File system is clean but is marked dirty.
         Run fsck with -p parameter to fix.
michael@x054:[/home/michael]fsck -p /dev/lvSpot

The current volume is: /dev/lvSpot
Open volume exclusive read or write returned, rc = 16
fsck: 0507-289 Device unavailable or locked by another process.
          Cannot continue.

# chfs -A no /export/spot

as well as for the other degraded filesystems and forced that volumn group off, then on again. As I became more experienced in this (the things you can learn from a failing iscsi system) the real key seems to be to offline the iscsi connection and connect the disk again. (easy way is #rmdev -l hdisk# -- do not bother with the -d then you can use mkdev -l hdisk# to bring it back online. If you use #rmdev -dl hdisk# you will need to use # cfgmgr to get the disk discovered again.)

If this is iscsi, and your disk does not reconnect immediately you can try my "trick", change the name of the iscsi disk/volume to something else and then change it back again. For me this seems to reset whatever is broken in terms of communication. FYI - the information on the disks has ALWAYS been okay.

The switch to degraded,ro seems to come fast enough that no damage can be done. At least I have not lost any data. The ultimate solution - new NAS firmware, and if that fails, a better NAS device.
 
< Prev   Next >
61 queries executed
1
SET sql_mode = 'MYSQL40'
2 SELECT folder, element, published, params FROM jos_mambots WHERE published >= 1 AND access <= 0 AND folder = 'system' ORDER BY ordering
3 SELECT template FROM jos_templates_menu WHERE client_id = 0 AND ( menuid = 0 OR menuid = 312 ) ORDER BY menuid DESC LIMIT 1
4 DELETE FROM jos_session WHERE ( ( time < '1548314259' ) AND guest = 0 AND gid > 0 ) OR ( ( time < '1548314259' ) AND guest = 1 AND userid = 0 )
5 SELECT COUNT(*) FROM jos_stats_agents WHERE agent = 'Unknown' AND type = 0
6 UPDATE jos_stats_agents SET hits = ( hits + 1 ) WHERE agent = 'Unknown' AND type = 0
7 SELECT COUNT(*) FROM jos_stats_agents WHERE agent = 'Unknown' AND type = 1
8 UPDATE jos_stats_agents SET hits = ( hits + 1 ) WHERE agent = 'Unknown' AND type = 1
9 SELECT COUNT(*) FROM jos_stats_agents WHERE agent = 'com' AND type = 2
10 UPDATE jos_stats_agents SET hits = ( hits + 1 ) WHERE agent = 'com' AND type = 2
11 SELECT * FROM jos_menu WHERE id = 312
12 SELECT a.*, u.name AS author, u.usertype, cc.name AS category, s.name AS section, g.name AS groups, s.published AS sec_pub, cc.published AS cat_pub, s.access AS sec_access, cc.access AS cat_access, s.id AS sec_id, cc.id as cat_id FROM jos_content AS a LEFT JOIN jos_categories AS cc ON cc.id = a.catid LEFT JOIN jos_sections AS s ON s.id = cc.section AND s.scope = 'content' LEFT JOIN jos_users AS u ON u.id = a.created_by LEFT JOIN jos_groups AS g ON a.access = g.id WHERE a.id = 543 AND ( a.state = 1 OR a.state = -1 ) AND ( a.publish_up = '0000-00-00 00:00:00' OR a.publish_up <= '2019-01-24 08:32' ) AND ( a.publish_down = '0000-00-00 00:00:00' OR a.publish_down >= '2019-01-24 08:32' ) AND a.access <= 0
13 SELECT a.id FROM jos_content AS a WHERE a.catid = 19 AND a.state = 1 AND a.access <= 0 AND ( a.state = 1 OR a.state = -1 ) AND ( a.publish_up = '0000-00-00 00:00:00' OR a.publish_up <= '2019-01-24 08:32' ) AND ( a.publish_down = '0000-00-00 00:00:00' OR a.publish_down >= '2019-01-24 08:32' ) ORDER BY a.created DESC
14 SELECT ROUND( v.rating_sum / v.rating_count ) AS rating, v.rating_count FROM jos_content AS a LEFT JOIN jos_content_rating AS v ON a.id = v.content_id WHERE a.id = 543
15 UPDATE jos_content SET hits = ( hits + 1 ) WHERE id = '543'
16 SELECT hits FROM jos_core_log_items WHERE time_stamp = '2019-01-24' AND item_table = '#__content' AND item_id = '543'
17 UPDATE jos_core_log_items SET hits = ( hits + 1 ) WHERE time_stamp = '2019-01-24' AND item_table = '#__content' AND item_id = '543'
18 SELECT folder, element, published, params FROM jos_mambots WHERE access <= 0 AND folder = 'content' ORDER BY ordering
19 SELECT value FROM jos_bookmarks_prefs WHERE userid = '-1' AND category = 'params' AND name = 'urlkey'
20 SELECT value FROM jos_bookmarks_prefs WHERE userid = '-1' AND category = 'params' AND name = 'snapshotactiv'
21 SELECT value FROM jos_bookmarks_prefs WHERE userid = '-1' AND category = 'params' AND name = 'snapshotother'
22 SELECT id FROM jos_mambots WHERE element = 'mosbookmarks' AND folder = 'content'
23 SELECT * FROM jos_mambots WHERE id = '32'
24 SELECT name, value FROM jos_bookmarks_prefs WHERE userid = '-1' AND category = 'params' ORDER BY name
25 SELECT c.id, c.parent FROM jos_bookmarks_categories as c WHERE c.id = -1
26 SELECT * FROM jos_bookmarks_columns WHERE category='' AND custom='0'
27 SELECT * FROM jos_bookmarks_columns WHERE category='' ORDER BY ordering
28 SELECT * FROM jos_bookmarks_columns WHERE category='admin'
29 SELECT * FROM jos_bookmarks_columns WHERE category='admin' ORDER BY ordering
30 SELECT id FROM jos_mambots WHERE element = 'plugin_jw_allvideos' AND folder = 'content'
31 SELECT * FROM jos_mambots WHERE id = '27'
32 SELECT id FROM jos_mambots WHERE element = 'plugin_jw_allvideos' AND folder = 'content'
33 SELECT * FROM jos_mambots WHERE id = '27'
34 SELECT id FROM jos_mambots WHERE element = 'plugin_jw_allvideos' AND folder = 'content'
35 SELECT * FROM jos_mambots WHERE id = '27'
36 SELECT id FROM jos_mambots WHERE element = 'plugin_jw_allvideos' AND folder = 'content'
37 SELECT * FROM jos_mambots WHERE id = '27'
38 SELECT params FROM jos_mambots WHERE element = 'mosemailcloak' AND folder = 'content'
39 SELECT a.* FROM jos_components AS a WHERE ( a.admin_menu_link = 'option=com_syndicate' OR a.admin_menu_link = 'option=com_syndicate&hidemainmenu=1' ) AND a.option = 'com_syndicate'
40 SELECT m.id FROM jos_modules AS m WHERE m.module = 'mod_rssfeed' AND m.published = 1
41 SELECT id, title, module, position, content, showtitle, params FROM jos_modules AS m INNER JOIN jos_modules_menu AS mm ON mm.moduleid = m.id WHERE m.published = 1 AND m.access <= 0 AND m.client_id != 1 AND ( mm.menuid = 0 OR mm.menuid = 312 ) ORDER BY ordering
42 SELECT id FROM jos_menu WHERE link = 'index.php?option=com_search' AND published = 1
43 SELECT m.*, sum(case when p.published=1 then 1 else 0 end) as cnt FROM jos_menu AS m LEFT JOIN jos_menu AS p ON p.parent = m.id WHERE m.menutype='rt_splitSuckerFish' AND m.published='1' AND m.access <= '0' GROUP BY m.id ORDER BY m.parent, m.ordering
44 SELECT m.* FROM jos_menu AS m WHERE menutype='rt_splitSuckerFish' AND m.published='1'
45 SELECT * FROM jos_banner WHERE showBanner=1
46 SELECT id FROM jos_mambots WHERE element = 'mosbookmarks' AND folder = 'content'
47 SELECT * FROM jos_mambots WHERE id = '32'
48 SELECT name, value FROM jos_bookmarks_prefs WHERE userid = '-1' AND category = 'params' ORDER BY name
49 SELECT c.id, c.parent FROM jos_bookmarks_categories as c WHERE c.id = -1
50 SELECT * FROM jos_bookmarks_columns WHERE category='' AND custom='0'
51 SELECT * FROM jos_bookmarks_columns WHERE category='' ORDER BY ordering
52 SELECT * FROM jos_bookmarks_columns WHERE category='admin'
53 SELECT * FROM jos_bookmarks_columns WHERE category='admin' ORDER BY ordering
54 SELECT id FROM jos_mambots WHERE element = 'plugin_jw_allvideos' AND folder = 'content'
55 SELECT * FROM jos_mambots WHERE id = '27'
56 SELECT id FROM jos_mambots WHERE element = 'plugin_jw_allvideos' AND folder = 'content'
57 SELECT * FROM jos_mambots WHERE id = '27'
58 SELECT id FROM jos_mambots WHERE element = 'plugin_jw_allvideos' AND folder = 'content'
59 SELECT * FROM jos_mambots WHERE id = '27'
60 SELECT id FROM jos_mambots WHERE element = 'plugin_jw_allvideos' AND folder = 'content'
61 SELECT * FROM jos_mambots WHERE id = '27'