HOWTO: Cleanup a PVMISSING disk PDF Print E-mail
User Rating: / 40
PoorBest 
Written by Michael Felt   

While wondering around the web I found a blog with comparisions between Solaris 10 and AIX 6. One of them is this blog with several AIX articles. One (scroll down a bit if you follow the link) was on how to trick the ODM into letting you remove a MISSING disk. Anyone who has followed an AIX administration course (well the advanced one) knows that there is a command to do all this for you! Even if editting ODM is fun for some of us (RAWR!).

Below, my extended guide for removing a MISSINGPV from the other disks VGDA and AIX ODM. 

Introduction

How a disk becomes PVMISSING is irrelevant. These things happen. Getting the system repaired is relevant! So, the simpler way! to correct volume group VGDA and AIX ODM.

The single command we will be using to remove the disk is:
ldeletepv -v VGID -p PVID

But, before we do, there are a number of steps we should follow as a matter of "best practice".

CASE: While the volume group is offline, maintenance is performed on the disks. One disk is/was damaged beyond repair, or replaced during the process. Now back at AIX the volumes are to be reactivated.

root@aix530:[/]lsvg -p vgExport 
0516-010 : Volume group must be varied on; use varyonvg command.
root@aix530:[/]varyonvg vgExport
PV Status:      hdisk1  00c39b8d69c45344        PVACTIVE
                hdisk2  00c39b8d043427b6        PVMISSING


The disk hdisk2 is PVMISSING. We assume hdisk2 with PVID 00c39b8d043427b6 is physically destroyed. All the data is lost; however, the AIX ODM and the VGDA on all the other disks in the volume group do not know this yet.

First document what is lost. We need to know which logical volumes are (were) on the missing disk. Normally we could use lspv -l hdiskX; (new: undocumented variation: lspv -l PVID) however, with the disk missing, this version of the command will not work. Instead, we use the VGID (volume group identifer).

1. Query the VGDA of the working disk to get the VGID and PVID of all disks in the volume group
root@aix530:[/]lqueryvg -p hdisk1 -vPt
Physical:       00c39b8d69c45344                2   0 
                00c39b8d043427b6                1   0 
VGid:           00c39b8d00004c000000011169c45a4b

2. Get a list of all the logical volumes on the missing disk
root@aix530:[/]lspv -l -v 00c39b8d00004c000000011169c45a4b hdisk2
hdisk2:
LV NAME               LPs   PPs   DISTRIBUTION          MOUNT POINT
lvTest                512   512   109..108..108..108..79 /scratch
loglv00               1     1     00..00..00..00..01    N/A

(Note: lspv -l  00c39b8d043427b6 should give us the same output!)

3. Verify all filesystems are unmounted.
root@aix530:[/]lsvg -l vgExport
vgExport:
LV NAME             TYPE       LPs   PPs   PVs  LV STATE      MOUNT POINT
lvExport            jfs2       416   416   1    closed/syncd  /export
lvTest              jfs        512   512   1    closed/syncd  /scratch
loglv00             jfslog     1     1     1    closed/syncd  N/A


With this info I know that any data in /scratch is suspect, and should be restored from a backup.

4. Remove the logical volumes from the volume group before deleting the VGDA from the other disks.
root@aix530:[/]rmfs /scratch
rmfs:  0506-936  Cannot read superblock on /dev/lvTest.
rmfs:  0506-936  Cannot read superblock on /scratch.
rmfs: Unable to clear superblock on /scratchrmlv: Logical volume lvTest is removed.
root@aix530:[/]rmlv loglv00
Warning, all data contained on logical volume loglv00 will be destroyed.
rmlv: Do you wish to continue? y(es) n(o)? y
rmlv: Logical volume loglv00 is removed.
root@aix530:[/]lsvg -p vgExport
vgExport:
PV_NAME           PV STATE          TOTAL PPs   FREE PPs    FREE DISTRIBUTION
hdisk1            active            511         95          00..00..00..00..95
hdisk2            missing           542         29          51..18..51..51..51
root@aix530:[/]lsvg -l vgExport
vgExport:
LV NAME             TYPE       LPs   PPs   PVs  LV STATE      MOUNT POINT
lvExport            jfs2       416   416   1   
closed/syncd  /export

5. The volume group has been prepared - all damaged logical volume definitions have been removed. All that is remaining for cleanup is to remove the definition of the damaged disk from the VGDA of the remaining disk(s).
root@aix530:[/]ldeletepv -g 00c39b8d00004c000000011169c45a4b -p 00c39b8d043427b6


Note: there is no output for the above command when all proceeds accordingly.

Now the regular AIX commands to verify VGDA and ODM are in order.
root@aix530:[/]lsvg -p vgExport                                                
vgExport:
PV_NAME           PV STATE          TOTAL PPs   FREE PPs    FREE DISTRIBUTION
hdisk1            active            511         95          00..00..00..00..95
root@aix530:[/]mount /export
root@aix530:[/]lsvg -l vgExport
vgExport:
LV NAME             TYPE       LPs   PPs   PVs  LV STATE      MOUNT POINT
lvExport            jfs2       416   416   1    open/syncd    /export

6. Various steps that I will only list here:
a. add a new disk to the volume group (extendvg)
b. remake the deleted logical partitions (mklv)
c. format, as needed, the log logical volumes (logform)
d. create the filesystems (crfs, or use smit)
e. restore the data from a backup (restore, tar, cpio, etc.)

Summary

This procedure is much less error prone than using ODM commands. All the commands demonstrated here have been available in AIX for disk management since at least 1995 (when AIX 4 first came out). They may have been in AIX 3 as well, taking it back to 1991 or earlier.

Important commands to review

lspv -l -v VGID hdiskX
lqueryvg
ldeletepv

 
< Prev   Next >
78 queries executed
1
SET sql_mode = 'MYSQL40'
2 SELECT folder, element, published, params FROM jos_mambots WHERE published >= 1 AND access <= 0 AND folder = 'system' ORDER BY ordering
3 SELECT template FROM jos_templates_menu WHERE client_id = 0 AND ( menuid = 0 OR menuid = 193 ) ORDER BY menuid DESC LIMIT 1
4 DELETE FROM jos_session WHERE ( ( time < '1548315094' ) AND guest = 0 AND gid > 0 ) OR ( ( time < '1548315094' ) AND guest = 1 AND userid = 0 )
5 SELECT COUNT(*) FROM jos_stats_agents WHERE agent = 'Unknown' AND type = 0
6 UPDATE jos_stats_agents SET hits = ( hits + 1 ) WHERE agent = 'Unknown' AND type = 0
7 SELECT COUNT(*) FROM jos_stats_agents WHERE agent = 'Unknown' AND type = 1
8 UPDATE jos_stats_agents SET hits = ( hits + 1 ) WHERE agent = 'Unknown' AND type = 1
9 SELECT COUNT(*) FROM jos_stats_agents WHERE agent = 'com' AND type = 2
10 UPDATE jos_stats_agents SET hits = ( hits + 1 ) WHERE agent = 'com' AND type = 2
11 SELECT * FROM jos_menu WHERE id = 193
12 SELECT a.*, u.name AS author, u.usertype, cc.name AS category, s.name AS section, g.name AS groups, s.published AS sec_pub, cc.published AS cat_pub, s.access AS sec_access, cc.access AS cat_access, s.id AS sec_id, cc.id as cat_id FROM jos_content AS a LEFT JOIN jos_categories AS cc ON cc.id = a.catid LEFT JOIN jos_sections AS s ON s.id = cc.section AND s.scope = 'content' LEFT JOIN jos_users AS u ON u.id = a.created_by LEFT JOIN jos_groups AS g ON a.access = g.id WHERE a.id = 174 AND ( a.state = 1 OR a.state = -1 ) AND ( a.publish_up = '0000-00-00 00:00:00' OR a.publish_up <= '2019-01-24 08:46' ) AND ( a.publish_down = '0000-00-00 00:00:00' OR a.publish_down >= '2019-01-24 08:46' ) AND a.access <= 0
13 SELECT a.id FROM jos_content AS a WHERE a.catid = 19 AND a.state = 1 AND a.access <= 0 AND ( a.state = 1 OR a.state = -1 ) AND ( a.publish_up = '0000-00-00 00:00:00' OR a.publish_up <= '2019-01-24 08:46' ) AND ( a.publish_down = '0000-00-00 00:00:00' OR a.publish_down >= '2019-01-24 08:46' ) ORDER BY a.ordering
14 SELECT ROUND( v.rating_sum / v.rating_count ) AS rating, v.rating_count FROM jos_content AS a LEFT JOIN jos_content_rating AS v ON a.id = v.content_id WHERE a.id = 174
15 UPDATE jos_content SET hits = ( hits + 1 ) WHERE id = '174'
16 SELECT hits FROM jos_core_log_items WHERE time_stamp = '2019-01-24' AND item_table = '#__content' AND item_id = '174'
17 UPDATE jos_core_log_items SET hits = ( hits + 1 ) WHERE time_stamp = '2019-01-24' AND item_table = '#__content' AND item_id = '174'
18 SELECT folder, element, published, params FROM jos_mambots WHERE access <= 0 AND folder = 'content' ORDER BY ordering
19 SELECT value FROM jos_bookmarks_prefs WHERE userid = '-1' AND category = 'params' AND name = 'urlkey'
20 SELECT value FROM jos_bookmarks_prefs WHERE userid = '-1' AND category = 'params' AND name = 'snapshotactiv'
21 SELECT value FROM jos_bookmarks_prefs WHERE userid = '-1' AND category = 'params' AND name = 'snapshotother'
22 SELECT id FROM jos_mambots WHERE element = 'mosbookmarks' AND folder = 'content'
23 SELECT * FROM jos_mambots WHERE id = '32'
24 SELECT name, value FROM jos_bookmarks_prefs WHERE userid = '-1' AND category = 'params' ORDER BY name
25 SELECT c.id, c.parent FROM jos_bookmarks_categories as c WHERE c.id = -1
26 SELECT * FROM jos_bookmarks_columns WHERE category='' AND custom='0'
27 SELECT * FROM jos_bookmarks_columns WHERE category='' ORDER BY ordering
28 SELECT * FROM jos_bookmarks_columns WHERE category='admin'
29 SELECT * FROM jos_bookmarks_columns WHERE category='admin' ORDER BY ordering
30 SELECT id FROM jos_mambots WHERE element = 'plugin_jw_allvideos' AND folder = 'content'
31 SELECT * FROM jos_mambots WHERE id = '27'
32 SELECT id FROM jos_mambots WHERE element = 'plugin_jw_allvideos' AND folder = 'content'
33 SELECT * FROM jos_mambots WHERE id = '27'
34 SELECT id FROM jos_mambots WHERE element = 'plugin_jw_allvideos' AND folder = 'content'
35 SELECT * FROM jos_mambots WHERE id = '27'
36 SELECT id FROM jos_mambots WHERE element = 'plugin_jw_allvideos' AND folder = 'content'
37 SELECT * FROM jos_mambots WHERE id = '27'
38 SELECT params FROM jos_mambots WHERE element = 'mosemailcloak' AND folder = 'content'
39 SELECT a.* FROM jos_components AS a WHERE ( a.admin_menu_link = 'option=com_syndicate' OR a.admin_menu_link = 'option=com_syndicate&hidemainmenu=1' ) AND a.option = 'com_syndicate'
40 SELECT m.id FROM jos_modules AS m WHERE m.module = 'mod_rssfeed' AND m.published = 1
41 SELECT id, title, module, position, content, showtitle, params FROM jos_modules AS m INNER JOIN jos_modules_menu AS mm ON mm.moduleid = m.id WHERE m.published = 1 AND m.access <= 0 AND m.client_id != 1 AND ( mm.menuid = 0 OR mm.menuid = 193 ) ORDER BY ordering
42 SELECT a.id, a.introtext, a.fulltext , a.images, a.attribs, a.title, a.state FROM jos_content AS a INNER JOIN jos_categories AS cc ON cc.id = a.catid INNER JOIN jos_sections AS s ON s.id = a.sectionid WHERE a.state = 1 AND a.access <= 0 AND cc.access <= 0 AND s.access <= 0 AND (a.publish_up = '0000-00-00 00:00:00' OR a.publish_up <= '2019-01-24 08:46' ) AND (a.publish_down = '0000-00-00 00:00:00' OR a.publish_down >= '2019-01-24 08:46' ) AND a.catid = 44 AND cc.published = 1 AND s.published = 1 ORDER BY a.ordering LIMIT 15
43 SELECT id FROM jos_mambots WHERE element = 'mosbookmarks' AND folder = 'content'
44 SELECT * FROM jos_mambots WHERE id = '32'
45 SELECT name, value FROM jos_bookmarks_prefs WHERE userid = '-1' AND category = 'params' ORDER BY name
46 SELECT c.id, c.parent FROM jos_bookmarks_categories as c WHERE c.id = -1
47 SELECT * FROM jos_bookmarks_columns WHERE category='' AND custom='0'
48 SELECT * FROM jos_bookmarks_columns WHERE category='' ORDER BY ordering
49 SELECT * FROM jos_bookmarks_columns WHERE category='admin'
50 SELECT * FROM jos_bookmarks_columns WHERE category='admin' ORDER BY ordering
51 SELECT id FROM jos_mambots WHERE element = 'plugin_jw_allvideos' AND folder = 'content'
52 SELECT * FROM jos_mambots WHERE id = '27'
53 SELECT id FROM jos_mambots WHERE element = 'plugin_jw_allvideos' AND folder = 'content'
54 SELECT * FROM jos_mambots WHERE id = '27'
55 SELECT id FROM jos_mambots WHERE element = 'plugin_jw_allvideos' AND folder = 'content'
56 SELECT * FROM jos_mambots WHERE id = '27'
57 SELECT id FROM jos_mambots WHERE element = 'plugin_jw_allvideos' AND folder = 'content'
58 SELECT * FROM jos_mambots WHERE id = '27'
59 SELECT id FROM jos_menu WHERE link = 'index.php?option=com_search' AND published = 1
60 SELECT m.*, sum(case when p.published=1 then 1 else 0 end) as cnt FROM jos_menu AS m LEFT JOIN jos_menu AS p ON p.parent = m.id WHERE m.menutype='rt_splitSuckerFish' AND m.published='1' AND m.access <= '0' GROUP BY m.id ORDER BY m.parent, m.ordering
61 SELECT m.* FROM jos_menu AS m WHERE menutype='rt_splitSuckerFish' AND m.published='1'
62 SELECT * FROM jos_banner WHERE showBanner=1
63 SELECT id FROM jos_mambots WHERE element = 'mosbookmarks' AND folder = 'content'
64 SELECT * FROM jos_mambots WHERE id = '32'
65 SELECT name, value FROM jos_bookmarks_prefs WHERE userid = '-1' AND category = 'params' ORDER BY name
66 SELECT c.id, c.parent FROM jos_bookmarks_categories as c WHERE c.id = -1
67 SELECT * FROM jos_bookmarks_columns WHERE category='' AND custom='0'
68 SELECT * FROM jos_bookmarks_columns WHERE category='' ORDER BY ordering
69 SELECT * FROM jos_bookmarks_columns WHERE category='admin'
70 SELECT * FROM jos_bookmarks_columns WHERE category='admin' ORDER BY ordering
71 SELECT id FROM jos_mambots WHERE element = 'plugin_jw_allvideos' AND folder = 'content'
72 SELECT * FROM jos_mambots WHERE id = '27'
73 SELECT id FROM jos_mambots WHERE element = 'plugin_jw_allvideos' AND folder = 'content'
74 SELECT * FROM jos_mambots WHERE id = '27'
75 SELECT id FROM jos_mambots WHERE element = 'plugin_jw_allvideos' AND folder = 'content'
76 SELECT * FROM jos_mambots WHERE id = '27'
77 SELECT id FROM jos_mambots WHERE element = 'plugin_jw_allvideos' AND folder = 'content'
78 SELECT * FROM jos_mambots WHERE id = '27'