トップ 一覧 検索 ヘルプ RSS ログイン

FreeBSD gmirror 障害復旧手順例 202008の変更点

  • 追加された行はこのように表示されます。
  • 削除された行はこのように表示されます。
*FreeBSD-11 でディスク障害が発生した。
*FreeBSD gmirror GPT 3T HDD トリプルミラーでの障害
*セクタエラー
*障害復旧手順の記録

*zfs ではない
*GPT のバックアップ領域と gmirror の管理領域が重複するため、GPTではディスク丸ごと gmirror できない。
*FreeBSD GPT gmirror 失敗
*MBR だったらディスク丸ごと gmirror ができる。
*回避のため、GPTで複数のパーティションを作成し、それぞれのパーティションで gmirror を動作している。
*複数の gmirror が動作している。
*ここでは、bootcode + freebsd-swap + freebsd-ufs

!!!発生状況

!4/4
""SMART error (FailedOpenDevice) detected on host: ns4
 This message was generated by the smartd daemon running on:
    host name:  ns4
    DNS domain: nona.to
 
 The following warning/error was logged by the smartd daemon:
 Device: /dev/ada0, unable to open ATA device
 
 Device info:
 TOSHIBA DT01ACA300, S/N:144K58KGS, WWN:5-000039-ff4d5e59e, FW:MX6OABB0, 3.00 TB

!5/4
""SMART error (FailedOpenDevice) detected on host: ns4
 This message was generated by the smartd daemon running on:
    host name:  ns4
    DNS domain: nona.to
 
 The following warning/error was logged by the smartd daemon:
 Device: /dev/ada1, unable to open ATA device
 
 Device info:
 TOSHIBA DT01ACA300, S/N:Z7647YTAS, WWN:5-000039-fe6de1e60, FW:MX6OABB0, 3.00 TB

*I/Oエラー
 root@ns4:/home/hoge # time dd if=/dev/ada1 of=/dev/null bs=1048576 skip=1295380
 dd: /dev/ada1: Input/output error
 5+0 records in
 5+0 records out
 5242880 bytes transferred in 14.679095 secs (357166 bytes/sec)
 0.000u 0.001s 0:14.68 0.0%      0+0k 43+0io 0pf+0w
*1.29GB 付近でIOエラー


!過去の対応
*FreeBSD gmirror 障害復旧手順例 201805
*FreeBSD gmirror 障害復旧手順例 201905

!!!作業概要
*gmirror ada0 ada1 ada2 のトリプルミラー
*ada1 が故障

*ada1 を入れ替え
*ada1 のパーティション設定
*ada1 の起動コード書き込み
*トリプルミラー再開

!FreeBSD 正常時状況
*/boot/loader.conf
 geom_mirror_load="YES"
*/etc/fstab
 # Device        Mountpoint      FStype  Options Dump    Pass#
 /dev/mirror/p2  /               ufs     rw      1       1
 /dev/mirror/p3  none            swap    sw      0       0
*gmirror 正常時
 root@ns4:/home/hoge # gmirror status -s
 mirror/p1  COMPLETE  ada0p1 (ACTIVE)
 mirror/p1  COMPLETE  ada2p1 (ACTIVE)
 mirror/p1  COMPLETE  ada1p1 (ACTIVE)
 mirror/p2  COMPLETE  ada0p2 (ACTIVE)
 mirror/p2  COMPLETE  ada2p2 (ACTIVE)
 mirror/p2  COMPLETE  ada1p2 (ACTIVE)
 mirror/p3  COMPLETE  ada0p3 (ACTIVE)
 mirror/p3  COMPLETE  ada2p3 (ACTIVE)
 mirror/p3  COMPLETE  ada1p3 (ACTIVE)

!!!作業記録

!!状況確認
*パーティション情報
 root@ns4:/home/hoge # gpart list
 Geom name: ada0
 modified: false
 state: OK
 fwheads: 16
 fwsectors: 63
 last: 5860533134
 first: 34
 entries: 128
 scheme: GPT
 Providers:
 1. Name: ada0p1
    Mediasize: 65536 (64K)
    Sectorsize: 512
    Stripesize: 4096
    Stripeoffset: 0
    Mode: r1w1e1
    efimedia: HD(1,GPT,eaa06e43-7cb0-11e9-8d65-6805ca0d3e8b,0x28,0x80)
    rawuuid: eaa06e43-7cb0-11e9-8d65-6805ca0d3e8b
    rawtype: 83bd6b9d-7f41-11dc-be0b-001560b84f0f
    label: (null)
    length: 65536
    offset: 20480
    type: freebsd-boot
    index: 1
    end: 167
    start: 40
 2. Name: ada0p2
    Mediasize: 2996813361152 (2.7T)
    Sectorsize: 512
    Stripesize: 4096
    Stripeoffset: 0
    Mode: r1w1e1
    efimedia: HD(2,GPT,35c63824-7cb1-11e9-8d65-6805ca0d3e8b,0xa8,0x15cdfff78)
    rawuuid: 35c63824-7cb1-11e9-8d65-6805ca0d3e8b
    rawtype: 516e7cb6-6ecf-11d6-8ff8-00022d09712b
    label: (null)
    length: 2996813361152
    offset: 86016
    type: freebsd-ufs
    index: 2
    end: 5853151263
    start: 168
 3. Name: ada0p3
    Mediasize: 3779517440 (3.5G)
    Sectorsize: 512
    Stripesize: 4096
    Stripeoffset: 0
    Mode: r1w1e1
    efimedia: HD(3,GPT,46597fa0-7cb1-11e9-8d65-6805ca0d3e8b,0x15ce00020,0x70a36e)
    rawuuid: 46597fa0-7cb1-11e9-8d65-6805ca0d3e8b
    rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
    label: (null)
    length: 3779517440
    offset: 2996813447168
    type: freebsd-swap
    index: 3
    end: 5860533133
    start: 5853151264
 Consumers:
 1. Name: ada0
    Mediasize: 3000592982016 (2.7T)
    Sectorsize: 512
    Stripesize: 4096
    Stripeoffset: 0
    Mode: r3w3e6

 Geom name: ada1
 modified: false
 state: OK
 fwheads: 16
 fwsectors: 63
 last: 5860533134
 first: 34
 entries: 128
 scheme: GPT
 Providers:
 1. Name: ada1p1
    Mediasize: 65536 (64K)
    Sectorsize: 512
    Stripesize: 4096
    Stripeoffset: 0
    Mode: r1w1e1
    efimedia: HD(1,GPT,afd1aa88-7d1d-11e9-8d65-6805ca0d3e8b,0x28,0x80)
    rawuuid: afd1aa88-7d1d-11e9-8d65-6805ca0d3e8b
    rawtype: 83bd6b9d-7f41-11dc-be0b-001560b84f0f
    label: (null)
    length: 65536
    offset: 20480
    type: freebsd-boot
    index: 1
    end: 167
    start: 40
 2. Name: ada1p2
    Mediasize: 2996813361152 (2.7T)
    Sectorsize: 512
    Stripesize: 4096
    Stripeoffset: 0
    Mode: r1w1e1
    efimedia: HD(2,GPT,55594ddf-7d1e-11e9-8d65-6805ca0d3e8b,0xa8,0x15cdfff78)
    rawuuid: 55594ddf-7d1e-11e9-8d65-6805ca0d3e8b
    rawtype: 516e7cb6-6ecf-11d6-8ff8-00022d09712b
    label: (null)
    length: 2996813361152
    offset: 86016
    type: freebsd-ufs
    index: 2
    end: 5853151263
    start: 168
 3. Name: ada1p3
    Mediasize: 3779517440 (3.5G)
    Sectorsize: 512
    Stripesize: 4096
    Stripeoffset: 0
    Mode: r1w1e1
    efimedia: HD(3,GPT,59e22286-7d1e-11e9-8d65-6805ca0d3e8b,0x15ce00020,0x70a36e)
    rawuuid: 59e22286-7d1e-11e9-8d65-6805ca0d3e8b
    rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
    label: (null)
    length: 3779517440
    offset: 2996813447168
    type: freebsd-swap
    index: 3
    end: 5860533133
    start: 5853151264
 Consumers:
 1. Name: ada1
    Mediasize: 3000592982016 (2.7T)
    Sectorsize: 512
    Stripesize: 4096
    Stripeoffset: 0
    Mode: r3w3e6

 Geom name: ada2
 modified: false
 state: OK
 fwheads: 16
 fwsectors: 63
 last: 5860533134
 first: 34
 entries: 128
 scheme: GPT
 Providers:
 1. Name: ada2p1
    Mediasize: 65536 (64K)
    Sectorsize: 512
    Stripesize: 4096
    Stripeoffset: 0
    Mode: r1w1e1
    efimedia: HD(1,GPT,461f6d0d-f3c7-11e3-a841-0022cfe54d22,0x28,0x80)
    rawuuid: 461f6d0d-f3c7-11e3-a841-0022cfe54d22
    rawtype: 83bd6b9d-7f41-11dc-be0b-001560b84f0f
    label: (null)
    length: 65536
    offset: 20480
    type: freebsd-boot
    index: 1
    end: 167
    start: 40
 2. Name: ada2p2
    Mediasize: 2996813361152 (2.7T)
    Sectorsize: 512
    Stripesize: 4096
    Stripeoffset: 0
    Mode: r1w1e1
    efimedia: HD(2,GPT,461fe080-f3c7-11e3-a841-0022cfe54d22,0xa8,0x15cdfff78)
    rawuuid: 461fe080-f3c7-11e3-a841-0022cfe54d22
    rawtype: 516e7cb6-6ecf-11d6-8ff8-00022d09712b
    label: (null)
    length: 2996813361152
    offset: 86016
    type: freebsd-ufs
    index: 2
    end: 5853151263
    start: 168
 3. Name: ada2p3
    Mediasize: 3779517440 (3.5G)
    Sectorsize: 512
    Stripesize: 4096
    Stripeoffset: 0
    Mode: r1w1e1
    efimedia: HD(3,GPT,468c041d-f3c7-11e3-a841-0022cfe54d22,0x15ce00020,0x70a36e)
    rawuuid: 468c041d-f3c7-11e3-a841-0022cfe54d22
    rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
    label: (null)
    length: 3779517440
    offset: 2996813447168
    type: freebsd-swap
    index: 3
    end: 5860533133
    start: 5853151264
 Consumers:
 1. Name: ada2
    Mediasize: 3000592982016 (2.7T)
    Sectorsize: 512
    Stripesize: 4096
    Stripeoffset: 0
    Mode: r3w3e6

*gmirror status 確認
 root@ns4:/home/hoge # gmirror status
      Name    Status  Components
 mirror/p1  COMPLETE  ada0p1 (ACTIVE)
                      ada1p1 (ACTIVE)
                      ada2p1 (ACTIVE)
 mirror/p2  COMPLETE  ada0p2 (ACTIVE)
                      ada1p2 (ACTIVE)
                      ada2p2 (ACTIVE)
 mirror/p3  COMPLETE  ada0p3 (ACTIVE)
                      ada1p3 (ACTIVE)
                      ada2p3 (ACTIVE)

*正常に見えるが、ada1 でIOエラー
 root@ns4:/home/hoge # time dd if=/dev/ada1 of=/dev/null bs=1048576 skip=1295380
 dd: /dev/ada1: Input/output error
 5+0 records in
 5+0 records out
 5242880 bytes transferred in 14.679095 secs (357166 bytes/sec)
 0.000u 0.001s 0:14.68 0.0%      0+0k 43+0io 0pf+0w
*ada1 1.29GB 付近でIOエラー

!GPT パーティション情報の確認

*ada0 ada2 確認
 gpart show ada0
 gpart show ada1
 gpart show ada2

 root@ns4:/home/hoge # gpart show ada0
 =>        34  5860533101  ada0  GPT  (2.7T)
           34           6        - free -  (3.0K)
           40         128     1  freebsd-boot  (64K)
          168  5853151096     2  freebsd-ufs  (2.7T)
   5853151264     7381870     3  freebsd-swap  (3.5G)
   5860533134           1        - free -  (512B)

 root@ns4:/home/hoge # gpart show ada1
 =>        34  5860533101  ada1  GPT  (2.7T)
           34           6        - free -  (3.0K)
           40         128     1  freebsd-boot  (64K)
          168  5853151096     2  freebsd-ufs  (2.7T)
   5853151264     7381870     3  freebsd-swap  (3.5G)
   5860533134           1        - free -  (512B)

 root@ns4:/home/hoge # gpart show ada2
 =>        34  5860533101  ada2  GPT  (2.7T)
           34           6        - free -  (3.0K)
           40         128     1  freebsd-boot  (64K)
          168  5853151096     2  freebsd-ufs  (2.7T)
   5853151264     7381870     3  freebsd-swap  (3.5G)
   5860533134           1        - free -  (512B)

!!ada1 交換

!物理ディスク ada1 除去
*/var/log/messages
 Aug 23 00:36:45 ns4 kernel: ada1 at ahcich1 bus 0 scbus1 target 0 lun 0
 Aug 23 00:36:45 ns4 kernel: ada1: <TOSHIBA MD04ACA300 FP2A> s/n 74S3K1H4FSDA detached
 Aug 23 00:36:45 ns4 kernel: GEOM_MIRROR: Device p3: provider ada1p3 disconnected.GEOM_MIRROR
 Aug 23 00:36:45 ns4 kernel: : Device p1: provider ada1p1 disconnected.
 Aug 23 00:36:45 ns4 kernel: GEOM_MIRROR: Device p2: provider ada1p2 disconnected.
 Aug 23 00:36:45 ns4 kernel: (ada1:ahcich1:0:0:0): Periph destroyed

 root@ns4:/home/hoge # gmirror status
      Name    Status  Components
 mirror/p1  DEGRADED  ada0p1 (ACTIVE)
                      ada2p1 (ACTIVE)
 mirror/p2  DEGRADED  ada0p2 (ACTIVE)
                      ada2p2 (ACTIVE)
 mirror/p3  DEGRADED  ada0p3 (ACTIVE)
                      ada2p3 (ACTIVE)

!物理ディスク ada1 接続
*/var/log/messages
 Aug 23 00:40:24 ns4 kernel: ada1 at ahcich1 bus 0 scbus1 target 0 lun 0
 Aug 23 00:40:24 ns4 kernel: ada1: <TOSHIBA DT01ACA300 MX6OABB0> ATA8-ACS SATA 3.x device
 Aug 23 00:40:24 ns4 kernel: ada1: Serial Number 29O6V7GAS
 Aug 23 00:40:24 ns4 kernel: ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
 Aug 23 00:40:24 ns4 kernel: ada1: Command Queueing enabled
 Aug 23 00:40:24 ns4 kernel: ada1: 2861588MB (5860533168 512 byte sectors)
 Aug 23 00:40:24 ns4 kernel: GEOM: ada1: the secondary GPT header is not in the last LBA.
 Aug 23 00:40:24 ns4 kernel: GEOM: diskid/DISK-29O6V7GAS: the secondary GPT header is not in the last LBA.

!ada1 GPT パーティション作成
*ada1 GPT パーティションを確認
 root@ns4:/home/hoge # gpart show ada1
 =>        34  5860533100  ada1  GPT  (2.7T) [CORRUPT]
           34       65501        - free -  (32M)
        65535  5860467375     1  linux-data  (2.7T)
   5860532910         224        - free -  (112K)
*Linux のパーティションが存在する。
 gpart delete -i 1 ada1

 root@ns4:/home/hoge # gpart delete -i 1 ada1
 gpart: table 'ada1' is corrupt: Operation not permitted

 root@ns4:/home/hoge # gpart recover /dev/ada1
 ada1 recovered

 root@ns4:/home/hoge # gpart show ada1
 =>        40  5860533088  ada1  GPT  (2.7T)
           40       65495        - free -  (32M)
        65535  5860467375     1  linux-data  (2.7T)
   5860532910         218        - free -  (109K)

 root@ns4:/home/hoge # gpart delete -i 1 ada1
 ada1p1 deleted

 root@ns4:/home/hoge # gpart show ada1
 =>        40  5860533088  ada1  GPT  (2.7T)
           40  5860533088        - free -  (2.7T)

*ada0 GPT 管理情報作成
 gpart create -s gpt /dev/ada1

 root@ns4:/home/hoge # gpart create -s gpt /dev/ada1
 gpart: geom 'ada1': File exists
*GPT 管理情報は既に存在している。

*ada1 GPT 初期状況確認
 root@ns4:/home/hoge # gpart show ada1
 =>        40  5860533088  ada1  GPT  (2.7T)
           40  5860533088        - free -  (2.7T)

!ada0 FreeBSD pmbr インストール
:pmbr:Proactive Master Boot Record
 root@ns4:/home/hoge # ls -l /boot/pmbr
 root@ns4:/home/hoge # gpart bootcode -b /boot/pmbr ada1
 bootcode written to ada1
*MBR ツールによる破壊を防止するための疑似MBR情報+GPT起動コード

!ada1 FreeBSD bootcode インストール
*コマンド
 ls -l /boot/gptboot
 gpart add -b 40 -s 128 -t freebsd-boot ada1
 gpart bootcode -p /boot/gptboot -i 1 ada1
*ada0 GPT 情報より freebsd-boot 領域のサイズは 128
*ログ
 root@ns4:/home/hoge # ls -l /boot/gptboot
 -r--r--r--  1 root  wheel  59074 Aug 10  2019 /boot/gptboot
 root@ns4:/home/hoge # gpart show ada1
 =>        40  5860533088  ada1  GPT  (2.7T)
           40  5860533088        - free -  (2.7T)

 root@ns4:/home/hoge # gpart add -b 40 -s 128 -t freebsd-boot ada1
 ada1p1 added
 root@ns4:/home/hoge # gpart show ada1
 =>        40  5860533088  ada1  GPT  (2.7T)
           40         128     1  freebsd-boot  (64K)
          168  5860532960        - free -  (2.7T)

*(参考) GPT パーティションの除去
*コマンド
 gpart delete -i 1 ada1
*ログ
 root@ns4:/home/hoge # gpart show ada1
 =>        34  5860533101  ada1  GPT  (2.7T)
           34           6        - free -  (3.0K)
           40  5860533088     1  freebsd-boot  (2.7T)
   5860533128           7        - free -  (3.5K)
 root@ns4:/home/hoge # gpart delete -i 1 ada1
 ada1p1 deleted
 root@ns4:/home/hoge # gpart show ada1
 =>        34  5860533101  ada1  GPT  (2.7T)
           34  5860533101        - free -  (2.7T)

!ada0 GPT パーティション作成
*コマンド
 gpart add -t freebsd-ufs -s 5853151096 ada1
 gpart add -t freebsd-swap -s   7381870 -a 2 ada1
*ログ
 root@ns4:/home/hoge # gpart add -t freebsd-ufs -s 5853151096 ada1
 ada1p2 added

 root@ns4:/home/hoge # gpart add -t freebsd-swap -s   7381870 -a 2 ada1
 gpart: autofill: No space left on device
 root@ns4:/home/hoge # gpart show ada1
 =>        40  5860533088  ada1  GPT  (2.7T)
           40         128     1  freebsd-boot  (64K)
          168  5853151096     2  freebsd-ufs  (2.7T)
   5853151264     7381864        - free -  (3.5G)

 root@ns4:/home/hoge # gpart add -t freebsd-swap -s 7381864 -a 2 ada1
 ada1p3 added
 root@ns4:/home/hoge # gpart show ada1
 =>        40  5860533088  ada1  GPT  (2.7T)
           40         128     1  freebsd-boot  (64K)
          168  5853151096     2  freebsd-ufs  (2.7T)
   5853151264     7381864     3  freebsd-swap  (3.5G)

*ada2 と*ほぼ*同じパーティション情報を ada1 に作成した。
*freebsd-swap のサイズが小さいので復旧できていない。別途対応

*gpart backup と gpart restore でパーティション情報を復旧するべき(要確認)

!GPT パーティション状況確認
 gpart show ada0
 gpart show ada2
*同じパーティション情報であることを確認した。

!gmirror 再構築
*単純に insert はできない。
 root@ns4:/home/hoge # gmirror insert p1 /dev/ada1p1
 gmirror: Not all disks connected.
!DEGRADED 確認
 root@ns4:/home/hoge # gmirror status
      Name    Status  Components
 mirror/p1  DEGRADED  ada0p1 (ACTIVE)
                      ada2p1 (ACTIVE)
 mirror/p2  DEGRADED  ada0p2 (ACTIVE)
                      ada2p2 (ACTIVE)
 mirror/p3  DEGRADED  ada0p3 (ACTIVE)
                      ada2p3 (ACTIVE)

!p1 DEGRADED 状況を解消する
 root@ns4:/home/hoge # gmirror forget p1
 root@ns4:/home/hoge # gmirror status
      Name    Status  Components
 mirror/p1  COMPLETE  ada0p1 (ACTIVE)
                      ada2p1 (ACTIVE)
 mirror/p2  DEGRADED  ada0p2 (ACTIVE)
                      ada2p2 (ACTIVE)
 mirror/p3  DEGRADED  ada0p3 (ACTIVE)
                      ada2p3 (ACTIVE)

!p1 に ada1p1 を追加
 root@ns4:/home/hoge # gmirror insert p1 /dev/ada1p1
 root@ns4:/home/hoge # gmirror status
      Name    Status  Components
 mirror/p1  COMPLETE  ada0p1 (ACTIVE)
                      ada2p1 (ACTIVE)
                      ada1p1 (ACTIVE)
 mirror/p2  DEGRADED  ada0p2 (ACTIVE)
                      ada2p2 (ACTIVE)
 mirror/p3  DEGRADED  ada0p3 (ACTIVE)
                      ada2p3 (ACTIVE)

!p3 に ada0p3 を追加

 root@ns4:/home/hoge # gmirror forget p3
 root@ns4:/home/hoge # gmirror status
      Name    Status  Components
 mirror/p1  COMPLETE  ada0p1 (ACTIVE)
                      ada2p1 (ACTIVE)
                      ada1p1 (ACTIVE)
 mirror/p2  DEGRADED  ada0p2 (ACTIVE)
                      ada2p2 (ACTIVE)
 mirror/p3  COMPLETE  ada0p3 (ACTIVE)
                      ada2p3 (ACTIVE)
 root@ns4:/home/hoge # gmirror insert p3 /dev/ada1p3
 gmirror: Provider ada1p3 too small.

*ディスクのサイズが小さく、p3 が追加できない。
*元ディスクの p3 のサイズは 7381870
*新ディスクの p3 のサイズは 7381864

*メッセージ確認
 > grep GEOM_MIRROR /var/log/messages
 Aug 23 01:40:39 ns4 kernel: GEOM_MIRROR: Device p1: rebuilding provider ada1p1.
 Aug 23 01:40:39 ns4 kernel: GEOM_MIRROR: Device p1: rebuilding provider ada1p1 finished.

!p2 に ada2p2 を追加

 root@ns4:/home/hoge # gmirror status p2
     Name    Status  Components
 mirror/p2  DEGRADED  ada0p2 (ACTIVE)
                      ada2p2 (ACTIVE)
 root@ns4:/home/hoge # gmirror forget p2
 root@ns4:/home/hoge # gmirror status p2
      Name    Status  Components
 mirror/p2  COMPLETE  ada0p2 (ACTIVE)
                      ada2p2 (ACTIVE)
 root@ns4:/home/hoge # gmirror insert p2 /dev/ada1p2
 root@ns4:/home/hoge # gmirror status p2
      Name    Status  Components
 mirror/p2  DEGRADED  ada0p2 (ACTIVE)
                      ada2p2 (ACTIVE)
                      ada1p2 (SYNCHRONIZING, 0%)

*途中経過
root@ns4:/home/hoge # gmirror status p2
     Name    Status  Components
mirror/p2  DEGRADED  ada0p2 (ACTIVE)
                     ada2p2 (ACTIVE)
                     ada1p2 (SYNCHRONIZING, 85%)

*復旧完了状況確認
 gmirror status

 root@ns4:/home/hoge # gmirror status p2
      Name    Status  Components
 mirror/p2  COMPLETE  ada0p2 (ACTIVE)
                      ada2p2 (ACTIVE)
                      ada1p2 (ACTIVE)
*p2 が DEGRADED から COMPLETE に変わった。

*/var/log/messages
 root@ns4:/home/hoge # grep GEOM_MIRROR /var/log/messages
 Aug 23 01:45:20 ns4 kernel: GEOM_MIRROR: Device p2: rebuilding provider ada1p2.
 Aug 23 11:36:34 ns4 kernel: GEOM_MIRROR: Device p2: rebuilding provider ada1p2 finished.

*2018年5月は9時間で完了
 May 28 18:32:14 ns4 kernel: GEOM_MIRROR: Device p2: rebuilding provider ada1p2.
 May 29 03:29:06 ns4 kernel: GEOM_MIRROR: Device p2: rebuilding provider ada1p2 finished.
*2019年5月は10時間で完了
 May 23 01:57:29 ns4 kernel: GEOM_MIRROR: Device p2: rebuilding provider ada0p2.
 May 23 11:47:46 ns4 kernel: GEOM_MIRROR: Device p2: rebuilding provider ada0p2 finished.
*2020年8月は10時間で完了
 Aug 23 01:45:20 ns4 kernel: GEOM_MIRROR: Device p2: rebuilding provider ada1p2.
 Aug 23 11:36:34 ns4 kernel: GEOM_MIRROR: Device p2: rebuilding provider ada1p2 finished.

!freebsd-swap 対応
 root@ns4:/home/hoge # swapinfo
 Device          1K-blocks     Used    Avail Capacity
 /dev/mirror/p3    3690932   198020  3492912     5%

 root@ns4:/home/hoge # gmirror status p3
      Name    Status  Components
 mirror/p3  COMPLETE  ada0p3 (ACTIVE)
                      ada2p3 (ACTIVE)

*作業前
 root@ns4:/home/nonaka # gpart show ada1
 root@ns4:/home/hoge # gpart show ada1
 =>        40  5860533088  ada1  GPT  (2.7T)
           40         128     1  freebsd-boot  (64K)
          168  5853151096     2  freebsd-ufs  (2.7T)
   5853151264     7381864     3  freebsd-swap  (3.5G)

*freebsd-swap 削除
 root@ns4:/home/nonaka # gpart delete -i 3 ada1
 root@ns4:/home/hoge # gpart delete -i 3 ada1
 ada1p3 deleted

 root@ns4:/home/nonaka # gpart show ada1
 root@ns4:/home/hoge # gpart show ada1
 =>        40  5860533088  ada1  GPT  (2.7T)
           40         128     1  freebsd-boot  (64K)
          168  5853151096     2  freebsd-ufs  (2.7T)
   5853151264     7381864        - free -  (3.5G)

*縮小して再作成
 gpart add -t freebsd-swap -s 7340032 ada1

 root@ns4:/home/nonaka # gpart add -t freebsd-swap -s 7340032 ada1
 root@ns4:/home/hoge # gpart add -t freebsd-swap -s 7340032 ada1
 ada1p3 added

 root@ns4:/home/nonaka # gpart show ada1
 root@ns4:/home/hoge # gpart show ada1
 =>        40  5860533088  ada1  GPT  (2.7T)
           40         128     1  freebsd-boot  (64K)
          168  5853151096     2  freebsd-ufs  (2.7T)
   5853151264     7340032     3  freebsd-swap  (3.5G)
   5860491296       41832        - free -  (20M)

*swap 停止
 root@ns4:/home/nonaka # swapinfo
 root@ns4:/home/hoge # swapinfo
 Device          1K-blocks     Used    Avail Capacity
 /dev/mirror/p3    3690932   198020  3492912     5%

 root@ns4:/home/nonaka # swapoff /dev/mirror/p3
 root@ns4:/home/hoge # swapoff /dev/mirror/p3

 root@ns4:/home/nonaka # swapinfo
 root@ns4:/home/hoge # swapinfo
 Device          1K-blocks     Used    Avail Capacity

*gmirror p3 削除
 gmirror stop p3
 gmirror destroy p3
 gmirror label -v -b round-robin -s 65536 p3 ada1p3 ada0p3 ada2p3

 root@ns4:/home/nonaka # gmirror stop p3
 root@ns4:/home/hoge # gmirror stop p3

 root@ns4:/home/nonaka # gmirror status
 root@ns4:/home/hoge # gmirror status
      Name    Status  Components
 mirror/p1  COMPLETE  ada0p1 (ACTIVE)
                      ada2p1 (ACTIVE)
                      ada1p1 (ACTIVE)
 mirror/p2  COMPLETE  ada0p2 (ACTIVE)
                      ada2p2 (ACTIVE)
                      ada1p2 (ACTIVE)
 mirror/p3  COMPLETE  gptid/468c041d-f3c7-11e3-a841-0022cfe54d22 (ACTIVE)
                      gptid/46597fa0-7cb1-11e9-8d65-6805ca0d3e8b (ACTIVE)

 root@ns4:/home/nonaka # gmirror destroy p3
 root@ns4:/home/hoge # gmirror destroy p3

 root@ns4:/home/nonaka # gmirror status
 root@ns4:/home/hoge # gmirror status
      Name    Status  Components
 mirror/p1  COMPLETE  ada0p1 (ACTIVE)
                      ada2p1 (ACTIVE)
                      ada1p1 (ACTIVE)
 mirror/p2  COMPLETE  ada0p2 (ACTIVE)
                      ada2p2 (ACTIVE)
                      ada1p2 (ACTIVE)

 root@ns4:/home/nonaka # gmirror label -v -b round-robin -s 65536 p3 ada1p3 ada0p3 ada2p3
 root@ns4:/home/hoge # gmirror label -v -b round-robin -s 65536 p3 ada1p3 ada0p3 ada2p3
 Metadata value stored on ada1p3.
 Metadata value stored on ada0p3.
 Metadata value stored on ada2p3.
 Done.
 root@ns4:/home/nonaka # gmirror status
 root@ns4:/home/hoge # gmirror status
      Name    Status  Components
 mirror/p1  COMPLETE  ada0p1 (ACTIVE)
                      ada2p1 (ACTIVE)
                      ada1p1 (ACTIVE)
 mirror/p2  COMPLETE  ada0p2 (ACTIVE)
                      ada2p2 (ACTIVE)
                      ada1p2 (ACTIVE)
 mirror/p3  COMPLETE  ada1p3 (ACTIVE)
                      ada0p3 (ACTIVE)
                      ada2p3 (ACTIVE)

 root@ns4:/home/nonaka # swapinfo
 root@ns4:/home/hoge # swapinfo
 Device          1K-blocks     Used    Avail Capacity
 root@ns4:/home/nonaka # swapon /dev/mirror/p3
 root@ns4:/home/nonaka # swapinfo
 root@ns4:/home/hoge # swapon /dev/mirror/p3
 root@ns4:/home/hoge # swapinfo
 Device          1K-blocks     Used    Avail Capacity
 /dev/mirror/p3    3670012        0  3670012     0%


*gmirror p3 縮小して再作成

!起動確認
*3台のHDDを入れ替えて3台とも起動できることを確認する。
*pmbr は正常か? bootcode は正常か? 確認する。

*shutdown
*HDD1-->HDD2-->HDD3-->HDD1
*起動確認
*shutdown
*HDD1-->HDD2-->HDD3-->HDD1
*起動確認
*shutdown
*HDD1-->HDD2-->HDD3-->HDD1
*起動確認