Shared Ethernet Adapter Configuration

Scenario: dual vios setup completed, we want to configure two different Shared Ethernet Adapters to separate management and production traffic.

ioserver:padmin # lsdev -type adapter
name status description
ent0 Available PCIe2 4-Port Adapter (10GbE SFP+) (e4148a1614109304)
ent1 Available PCIe2 4-Port Adapter (10GbE SFP+) (e4148a1614109304)
ent2 Available PCIe2 4-Port Adapter (1GbE RJ45) (e4148a1614109404)
ent3 Available PCIe2 4-Port Adapter (1GbE RJ45) (e4148a1614109404)
ent4 Available PCIe2 4-Port Adapter (10GbE SFP+) (e4148a1614109304)
ent5 Available PCIe2 4-Port Adapter (10GbE SFP+) (e4148a1614109304)
ent6 Available PCIe2 4-Port Adapter (1GbE RJ45) (e4148a1614109404)
ent7 Available PCIe2 4-Port Adapter (1GbE RJ45) (e4148a1614109404)
ent8 Available Virtual I/O Ethernet Adapter (l-lan)
ent9 Available Virtual I/O Ethernet Adapter (l-lan)
ent10 Available Virtual I/O Ethernet Adapter (l-lan)

ioserver:padmin # for x in 0 1 2 3 4 5 6 7; do echo "ENT$x ==============================" ; lsattr -El ent$x | egrep "flow_ctrl|jumbo|large|chksum_offload" ; done
ENT0 ==============================
chksum_offload yes Request checksum offload True
flow_ctrl yes Request flow control True
flow_ctrl_rx yes Receive pause frames True
flow_ctrl_tx yes Transmit pause frames True
jumbo_frames no Request jumbo frames True
jumbo_size 9014 Requested jumbo frame size False
large_receive yes Request Rx TCP segment aggregation True
large_send yes Request Tx TCP segment offload True
ENT1 ==============================
chksum_offload yes Request checksum offload True
flow_ctrl yes Request flow control True
flow_ctrl_rx yes Receive pause frames True
flow_ctrl_tx yes Transmit pause frames True
jumbo_frames no Request jumbo frames True
jumbo_size 9014 Requested jumbo frame size False
large_receive yes Request Rx TCP segment aggregation True
large_send yes Request Tx TCP segment offload True
ENT2 ==============================
chksum_offload yes Request checksum offload True
flow_ctrl yes Request flow control True
flow_ctrl_rx yes Receive pause frames True
flow_ctrl_tx yes Transmit pause frames True
jumbo_frames no Request jumbo frames True
jumbo_size 9014 Requested jumbo frame size False
large_receive yes Request Rx TCP segment aggregation True
large_send yes Request Tx TCP segment offload True
ENT3 ==============================
chksum_offload yes Request checksum offload True
flow_ctrl yes Request flow control True
flow_ctrl_rx yes Receive pause frames True
flow_ctrl_tx yes Transmit pause frames True
jumbo_frames no Request jumbo frames True
jumbo_size 9014 Requested jumbo frame size False
large_receive yes Request Rx TCP segment aggregation True
large_send yes Request Tx TCP segment offload True
ENT4 ==============================
chksum_offload yes Request checksum offload True
flow_ctrl yes Request flow control True
flow_ctrl_rx yes Receive pause frames True
flow_ctrl_tx yes Transmit pause frames True
jumbo_frames no Request jumbo frames True
jumbo_size 9014 Requested jumbo frame size False
large_receive yes Request Rx TCP segment aggregation True
large_send yes Request Tx TCP segment offload True
ENT5 ==============================
chksum_offload yes Request checksum offload True
flow_ctrl yes Request flow control True
flow_ctrl_rx yes Receive pause frames True
flow_ctrl_tx yes Transmit pause frames True
jumbo_frames no Request jumbo frames True
jumbo_size 9014 Requested jumbo frame size False
large_receive yes Request Rx TCP segment aggregation True
large_send yes Request Tx TCP segment offload True
ENT6 ==============================
chksum_offload yes Request checksum offload True
flow_ctrl yes Request flow control True
flow_ctrl_rx yes Receive pause frames True
flow_ctrl_tx yes Transmit pause frames True
jumbo_frames no Request jumbo frames True
jumbo_size 9014 Requested jumbo frame size False
large_receive yes Request Rx TCP segment aggregation True
large_send yes Request Tx TCP segment offload True
ENT7 ==============================
chksum_offload yes Request checksum offload True
flow_ctrl yes Request flow control True
flow_ctrl_rx yes Receive pause frames True
flow_ctrl_tx yes Transmit pause frames True
jumbo_frames no Request jumbo frames True
jumbo_size 9014 Requested jumbo frame size False
large_receive yes Request Rx TCP segment aggregation True
large_send yes Request Tx TCP segment offload True

ioserver:padmin # for x in 0 1 4 5; do chdev -l ent$x -a jumbo_frames=yes; done
ent0 changed
ent1 changed
ent4 changed
ent5 changed

Sea su mgmt

Now it is possible to define the shared ethernet adapter using the physical (ent2) adapter and virtual (ent8 ent9) to bridge data and sea control channel.

En10 will be used just to assign an ip address to the vios server.

mkvdev -sea ent2 -vadapter ent8 -default ent8 -defaultid 606 -attr ha_mode=auto ctl_chan=ent9
chdev -dev ent11 -attr health_time=60 link_time=10
chdev -dev ent11 -attr large_receive=yes

mktcpip -hostname vios1 -inetaddr 192.168.61.77 -interface en10 -netmask 255.255.255.0 -gateway 10.15.61.1
mktcpip -hostname vios2 -inetaddr 192.168.61.78 -interface en10 -netmask 255.255.255.0 -gateway 10.15.61.1

Configurazione LAN VIOS per Jumbo Frames

1) VERIFICA parametri schede fisiche 

Verificare che flow_ctrl, jumbo_frames, large_receive e large_send siano impostati a “yes” sulle schede fisiche che compongono l’etherchannel 

lsattr -El entX | egrep “flow_ctrl|jumbo|large” 

2) CONFIGURAZIONE JF su Etherchannel 

Impostare use_jumbo_frame=yes sul device EC , dopo averlo messo offline

rmdev -l entX   # where entX represents the EC device if one exists

chdev -l entX -a use_jumbo_frame=

mkdev -l entX   # where entX represents the EC device

Verifica: 

lsattr -El entX | egrep “mode|jumbo” 

3) IMPOSTAZIONE parametri virtual Ethernet 

Impostare i seguenti parametri sull’interfaccia con IP address: 

$ chdev -dev enX -attr mtu_bypass=on rfc1323=1 mtu=9000 

Impostare i seguenti parametri sulle virtual Ethernet (trunk) che compongono la SEA (non sono necessari sul control channel): 

max_buf_huge=128 

max_buf_large=256 

max_buf_medium=2048 

max_buf_small=4096 

max_buf_tiny=4096 

min_buf_huge=127 

min_buf_large=255 

min_buf_medium=2047 

min_buf_small=4095 

min_buf_tiny=4095 

chdev -dev entX -perm -attr max_buf_huge=128 max_buf_large=256 max_buf_medium=2048 max_buf_small=4096 max_buf_tiny=4096 min_buf_huge=127 min_buf_large=255 min_buf_medium=2047 min_buf_small=4095 min_buf_tiny=4095 

5) CONFIGURAZIONE SEA 

Configurare la SEA, abilitando largesend, large_receive e jumbo_frames. 

Nell’esempio, avendo un solo trunk, è stata impostata la modalità auto (ha_mode=auto), nel caso di SEA con più trunk impostare il load sharing (ha_mode=sharing): 

mkvdev -sea entXX -vadapter entYY -default entYY -defaultid 1 -attr ha_mode=auto ctl_chan=entZZ largesend=1 jumbo_frames=yes large_receive=yes adapter_reset=no thread=0 

chdev -l entX -a ha_mode=standby

rmdev -l entX

chdev -l entX -a jumbo_frames=yes largesend=1 jumbo_frames=yes large_receive=yes

mkdev -l entX

chdev -l entX -a ha_mode=auto

6) IMPOSTAZIONE parametri di rete sulle virtual Ethernet delle LPAR client 

Impostare i seguenti parametri sulla scheda di rete: 

max_buf_huge=128 

max_buf_large=256 

max_buf_medium=2048 

max_buf_small=4096 

max_buf_tiny=4096 

min_buf_huge=127 

min_buf_large=255 

min_buf_medium=2047 

min_buf_small=4095 

min_buf_tiny=4095 

chdev -l entX -a max_buf_huge=128 -a max_buf_large=256 -a max_buf_medium=2048 -a max_buf_small=4096 -a max_buf_tiny=4096 -a min_buf_huge=127 -a min_buf_large=255 -a min_buf_medium=2047 -a min_buf_small=4095 -a min_buf_tiny=4095 -P 

chdev -l enX  -a mtu_bypass=on -a tcp_nodelay=1 -a rfc1323=1 -a mtu=9000

Reboot 

SEA Failover definition in dual vios setup

Scenario , ambiente dual vios su cui vogliamo costruire una shared ethernet adapter con failover e vlan multiple.

Su entrambi I vios definire due virtual adapters su cui verranno configurate le vlan necessarie , in questo caso la 1 per la rete di produzione e la 606 per la management . E’ necessario attivare il 802.1q per poter assegnare vlan multiple all’adapter. Sul primo vios della coppia assegnamo anche trunk priority uguale a 1 .

 

 

 

 

Sul secondo vio replichiamo la stessa configurazione con però trunk priority uguale a 2.

 

 

 

 

A questo punto configuriamo I due virtual adapter che costituiranno il control channel per il failover della SEA . In questo caso utilizziamo un vlanid non usato , ad es . 999 , senza settare 802.1q e il

bridging.

 

 

 

 

 

Definiamo infine la SEA :

 

mkvdev -sea ent2 -vadapter ent3 -default ent3 -defaultid 1 -attr ha_mode=auto ctl_chan=ent4

ent5 Available

en5

et5

 

dove ent2 è la fisica , nel nostro caso un etherchannel costruito su due fisiche;

ent3 è il vadapter che conosce le vlan di prod e mgmt ;

defaultid = il vlanid di default ; ctl_chan = il vadapter sulla vlan 999 ;

 

Configuriamo l’ip address sulla sea oppure aggiungiamo un ulteriore vadapter per rimanere indipendenti dalla SEA e non perdere connettività in caso di manutenzione.

 

mkvdev -vlan ent5 -tagid 606

ent6 Available

en6

et6

 

e assegnamo l’ip address al nuovo adapter

 

mktcpip -hostname viosXX1 -inetaddr XX.XX.XX.XX -interface en5 -netmask 255.255.255.0 -gateway XX.XX.XX.XX mktcpip -hostname viosXX2 -inetaddr XX.XX.XX.XX -interface en5 -netmask 255.255.255.0 -gateway XX.XX.XX.XX

 

 

 

VIOS – Aggiunta dischi

Scenario: 2 sistemi p770 con 4 vios ognuno (2 dedicati scsi e 2 dedicati net)

SU TUTTI I 4 VIOS DEDICATI PER RISORSE SCSI :

oem_setup_env + cfgmgr
oppure
cfgdev (da restricted shell)

verificare che il vio veda le nuove lun
lsdev | grep -i hdisk

I seguenti comandi modificano alcuni parametri dei dischi , se necessari.
Nel mio caso i dischi sono su storage Hitachi.

da shell restricted
chdev -l hdiskXXX -attr pv=yes
chdev -dev hdiskXXX -attr reserve_policy=no_reserve
chdev -dev hdiskXXX -attr queue_depth=2

SOLO SUI VIOS RELATIVI AL SISTEMA OWNER DELLA LPAR
Ora rimane da verificare quale vhost corrisponda alla lpar a cui dobbiamo rendere disponibili i dischi.
In HMC vedo il numero di controller Cxx
in vios shell
lsdev -slots | grep Cxx

ottengo così il vadapter , nel seguente esempio creo i vdev per 3 volumi fisici
che diventeranno parte del rootvg e del pagingvg di una lpar in fase di installazione.

mkvdev -vdev hdisk300 -vadapter vhost11 -dev soa02_rootvg
mkvdev -vdev hdisk301 -vadapter vhost11 -dev soa02paging1
mkvdev -vdev hdisk302 -vadapter vhost11 -dev soa02paging2