UNIX Essentials: March 2008

Thursday, March 27, 2008 at 2:58 PM | 0 comments

Today I got a call from one of my peers regarding a problem with missing LUNs.

The SAN team presented certain number of LUNs to his 3 AIX servers but all 3 had the same exact issue. All are missing 7 LUNs.

I helped out because I thought that this should be an easy one. First he told me he ran these 3 commands:

# cfgmgr
# vxdisk scandisks
# vxdctl enable

That should do it. He ran the right commands. But didn't work. So, I had to get in to the servers and look around. First, I saw some old disks from the old SHARK array which was already been disconnected. I cleared those up and re-ran those 3 commands. Still did not work.

Then, I counted all the hdisks on each HBA and compared with the hdisks listed in "vxdisk path" command. They matched. So, I thought that didn't make sense. So, I carefully reviewed the vxdisk path output and noticed that some of the devices have 4 paths and I counted 7 of those. There, I found out what the problem was. 4 paths to a LUN with only 2 HBAs active on the server. I have seen this on a Clariion but not on Symmetrix where these servers are connected to.

So, these are the steps I took to fix the problem:

1. Removed the hdisks on those 7 devices
2. Ran vxdisk scandisks ; vxdctl enable
3. Ran cfgmgr -vl fcs1 ; cfgmgr -vl fcs4
4. Ran vxdisk scandisks ; vxdctl enable

Now, how that happened, I don't know. Could be a bug in Veritas 4.0.

Here are the detailed steps:

# vxdmpadm listctlr all
CTLR-NAME       ENCLR-TYPE      STATE      ENCLR-NAME
=====================================================
scsi0           Disk            ENABLED      Disk
fscsi4          EMC             ENABLED      EMC0
fscsi1          EMC             ENABLED      EMC0
fscsi4          EMC             ENABLED      EMC1
fscsi1          EMC             ENABLED      EMC1

# lscfg | grep hdisk | grep EMC | cut -d"W" -f2 | cut -d"-" -f1 | sort -u
50060482CCB501C6
50060482CCB501C9
50060482CCAB5543
50060482CCAB554C

# lscfg | grep 50060482CCB501C6 | wc -l
      48

# lscfg | grep 50060482CCB501C9 | wc -l
      48

# lscfg | grep 50060482CCAB5543 | wc -l
      58

# lscfg | grep 50060482CCAB554C | wc -l
      58

# vxdisk list | grep EMC0 | wc -l
      48

# vxdisk list | grep EMC1 | wc -l
      51

# vxdisk list EMC1_19 | grep hdisk
hdisk97         state=enabled
hdisk107        state=enabled
hdisk211        state=enabled
hdisk182        state=enabled

# lscfg | egrep 'hdisk97|hdisk107|hdisk211|hdisk182'
* hdisk107         U5791.001.99205GR-P2-C10-T1-W50060482CCAB5543-L92000000000000  EMC Symmetrix FCP Raid5
* hdisk211         U5791.001.99205GR-P2-C10-T1-W50060482CCAB5543-LAD000000000000  EMC Symmetrix FCP Raid5
* hdisk97          U5791.001.99205LM-P2-C03-T1-W50060482CCAB554C-L92000000000000  EMC Symmetrix FCP Raid5
* hdisk182         U5791.001.99205LM-P2-C03-T1-W50060482CCAB554C-LAD000000000000  EMC Symmetrix FCP Raid5

# for i in 19 20 21 22 23 24 25
do
  vxdisk list EMC1_$i | grep hdisk | awk '{ print $1 }'
done | while read disk
do
  rmdev -dl $disk
done

# vxdisk scandisks ; vxdctl enable

# cfgmgr -vl fcs1; cfgmgr -vl fcs4

# vxdisk scandisks ; vxdctl enable

# vxdisk list | grep EMC1 | wc -l
      58

# vxdisk list EMC1_19 | grep hdisk
hdisk107        state=enabled
hdisk211        state=enabled

Posted by JAUGHN Labels: A Daily Journal

AIX Index

at 9:52 AM | 0 comments

Here’s a collection of my AIX documents. Some were from my own work and some were taken from the web.

These documents were posted mainly for my personal references and for others that may find them useful.

Operating System

Switching Between 32bit and 64bit mode

Tuning

Resource Limits

Redbooks

Fixes, Tips and Tricks

Some Very Helpful Sites

Man Pages

Posted by JAUGHN Labels: AIX

AIX - Switching Between 32bit and 64bit

at 9:22 AM | 0 comments

Before switching to 64bit mode, make sure the hardware supports it.

To verify what is running and what the hardware can support, run the following as root:

# echo "Hardware:\t`bootinfo -y` bits capable"
Hardware:       64 bits capable

# echo "Running:\t`bootinfo -K` bits mode"
Running:        32 bits mode

# ls -l /unix /usr/lib/boot/unix
lrwxrwxrwx 1 root  system  21 Aug 15 2006  /unix -> /usr/lib/boot/unix_mp
lrwxrwxrwx 1 root  system  21 Aug 15 2006  /usr/lib/boot/unix -> /usr/lib/boot/unix_mp

Switching From 32 to 64 Bit Mode

To switch from 32-bit mode to 64-bit mode run the following commands, in the given order:

1. ln -sf /usr/lib/boot/unix_64 /unix
2. ln -sf /usr/lib/boot/unix_64 /usr/lib/boot/unix
3. bosboot -ad /dev/ipldevice
4. shutdown -Fr

Switching From 64 To 32-Bit Mode

To switch from 64-bit mode to 32-bit mode run the following commands, in the given order:

1. ln -sf /usr/lib/boot/unix_mp /unix
2. ln -sf /usr/lib/boot/unix_mp /usr/lib/boot/unix
3. bosboot -ad /dev/ipldevice
4. shutdown -Fr

Posted by JAUGHN Labels: AIX

AIX - Resource Limits

at 8:33 AM | 1 comments

I just got a request from the development team to set certain security settings for resource limits.

Here's another good document I found from the IBM website.

Resource limits on UNIX systems (ulimit)

Resource limits is the concept where you regulate several resources consumed by a process on an UNIX operating systems. Although the resource limits are set on a per user basis, they are applied per process basis. Therefore, if a user is executing hundreds of processes, the user may consume huge amount of resources, even if the resource setting values for the user are relatively small numbers.

On UNIX systems, the ulimit command controls the limits on system resource, such as process data size, process virtual memory, and process file size. Specifically:

On Solaris systems, by default, the root user has unlimited access to these resources (for example, unlimited).
On AIX, some limits might apply to the root user.

On UNIX systems, each user can either inherit resource limits from the root user or have specific limits defined. When setting resource limits for a process, it is important to know that the limits that apply are those that are in effect for the parent process and not the limits for the user under which the process runs. For example, the IBM Directory server runs under the ldap user account that was created at install time. However, the IBM Directory server is typically started while logged in as the root user. Starting while logged in as the root user means that any limits that are in effect for the ldap user have no effect on the IBM Directory server process unless the IBM Directory server process is started while logged in as the ldap user.

To display the current user’s resource limits, use the ulimit command (see the following example):

# ulimit -Ha
time(seconds) unlimited
file(blocks) 2097151
data(kbytes) unlimited
stack(kbytes) unlimited
memory(kbytes) unlimited
coredump(blocks) unlimited
nofiles(descriptors) unlimited

# ulimit -Sa
time(seconds) unlimited
file(blocks) 2097151
data(kbytes) 131072
stack(kbytes) 32768
memory(kbytes) 32768
coredump(blocks) 2097151
nofiles(descriptors) 2000

The -H option instructs the command to display hard resource limits, while the -S option instructs the command to display soft resource limits. The hard resource limit values are set by the root user using the chuser command for each user. The soft resource limit values can be relaxed by the individual user using the ulimit command, as long as the values are smaller than the hard resource limit values.

Increasing process memory size limit

Enter the following command to check the current process data size and virtual memory size limits:

ulimit -d
ulimit -m

It is recommended that the process data size and virtual memory size be set to unlimited. Setting to unlimited can be done by modifying the following lines in the /etc/security/limits file:

default:
data = -1
rss = -1

For changes to the /etc/security/limits file to take effect, the user must log out of the current login session and log back in.

At minimum, set these size limits to 256 MB, which is the value of 256000 in the /etc/security/limits file. Increase these limits when a larger-than-default IBM Directory server cache is to be used. For more information, see the IBM Directory Server documentation.

In addition to the /etc/security/limits file, the process virtual memory size is limited by the number of segments that a process can use. By default, a process can only use one memory segment, which limits a process to 128 MB. AIX support a large memory model that is enabled through the LDR_CNTRL environment variable.

Increase file size limit

Enter the following command to check the current file size limits:

ulimit -f

It is recommended that the file size limit be set to unlimited. Setting to unlimited can be done by modifying the following lines in the /etc/security/limits file:

default:
fsize = -1

For changes to the /etc/security/limits file to take effect, the user must log out of the current login session and log back in.

Create file systems with large file support

The standard file system on AIX has a 2 GB file size limit, regardless of the ulimit setting. One way to enable files larger than the 2 GB limit is to create the file system with the Large File Enabled option. This option can be found through the Add a Journaled File System option of the smit menu. Refer to AIX documentation for additional information and file system options.

Posted by JAUGHN Labels: AIX, AIX Tuning

Removing a Disk from a DiskGroup

Monday, March 24, 2008 at 9:32 PM | 0 comments

Details:

Disks that are no longer required and not being used can be removed from its diskgroup.

The command to remove a disk from a diskgroup is:

vxdg -g <dg> rmdisk <disk>

The example below is from an actual server with EMC LUNs. I normally name the disks that includes the last few digits of the Storage WWN plus the actual LUN number.

# vxdisk list
DEVICE       TYPE            DISK            GROUP        STATUS
EMC0_0       auto:cdsdisk    dvgy404B23L002  dvgy404      online
EMC0_1       auto:cdsdisk    dvgy404B23L009  dvgy404      online
EMC0_2       auto:cdsdisk    dvgy404B23L057  dvgy404      online
EMC0_3       auto:cdsdisk    dvgy404B23L035  dvgy404      online
EMC0_4       auto:cdsdisk    dvgy404B23L063  dvgy404      online
EMC0_5       auto:cdsdisk    dvgy404B23L056  dvgy404      online
EMC0_6       auto:cdsdisk    dvgy404B23L027  dvgy404      online
EMC0_7       auto:cdsdisk    dvgy404B23L030  dvgy404      online
EMC0_8       auto:cdsdisk    dvgy404B23L031  dvgy404      online

# vxdg -g dvgy404 rmdisk dvgy404B23L031

# vxdisk list
DEVICE       TYPE            DISK            GROUP        STATUS
EMC0_0       auto:cdsdisk    dvgy404B23L002  dvgy404      online
EMC0_1       auto:cdsdisk    dvgy404B23L009  dvgy404      online
EMC0_2       auto:cdsdisk    dvgy404B23L057  dvgy404      online
EMC0_3       auto:cdsdisk    dvgy404B23L035  dvgy404      online
EMC0_4       auto:cdsdisk    dvgy404B23L063  dvgy404      online
EMC0_5       auto:cdsdisk    dvgy404B23L056  dvgy404      online
EMC0_6       auto:cdsdisk    dvgy404B23L027  dvgy404      online
EMC0_7       auto:cdsdisk    dvgy404B23L030  dvgy404      online
EMC0_8       auto:cdsdisk    -               -            online

The disk has been removed from the diskgroup.

Posted by JAUGHN

UNIX Essentials

About This Site

Categories

Blog Archive

		UNIX Essentials Jaughn's official UNIX library on the web.
		VxVM - VxDMP Confused Thursday, March 27, 2008 at 2:58 PM \| 0 comments Today I got a call from one of my peers regarding a problem with missing LUNs. The SAN team presented certain number of LUNs to his 3 AIX servers but all 3 had the same exact issue. All are missing 7 LUNs. I helped out because I thought that this should be an easy one. First he told me he ran these 3 commands: `#` `cfgmgr` `#` `vxdisk scandisks` `#` `vxdctl enable` That should do it. He ran the right commands. But didn't work. So, I had to get in to the servers and look around. First, I saw some old disks from the old SHARK array which was already been disconnected. I cleared those up and re-ran those 3 commands. Still did not work. Then, I counted all the hdisks on each HBA and compared with the hdisks listed in "vxdisk path" command. They matched. So, I thought that didn't make sense. So, I carefully reviewed the vxdisk path output and noticed that some of the devices have 4 paths and I counted 7 of those. There, I found out what the problem was. 4 paths to a LUN with only 2 HBAs active on the server. I have seen this on a Clariion but not on Symmetrix where these servers are connected to. So, these are the steps I took to fix the problem: 1. Removed the hdisks on those 7 devices 2. Ran `vxdisk scandisks ; vxdctl enable` 3. Ran `cfgmgr -vl fcs1 ; cfgmgr -vl fcs4` 4. Ran `vxdisk scandisks ; vxdctl enable` Now, how that happened, I don't know. Could be a bug in Veritas 4.0. Here are the detailed steps: `#` `vxdmpadm listctlr all` CTLR-NAME ENCLR-TYPE STATE ENCLR-NAME ===================================================== scsi0 Disk ENABLED Disk fscsi4 EMC ENABLED EMC0 fscsi1 EMC ENABLED EMC0 fscsi4 EMC ENABLED EMC1 fscsi1 EMC ENABLED EMC1 `#` `lscfg \| grep hdisk \| grep EMC \| cut -d"W" -f2 \| cut -d"-" -f1 \| sort -u` 50060482CCB501C6 50060482CCB501C9 50060482CCAB5543 50060482CCAB554C `#` `lscfg \| grep 50060482CCB501C6 \| wc -l` 48 `#` `lscfg \| grep 50060482CCB501C9 \| wc -l` 48 `#` `lscfg \| grep 50060482CCAB5543 \| wc -l` 58 `#` `lscfg \| grep 50060482CCAB554C \| wc -l` 58 `#` `vxdisk list \| grep EMC0 \| wc -l` 48 `#` `vxdisk list \| grep EMC1 \| wc -l` 51 `#` `vxdisk list EMC1_19 \| grep hdisk` hdisk97 state=enabled hdisk107 state=enabled hdisk211 state=enabled hdisk182 state=enabled `#` `lscfg \| egrep 'hdisk97\|hdisk107\|hdisk211\|hdisk182'` * hdisk107 U5791.001.99205GR-P2-C10-T1-W50060482CCAB5543-L92000000000000 EMC Symmetrix FCP Raid5 * hdisk211 U5791.001.99205GR-P2-C10-T1-W50060482CCAB5543-LAD000000000000 EMC Symmetrix FCP Raid5 * hdisk97 U5791.001.99205LM-P2-C03-T1-W50060482CCAB554C-L92000000000000 EMC Symmetrix FCP Raid5 * hdisk182 U5791.001.99205LM-P2-C03-T1-W50060482CCAB554C-LAD000000000000 EMC Symmetrix FCP Raid5 `#` `for i in 19 20 21 22 23 24 25` do vxdisk list EMC1_$i \| grep hdisk \| awk '{ print $1 }' done \| while read disk do rmdev -dl $disk done `#` `vxdisk scandisks ; vxdctl enable` `#` `cfgmgr -vl fcs1; cfgmgr -vl fcs4` `#` `vxdisk scandisks ; vxdctl enable` `#` `vxdisk list \| grep EMC1 \| wc -l` 58 `#` `vxdisk list EMC1_19 \| grep hdisk` hdisk107 state=enabled hdisk211 state=enabled Posted by JAUGHN Labels: A Daily Journal AIX Index at 9:52 AM \| 0 comments Here’s a collection of my AIX documents. Some were from my own work and some were taken from the web. These documents were posted mainly for my personal references and for others that may find them useful. Operating System Switching Between 32bit and 64bit mode Tuning Resource Limits Redbooks AIX LVM from A to Z: Introduction and Concepts AIX LVM from A to Z: Troubleshooting and Commands AIX LVM and Veritas Volume Manager Fixes, Tips and Tricks Some Very Helpful Sites Man Pages Posted by JAUGHN Labels: AIX AIX - Switching Between 32bit and 64bit at 9:22 AM \| 0 comments Before switching to 64bit mode, make sure the hardware supports it. To verify what is running and what the hardware can support, run the following as root: `#` echo "Hardware:\t`bootinfo -y` bits capable" Hardware: 64 bits capable `#` echo "Running:\t`bootinfo -K` bits mode" Running: 32 bits mode `#` `ls -l /unix /usr/lib/boot/unix` lrwxrwxrwx 1 root system 21 Aug 15 2006 /unix -> /usr/lib/boot/unix_mp lrwxrwxrwx 1 root system 21 Aug 15 2006 /usr/lib/boot/unix -> /usr/lib/boot/unix_mp Switching From 32 to 64 Bit Mode To switch from 32-bit mode to 64-bit mode run the following commands, in the given order: `1. ln -sf /usr/lib/boot/unix_64 /unix 2. ln -sf /usr/lib/boot/unix_64 /usr/lib/boot/unix 3. bosboot -ad /dev/ipldevice 4. shutdown -Fr` Switching From 64 To 32-Bit Mode To switch from 64-bit mode to 32-bit mode run the following commands, in the given order: `1. ln -sf /usr/lib/boot/unix_mp /unix 2. ln -sf /usr/lib/boot/unix_mp /usr/lib/boot/unix 3. bosboot -ad /dev/ipldevice 4. shutdown -Fr` Posted by JAUGHN Labels: AIX AIX - Resource Limits at 8:33 AM \| 1 comments I just got a request from the development team to set certain security settings for resource limits. Here's another good document I found from the IBM website. Resource limits on UNIX systems (ulimit) Resource limits is the concept where you regulate several resources consumed by a process on an UNIX operating systems. Although the resource limits are set on a per user basis, they are applied per process basis. Therefore, if a user is executing hundreds of processes, the user may consume huge amount of resources, even if the resource setting values for the user are relatively small numbers. On UNIX systems, the ulimit command controls the limits on system resource, such as process data size, process virtual memory, and process file size. Specifically: On Solaris systems, by default, the `root` user has unlimited access to these resources (for example, unlimited). On AIX, some limits might apply to the `root` user. On UNIX systems, each user can either inherit resource limits from the `root` user or have specific limits defined. When setting resource limits for a process, it is important to know that the limits that apply are those that are in effect for the parent process and not the limits for the user under which the process runs. For example, the IBM Directory server runs under the `ldap` user account that was created at install time. However, the IBM Directory server is typically started while logged in as the `root` user. Starting while logged in as the `root` user means that any limits that are in effect for the `ldap` user have no effect on the IBM Directory server process unless the IBM Directory server process is started while logged in as the `ldap` user. To display the current user’s resource limits, use the ulimit command (see the following example): `#` `ulimit -Ha` time(seconds) unlimited file(blocks) 2097151 data(kbytes) unlimited stack(kbytes) unlimited memory(kbytes) unlimited coredump(blocks) unlimited nofiles(descriptors) unlimited `#` `ulimit -Sa` time(seconds) unlimited file(blocks) 2097151 data(kbytes) 131072 stack(kbytes) 32768 memory(kbytes) 32768 coredump(blocks) 2097151 nofiles(descriptors) 2000 The -H option instructs the command to display hard resource limits, while the -S option instructs the command to display soft resource limits. The hard resource limit values are set by the root user using the chuser command for each user. The soft resource limit values can be relaxed by the individual user using the ulimit command, as long as the values are smaller than the hard resource limit values. Increasing process memory size limit Enter the following command to check the current process data size and virtual memory size limits: `ulimit -d ulimit -m` It is recommended that the process data size and virtual memory size be set to unlimited. Setting to unlimited can be done by modifying the following lines in the `/etc/security/limits` file: `default: data = -1 rss = -1` For changes to the `/etc/security/limits` file to take effect, the user must log out of the current login session and log back in. At minimum, set these size limits to 256 MB, which is the value of 256000 in the `/etc/security/limits` file. Increase these limits when a larger-than-default IBM Directory server cache is to be used. For more information, see the IBM Directory Server documentation. In addition to the `/etc/security/limits` file, the process virtual memory size is limited by the number of segments that a process can use. By default, a process can only use one memory segment, which limits a process to 128 MB. AIX support a large memory model that is enabled through the LDR_CNTRL environment variable. Increase file size limit Enter the following command to check the current file size limits: `ulimit -f` It is recommended that the file size limit be set to unlimited. Setting to unlimited can be done by modifying the following lines in the `/etc/security/limits` file: `default: fsize = -1` For changes to the `/etc/security/limits` file to take effect, the user must log out of the current login session and log back in. Create file systems with large file support The standard file system on AIX has a 2 GB file size limit, regardless of the ulimit setting. One way to enable files larger than the 2 GB limit is to create the file system with the Large File Enabled option. This option can be found through the Add a Journaled File System option of the smit menu. Refer to AIX documentation for additional information and file system options. Posted by JAUGHN Labels: AIX, AIX Tuning Removing a Disk from a DiskGroup Monday, March 24, 2008 at 9:32 PM \| 0 comments Details: Disks that are no longer required and not being used can be removed from its diskgroup. The command to remove a disk from a diskgroup is: `vxdg -g <dg> rmdisk <disk>` The example below is from an actual server with EMC LUNs. I normally name the disks that includes the last few digits of the Storage WWN plus the actual LUN number. `#` `vxdisk list` DEVICE TYPE DISK GROUP STATUS EMC0_0 auto:cdsdisk dvgy404B23L002 dvgy404 online EMC0_1 auto:cdsdisk dvgy404B23L009 dvgy404 online EMC0_2 auto:cdsdisk dvgy404B23L057 dvgy404 online EMC0_3 auto:cdsdisk dvgy404B23L035 dvgy404 online EMC0_4 auto:cdsdisk dvgy404B23L063 dvgy404 online EMC0_5 auto:cdsdisk dvgy404B23L056 dvgy404 online EMC0_6 auto:cdsdisk dvgy404B23L027 dvgy404 online EMC0_7 auto:cdsdisk dvgy404B23L030 dvgy404 online EMC0_8 auto:cdsdisk dvgy404B23L031 dvgy404 online `#` `vxdg -g dvgy404 rmdisk dvgy404B23L031` `#` `vxdisk list` DEVICE TYPE DISK GROUP STATUS EMC0_0 auto:cdsdisk dvgy404B23L002 dvgy404 online EMC0_1 auto:cdsdisk dvgy404B23L009 dvgy404 online EMC0_2 auto:cdsdisk dvgy404B23L057 dvgy404 online EMC0_3 auto:cdsdisk dvgy404B23L035 dvgy404 online EMC0_4 auto:cdsdisk dvgy404B23L063 dvgy404 online EMC0_5 auto:cdsdisk dvgy404B23L056 dvgy404 online EMC0_6 auto:cdsdisk dvgy404B23L027 dvgy404 online EMC0_7 auto:cdsdisk dvgy404B23L030 dvgy404 online EMC0_8 auto:cdsdisk - - online The disk has been removed from the diskgroup. Posted by JAUGHN Newer Posts Older Posts Home	About This Site UNIX World is constantly changing. It is almost impossible to remember every changes, every commands, every functions, every parameters, etc. I have bookmarked any links that I found important and useful but the locations keep changing. That is why I decided to create this site so I can post my works as well as my references. Categories A Daily Journal (1) AIX (3) AIX Tuning (1) HBA (3) SAN (2) Solaris - SAN (1) Veritas Cluster Filesystem (1) Veritas Cluster Server (3) Veritas Volume Manager (10) Veritas Volume Replicator (3) VxVM - DMP (2) VxVM - Problem Solving (4) VxVM - Volumes (6) Blog Archive ► 2012 (1) ► December (1) ► 2009 (1) ► March (1) ▼ 2008 (19) ► May (1) ► April (2) ▼ March (5) VxVM - VxDMP Confused AIX Index AIX - Switching Between 32bit and 64bit AIX - Resource Limits Removing a Disk from a DiskGroup ► January (11) ► 2007 (8) ► December (6) ► November (2) MARVEL and SPIDER-MAN: TM & 2007 Marvel Characters, Inc. Motion Picture © 2007 Columbia Pictures Industries, Inc. All Rights Reserved. 2007 Sony Pictures Digital Inc. All rights reserved. blogger template by blog forum