Thursday, March 27, 2008 at 2:58 PM | 0 comments
Today I got a call from one of my peers regarding a problem with missing LUNs.

The SAN team presented certain number of LUNs to his 3 AIX servers but all 3 had the same exact issue. All are missing 7 LUNs.

I helped out because I thought that this should be an easy one. First he told me he ran these 3 commands:
# cfgmgr
# vxdisk scandisks
# vxdctl enable


That should do it. He ran the right commands. But didn't work. So, I had to get in to the servers and look around. First, I saw some old disks from the old SHARK array which was already been disconnected. I cleared those up and re-ran those 3 commands. Still did not work.

Then, I counted all the hdisks on each HBA and compared with the hdisks listed in "vxdisk path" command. They matched. So, I thought that didn't make sense. So, I carefully reviewed the vxdisk path output and noticed that some of the devices have 4 paths and I counted 7 of those. There, I found out what the problem was. 4 paths to a LUN with only 2 HBAs active on the server. I have seen this on a Clariion but not on Symmetrix where these servers are connected to.

So, these are the steps I took to fix the problem:

1. Removed the hdisks on those 7 devices
2. Ran vxdisk scandisks ; vxdctl enable
3. Ran cfgmgr -vl fcs1 ; cfgmgr -vl fcs4
4. Ran vxdisk scandisks ; vxdctl enable

Now, how that happened, I don't know. Could be a bug in Veritas 4.0.

Here are the detailed steps:

# vxdmpadm listctlr all
CTLR-NAME ENCLR-TYPE STATE ENCLR-NAME
=====================================================
scsi0 Disk ENABLED Disk
fscsi4 EMC ENABLED EMC0
fscsi1 EMC ENABLED EMC0
fscsi4 EMC ENABLED EMC1
fscsi1 EMC ENABLED EMC1

# lscfg | grep hdisk | grep EMC | cut -d"W" -f2 | cut -d"-" -f1 | sort -u
50060482CCB501C6
50060482CCB501C9
50060482CCAB5543
50060482CCAB554C

# lscfg | grep 50060482CCB501C6 | wc -l
48

# lscfg | grep 50060482CCB501C9 | wc -l
48

# lscfg | grep 50060482CCAB5543 | wc -l
58

# lscfg | grep 50060482CCAB554C | wc -l
58

# vxdisk list | grep EMC0 | wc -l
48

# vxdisk list | grep EMC1 | wc -l
51

# vxdisk list EMC1_19 | grep hdisk
hdisk97 state=enabled
hdisk107 state=enabled
hdisk211 state=enabled
hdisk182 state=enabled

# lscfg | egrep 'hdisk97|hdisk107|hdisk211|hdisk182'
* hdisk107 U5791.001.99205GR-P2-C10-T1-W50060482CCAB5543-L92000000000000 EMC Symmetrix FCP Raid5
* hdisk211 U5791.001.99205GR-P2-C10-T1-W50060482CCAB5543-LAD000000000000 EMC Symmetrix FCP Raid5
* hdisk97 U5791.001.99205LM-P2-C03-T1-W50060482CCAB554C-L92000000000000 EMC Symmetrix FCP Raid5
* hdisk182 U5791.001.99205LM-P2-C03-T1-W50060482CCAB554C-LAD000000000000 EMC Symmetrix FCP Raid5

# for i in 19 20 21 22 23 24 25
do
vxdisk list EMC1_$i | grep hdisk | awk '{ print $1 }'
done | while read disk
do
rmdev -dl $disk
done

# vxdisk scandisks ; vxdctl enable

# cfgmgr -vl fcs1; cfgmgr -vl fcs4

# vxdisk scandisks ; vxdctl enable

# vxdisk list | grep EMC1 | wc -l
58

# vxdisk list EMC1_19 | grep hdisk
hdisk107 state=enabled
hdisk211 state=enabled
Posted by JAUGHN Labels:
Here’s a collection of my AIX documents. Some were from my own work and some were taken from the web.

These documents were posted mainly for my personal references and for others that may find them useful.


Operating System

Tuning

Redbooks

Fixes, Tips and Tricks




Some Very Helpful Sites




Man Pages






Posted by JAUGHN Labels:
Before switching to 64bit mode, make sure the hardware supports it.

To verify what is running and what the hardware can support, run the following as root:
# echo "Hardware:\t`bootinfo -y` bits capable"
Hardware: 64 bits capable

# echo "Running:\t`bootinfo -K` bits mode"
Running: 32 bits mode

# ls -l /unix /usr/lib/boot/unix
lrwxrwxrwx 1 root system 21 Aug 15 2006 /unix -> /usr/lib/boot/unix_mp
lrwxrwxrwx 1 root system 21 Aug 15 2006 /usr/lib/boot/unix -> /usr/lib/boot/unix_mp


Switching From 32 to 64 Bit Mode

To switch from 32-bit mode to 64-bit mode run the following commands, in the given order:
1. ln -sf /usr/lib/boot/unix_64 /unix
2. ln -sf /usr/lib/boot/unix_64 /usr/lib/boot/unix
3. bosboot -ad /dev/ipldevice
4. shutdown -Fr


Switching From 64 To 32-Bit Mode

To switch from 64-bit mode to 32-bit mode run the following commands, in the given order:
1. ln -sf /usr/lib/boot/unix_mp /unix
2. ln -sf /usr/lib/boot/unix_mp /usr/lib/boot/unix
3. bosboot -ad /dev/ipldevice
4. shutdown -Fr
Posted by JAUGHN Labels:
I just got a request from the development team to set certain security settings for resource limits.

Here's another good document I found from the IBM website.


Resource limits on UNIX systems (ulimit)

Resource limits is the concept where you regulate several resources consumed by a process on an UNIX operating systems. Although the resource limits are set on a per user basis, they are applied per process basis. Therefore, if a user is executing hundreds of processes, the user may consume huge amount of resources, even if the resource setting values for the user are relatively small numbers.

On UNIX systems, the ulimit command controls the limits on system resource, such as process data size, process virtual memory, and process file size. Specifically:

  • On Solaris systems, by default, the root user has unlimited access to these resources (for example, unlimited).
  • On AIX, some limits might apply to the root user.

On UNIX systems, each user can either inherit resource limits from the root user or have specific limits defined. When setting resource limits for a process, it is important to know that the limits that apply are those that are in effect for the parent process and not the limits for the user under which the process runs. For example, the IBM Directory server runs under the ldap user account that was created at install time. However, the IBM Directory server is typically started while logged in as the root user. Starting while logged in as the root user means that any limits that are in effect for the ldap user have no effect on the IBM Directory server process unless the IBM Directory server process is started while logged in as the ldap user.


To display the current user’s resource limits, use the ulimit command (see the following example):
# ulimit -Ha
time(seconds) unlimited
file(blocks) 2097151
data(kbytes) unlimited
stack(kbytes) unlimited
memory(kbytes) unlimited
coredump(blocks) unlimited
nofiles(descriptors) unlimited

# ulimit -Sa
time(seconds) unlimited
file(blocks) 2097151
data(kbytes) 131072
stack(kbytes) 32768
memory(kbytes) 32768
coredump(blocks) 2097151
nofiles(descriptors) 2000

The -H option instructs the command to display hard resource limits, while the -S option instructs the command to display soft resource limits. The hard resource limit values are set by the root user using the chuser command for each user. The soft resource limit values can be relaxed by the individual user using the ulimit command, as long as the values are smaller than the hard resource limit values.


Increasing process memory size limit

Enter the following command to check the current process data size and virtual memory size limits:
ulimit -d
ulimit -m

It is recommended that the process data size and virtual memory size be set to unlimited. Setting to unlimited can be done by modifying the following lines in the /etc/security/limits file:
default:
data = -1
rss = -1

For changes to the /etc/security/limits file to take effect, the user must log out of the current login session and log back in.

At minimum, set these size limits to 256 MB, which is the value of 256000 in the /etc/security/limits file. Increase these limits when a larger-than-default IBM Directory server cache is to be used. For more information, see the IBM Directory Server documentation.

In addition to the /etc/security/limits file, the process virtual memory size is limited by the number of segments that a process can use. By default, a process can only use one memory segment, which limits a process to 128 MB. AIX support a large memory model that is enabled through the LDR_CNTRL environment variable.


Increase file size limit

Enter the following command to check the current file size limits:
ulimit -f

It is recommended that the file size limit be set to unlimited. Setting to unlimited can be done by modifying the following lines in the /etc/security/limits file:
default:
fsize = -1

For changes to the /etc/security/limits file to take effect, the user must log out of the current login session and log back in.


Create file systems with large file support

The standard file system on AIX has a 2 GB file size limit, regardless of the ulimit setting. One way to enable files larger than the 2 GB limit is to create the file system with the Large File Enabled option. This option can be found through the Add a Journaled File System option of the smit menu. Refer to AIX documentation for additional information and file system options.
Posted by JAUGHN Labels: ,
Details:

Disks that are no longer required and not being used can be removed from its diskgroup.

The command to remove a disk from a diskgroup is:

vxdg -g <dg> rmdisk <disk>


The example below is from an actual server with EMC LUNs. I normally name the disks that includes the last few digits of the Storage WWN plus the actual LUN number.

# vxdisk list
DEVICE TYPE DISK GROUP STATUS
EMC0_0 auto:cdsdisk dvgy404B23L002 dvgy404 online
EMC0_1 auto:cdsdisk dvgy404B23L009 dvgy404 online
EMC0_2 auto:cdsdisk dvgy404B23L057 dvgy404 online
EMC0_3 auto:cdsdisk dvgy404B23L035 dvgy404 online
EMC0_4 auto:cdsdisk dvgy404B23L063 dvgy404 online
EMC0_5 auto:cdsdisk dvgy404B23L056 dvgy404 online
EMC0_6 auto:cdsdisk dvgy404B23L027 dvgy404 online
EMC0_7 auto:cdsdisk dvgy404B23L030 dvgy404 online
EMC0_8 auto:cdsdisk dvgy404B23L031 dvgy404 online

# vxdg -g dvgy404 rmdisk dvgy404B23L031

# vxdisk list
DEVICE TYPE DISK GROUP STATUS
EMC0_0 auto:cdsdisk dvgy404B23L002 dvgy404 online
EMC0_1 auto:cdsdisk dvgy404B23L009 dvgy404 online
EMC0_2 auto:cdsdisk dvgy404B23L057 dvgy404 online
EMC0_3 auto:cdsdisk dvgy404B23L035 dvgy404 online
EMC0_4 auto:cdsdisk dvgy404B23L063 dvgy404 online
EMC0_5 auto:cdsdisk dvgy404B23L056 dvgy404 online
EMC0_6 auto:cdsdisk dvgy404B23L027 dvgy404 online
EMC0_7 auto:cdsdisk dvgy404B23L030 dvgy404 online
EMC0_8 auto:cdsdisk - - online


The disk has been removed from the diskgroup.
Posted by JAUGHN
Visit the Site
MARVEL and SPIDER-MAN: TM & 2007 Marvel Characters, Inc. Motion Picture © 2007 Columbia Pictures Industries, Inc. All Rights Reserved. 2007 Sony Pictures Digital Inc. All rights reserved. blogger template by blog forum