Environemnt
FSM 1.2.X or higher
Problem Description
Command “smstart” failure within FSM. This kind of error most of times can happen after a hardware problem, power outage, etc.
Error
USERID@fsm:~> smstart
DB2SET processing complete, rc = 4, SQLCODE = 4 SQL6048N A communication error occurred during START or STOP DATABASE MANAGER processing.
Starting IBM Director…The starting process may take a while. Please use smstatus to check if the server is active.
Action Taken
First off, you will need the ‘pesh’ password to solve this problem, so you will need to call IBM Support and ask how to obtain this password.
IBM will request the output of the “lsconfig” command.
# lsconfig -v
vpd=*FC ????????
*N2 Tue Jul 30 09:58:18 BRT 2013
*DS FSM
*TM -[795501M]-
*SE 1024CEB
*MN IBM
*OS Embedded Operating Systems
*NA Unknown
*RM 1.2.1
*UUID 682890CB-28AF-499E-A9E7-1341E0E93A73
*UVMID 657f:782b:a541:f9fe
With the pesh password provided by IBM Support you have to login with ‘pe’ user through SSH client (like Putty) and run these commands to gain root access over FSM;
pe@fsm1:~> pesh <UVMID>
Password: <- imput pesh password
pe@fsm1:~> su - root
Password: <- imput password of ‘pe’ user
fsm1:~ #
Verify if there are only the monitor daemon of DB2 (db2fmcd) running on FSM;
# ps -ef |grep -i DB2
root 11997 1 0 Jul26 ? 00:00:14 /opt/ibm/director/db2/bin/db2fmcd
root 22637 22029 0 10:01 pts/1 00:00:00 grep -i DB2
Check the output of db2_all command.
# db2_all “echo OK”
ssh: Could not resolve hostname PURESYSTEMFSM: Name or service not known
This output indicates that FSM has a problem to resolve the DB2 instance hostname. The hostname can be found in the db2nodes.cfg and the same hostname will be missing from /etc/hosts file.
# cat /home/db2inst1/sqllib/db2nodes.cfg
0 PURESYSTEMFSM 0
# ping PURESYSTEMFSM
ping: unknown host PURESYSTEMFSM
# cat /etc/hosts
127.0.0.1 localhost.localdomain localhost
::1 localhost ipv6-localhost ipv6-loopback
fe00::0 ipv6-localnet
ff00::0 ipv6-mcastprefix
ff02::1 ipv6-allnodes
ff02::2 ipv6-allrouters
ff02::3 ipv6-allhosts
After confirming that we have a resolution problem and adding 'PURESYSTEMFSM' hostname to /etc/hosts, the db2stop/db2start should work just fine.
# cat /etc/hosts
127.0.0.1 localhost.localdomain localhost
<IP ADDRESS> PURESYSTEMFSM
::1 localhost ipv6-localhost ipv6-loopback
fe00::0 ipv6-localnet
ff00::0 ipv6-mcastprefix
ff02::1 ipv6-allnodes
ff02::2 ipv6-allrouters
ff02::3 ipv6-allhosts
# db2stop
07/30/2013 10:14:53 0 0 SQL1032N No start database manager command was issued.
SQL1032N No start database manager command was issued. SQLSTATE=57019
# db2start
07/30/2013 10:15:09 0 0 SQL1063N DB2START processing was successful.
SQL1063N DB2START processing was successful.
# ps -ef |grep -i DB2
root 11301 22029 0 10:14 pts/1 00:00:00 su - db2inst1
db2inst1 11303 11301 0 10:14 pts/1 00:00:00 -bash
root 11997 1 0 Jul26 ? 00:00:14 /opt/ibm/director/db2/bin/db2fmcd
root 13087 1 0 10:15 pts/1 00:00:00 db2wdog 0
db2inst1 13089 13087 0 10:15 pts/1 00:00:00 db2sysc 0
root 13090 13089 0 10:15 pts/1 00:00:00 db2ckpwd 0
root 13091 13089 0 10:15 pts/1 00:00:00 db2ckpwd 0
root 13092 13089 0 10:15 pts/1 00:00:00 db2ckpwd 0
db2inst1 13096 13089 0 10:15 pts/1 00:00:00 db2vend (PD Vendor Process - 1)
db2inst1 13120 13087 0 10:15 pts/1 00:00:00 db2acd 0 ,0,0,0,1,0,0,0000,1,0,8a6740,14,1e014,2,0,1,11fc0,0x210000000,0x210000000,1600000,7800a,2,15001b
db2inst1 14693 11303 0 10:16 pts/1 00:00:00 ps -ef
db2inst1 14694 11303 0 10:16 pts/1 00:00:00 grep db2
# tail -n 10 /home/db2inst1/sqllib/db2dump/db2diag.log
2013-07-30-10.15.09.094936-180 E58894787E306 LEVEL: Event
PID : 13085 TID : 140737352849152PROC : db2star2
INSTANCE: db2inst1 NODE : 000
FUNCTION: DB2 UDB, base sys utilities, DB2StartMain, probe:911
MESSAGE : ADM7513W Database manager has started.
START : DB2 DBM
# smstart
Starting IBM Director…The starting process may take a while. Please use smstatus to check if the server is active.
# smstatus
Inactive
# smstatus
Starting
# smstatus
Active
[Link Reference]