升级11.1.0.7后启动数据库报错error 27103

感谢：http://www.dbsupport.cn/thread-372-1-1.html

 

升级到11.1.0.7.12启动数据库报错：

OS版本：suse linux X86-64

DB版本：11.1.0.7.8 升级到11.1.0.7.12

 

Total System Global Area 1.2560E+10 bytes

Fixed Size                  2171344 bytes

Variable Size            7918849584 bytes

Database Buffers         4630511616 bytes

Redo Buffers                8601600 bytes

ORA-03113: end-of-file on communication channel

Process ID: 13056

Session ID: 1105 Serial number: 3

 

alert日志如下：

oracle@PGM_DB1:/oracle/app/diag/rdbms/ora11g/ora11g/trace> tail -100 alert_ora11g.log |more

RECO started with pid=13, OS id=9038 

Tue Dec 10 14:49:52 2013

MMON started with pid=14, OS id=9040 

starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...

Tue Dec 10 14:49:52 2013

MMNL started with pid=15, OS id=9042 

starting up 1 shared server(s) ...

ORACLE_BASE from environment = /oracle/app

Tue Dec 10 14:49:52 2013

ALTER DATABASE   MOUNT

Errors in file /oracle/app/diag/rdbms/ora11g/ora11g/trace/ora11g_mman_9025.trc:

ORA-27103: internal error

Linux-x86_64 Error: 11: Resource temporarily unavailable

Additional information: -1

Additional information: 1

MMAN (ospid: 9025): terminating the instance due to error 27103

Tue Dec 10 14:49:56 2013

ORA-1092 : opitsk aborting process

Tue Dec 10 14:49:56 2013

ORA-1092 : opitsk aborting process

Tue Dec 10 14:49:56 2013

ORA-1092 : opitsk aborting process

Instance terminated by MMAN, pid = 9025

Tue Dec 10 14:50:24 2013

Starting ORACLE instance (normal)

LICENSE_MAX_SESSION = 0

LICENSE_SESSIONS_WARNING = 0

Picked latch-free SCN scheme 3

Autotune of undo retention is turned on. 

IMODE=BR

ILAT =121

LICENSE_MAX_USERS = 0

SYS auditing is disabled

Starting up ORACLE RDBMS Version: 11.1.0.7.0.

Using parameter settings in server-side pfile /oracle/app/product/11g/db/dbs/initora11g.ora

System parameters with non-default values:

  processes                = 1000

  spfile                   = "/dev/vx/rdsk/vgora/lvspfile"

  memory_target            = 12032M

  control_files            = "/dev/vx/rdsk/vgora/lvctl1"

  control_files            = "/dev/vx/rdsk/vgora/lvctl2"

  control_files            = "/dev/vx/rdsk/vgora/lvctl3"

  db_block_size            = 8192

  compatible               = "11.1.0.0.0"

  log_archive_dest_1       = "location=/home/oracle/archive"

  db_recovery_file_dest    = ""

  db_recovery_file_dest_size= 2G

  undo_tablespace          = "UNDOTBS1"

  sec_case_sensitive_logon = FALSE

  remote_login_passwordfile= "EXCLUSIVE"

  db_domain                = ""

  dispatchers              = "(PROTOCOL=TCP) (SERVICE=ora11gXDB)"

  local_listener           = ""

  remote_listener          = ""

  audit_file_dest          = "/oracle/app/admin/ora11g/adump"

  audit_trail              = "NONE"

  db_name                  = "ora11g"

  open_cursors             = 300

  diagnostic_dest          = "/oracle/app"

Tue Dec 10 14:50:26 2013

PMON started with pid=2, OS id=13015 

Tue Dec 10 14:50:26 2013

VKTM started with pid=3, OS id=13017 

VKTM running at (100ms) precision 

Tue Dec 10 14:50:26 2013

DIAG started with pid=4, OS id=13024 

Tue Dec 10 14:50:26 2013

DBRM started with pid=5, OS id=13026 

Tue Dec 10 14:50:26 2013

PSP0 started with pid=6, OS id=13028 

Tue Dec 10 14:50:26 2013

DIA0 started with pid=7, OS id=13030 

Tue Dec 10 14:50:26 2013

MMAN started with pid=8, OS id=13032 

Tue Dec 10 14:50:26 2013

DBW0 started with pid=9, OS id=13034 

Tue Dec 10 14:50:26 2013

LGWR started with pid=10, OS id=13036 

Tue Dec 10 14:50:26 2013

CKPT started with pid=11, OS id=13038 

Tue Dec 10 14:50:26 2013

SMON started with pid=12, OS id=13040 

Tue Dec 10 14:50:26 2013

RECO started with pid=13, OS id=13042 

Tue Dec 10 14:50:26 2013

MMON started with pid=14, OS id=13044 

starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...

Tue Dec 10 14:50:26 2013

MMNL started with pid=15, OS id=13046 

starting up 1 shared server(s) ...

ORACLE_BASE from environment = /oracle/app

Tue Dec 10 14:50:26 2013

ALTER DATABASE   MOUNT

Errors in file /oracle/app/diag/rdbms/ora11g/ora11g/trace/ora11g_mman_13032.trc:

ORA-27103: internal error

Linux-x86_64 Error: 11: Resource temporarily unavailable

Additional information: -1

Additional information: 1

MMAN (ospid: 13032): terminating the instance due to error 27103

Instance terminated by MMAN, pid = 13032

 

trc文件如下：

oracle@PGM_DB1:/oracle/app/diag/rdbms/ora11g/ora11g/trace> more /oracle/app/diag/rdbms/ora11g/ora11g/trace/ora11g_mman_13032.trc

Trace file /oracle/app/diag/rdbms/ora11g/ora11g/trace/ora11g_mman_13032.trc

Oracle Database 11g Enterprise Edition Release 11.1.0.7.0 - 64bit Production

With the Partitioning, Oracle Label Security, OLAP, Data Mining,

Oracle Database Vault and Real Application Testing options

ORACLE_HOME = /oracle/app/product/11g/db

System name:    Linux

Node name:      PGM_DB1

Release:        2.6.32.45-0.3.2.4445.1.PTF-default

Version:        #1 SMP 2011-08-22 10:12:58 +0200

Machine:        x86_64

Instance name: ora11g

Redo thread mounted by this instance: 0 <none>

Oracle process number: 8

Unix process pid: 13032, image: oracle@PGM_DB1 (MMAN)

 

*** 2013-12-10 14:50:29.525

*** SESSION ID:(1098.1) 2013-12-10 14:50:29.525

*** CLIENT ID:() 2013-12-10 14:50:29.525

*** SERVICE NAME:() 2013-12-10 14:50:29.525

*** MODULE NAME:() 2013-12-10 14:50:29.525

*** ACTION NAME:() 2013-12-10 14:50:29.525

 

error 27103 detected in background process

ORA-27103: internal error

Linux-x86_64 Error: 11: Resource temporarily unavailable

Additional information: -1

Additional information: 1

 

*** 2013-12-10 14:50:29.525

MMAN (ospid: 13032): terminating the instance due to error 27103

 

 

结合alert和trace文件查询MOS，发现ORA-27103 when Memory target parameter is set to more than 3 GB [ID 743012.1]描述相符，是由于Bug:7272646引起.

 

打补丁：

 

oracle@PGM_DB1:~/psu> cd 7272646/

oracle@PGM_DB1:~/psu/7272646> 

oracle@PGM_DB1:~/psu/7272646> $ORACLE_HOME/OPatch/opatch apply

Oracle Interim Patch Installer version 11.1.0.9.10

Copyright (c) 2012, Oracle Corporation.  All rights reserved.

 

 

Oracle Home       : /oracle/app/product/11g/db

Central Inventory : /oracle/app/oraInventory

   from           : /oracle/app/product/11g/db/oraInst.loc

OPatch version    : 11.1.0.9.10

OUI version       : 11.1.0.7.0

Log file location : /oracle/app/product/11g/db/cfgtoollogs/opatch/7272646_Dec_10_2013_14_59_02/apply2013-12-10_14-59-01PM_1.log

 

Applying interim patch '7272646' to OH '/oracle/app/product/11g/db'

Verifying environment and performing prerequisite checks...

All checks passed.

Provide your email address to be informed of security issues, install and

initiate Oracle Configuration Manager. Easier for you if you use your My

Oracle Support Email address/User Name.

Visit http://www.oracle.com/support/policies.html for details.

Email address/User Name: 

 

You have not provided an email address for notification of security issues.

Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]:  

Email address/User Name: 

 

You have not provided an email address for notification of security issues.

Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]:  

Email address/User Name: 

 

You have not provided an email address for notification of security issues.

Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]:  y

 

 

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.

(Oracle Home = '/oracle/app/product/11g/db')

 

 

Is the local system ready for patching? [y|n]

y

User Responded with: Y

Backing up files...

 

Patching component oracle.rdbms, 11.1.0.7.0...

 

Verifying the update...

Patch 7272646 successfully applied

Log file location: /oracle/app/product/11g/db/cfgtoollogs/opatch/7272646_Dec_10_2013_14_59_02/apply2013-12-10_14-59-01PM_1.log

 

OPatch succeeded.

 

 

 

重启数据库成功。

 

至此数据库升级完成。但应用侧反馈使用JDBC无法连接数据库。

检查监听状态，和使用tnsname连接数据库后，确认数据库，监听无问题。

 

应用侧的报错如下：

errdesc = libclntsh.so : cannot open shared object file :No such file or directory

 

检查libclntsh.so 是$ORACLE_HOME/lib，和$ORACLE_HOME/lib32里面的一个连接文件

链接到libclntsh.so.11.1。升级前此文件的权限是755，此时的权限是700.修改权限后应用正常。

 

应用的研发说这个和oracle的umask有关。

umaks是用来给新建文件的赋予初始化权限的。

如，本次升级的oracle的umask是0077，用777-077后，新建文件的权限就是700了。