2011-07-11

semget failed, errno = 17, file ipcmutex.cpp, line 167

Mapping Rules Fail with User Requested Termination / Semget Failed Errors [ID 748700.1]


Applies to:

Oracle Profitability Manager - Version: 11.5.10 to 11.5.10.2
Oracle Enterprise Performance Foundation - Version: 11.5.10 to 11.5.10.2
This problem can occur on any platform.

Symptoms

In Oracle Profitability Manager 11i, all Mapping Rules are failing with User Requested Termination errors.  After you turn on the FEMCCE Debug, "semget failed" errors appear.  The errors are preventing the Mapping Rules from outputting results.  After one Mapping Rule receives the error, all Mapping Rules receive the errors.
Initially, the View Log contains User Requested Termination errors:
*** FCFilterCondition: SqlStr returned: (a.FINANCIAL_ELEM_ID = 280248) AND (a.USER_DIM6_ID = 280334) AND (a.LINE_ITEM_ID = 307183)
AND (((a.DATASET_CODE =11002) and a.CAL_PERIOD_ID =24545260000000000000021100200140)) OR((a.DATASET_CODE =28008) AND (a.CREATED_BY_OBJECT_ID NOT IN 11438)) AND (a.LEDGER_ID = 21003)
[31156] User requested termination
[31159] User requested termination
[31160] User requested termination
[31167] User requested termination
[31158] User requested termination

However, after turning on the FEMCCE debug  (see Note 417446.1), the user requested termination errors are replaced by the following in the View Log:
Module Logging OFS Access module data: begin FEM_Engines_Pkg.Get_PB_Param_Value( P_OBJECT_TYPE_CODE => 
:ObjectTypeCode, P_STEP_NAME => :StepName, P_OBJECT_ID => 
:ObjectId, P_PARAMETER_NAME => :ParamName, X_PB_PARAM_DATA_TYPE 
=> :ParamDataType, X_PB_PARAM_VALUE => :ParamValue, X_MSG_COUNT 
=> :MsgCount, X_MSG_DATA => :MsgData, X_RETURN_STATUS => 
:ReturnStatus) ; end; 

Module Logging OFS errors: semget failed, errno = 17, file 6941520/ fem/ lib/ ipcmutex.cpp, 
line 185 

Module Logging OFS errors: semget failed, errno = 17, file 6941520/ fem/ lib/ ipcmutex.cpp, 
line 185 

Module Logging OFS errors: semget failed, errno = 17, file 6941520/ fem/ lib/ ipcmutex.cpp, 
line 185 

Module Logging OFS errors: semget failed, errno = 17, file 6941520/ fem/ lib/ ipcmutex.cpp, 
line 185
The following errors also appear in the View Log:
Module Logging OFS errors: APP- FEM- 416961: Failed to run CCE.

Module Logging OFS errors: APP- FEM- 416645: Fatal error during processing.

There are no errors in the database alert log.

Cause

There is not enough shared memory / semaphores available on the server.

Run the "ipcs -a" command on the server to get a print out of shared memory segments.

Solution

Review the following MetaLink document:
Note:760633.1 "How to Identify Shared Memory Held by Cancelled Mapping Rules"
There are two options for releasing shared memory:
1. Shutdown and re-start the entire server.  This is the safest way to resolve the issue as shutting down the server will release all memory being held by defunct processes.
or
2. Use the "ipcrm" command to free shared memory.  You can use the instructions in Note 760633.1 to identify shared memory held by Mapping Rule processes.  Once you have identified segments to remove, use a command like "ipcrm –m <shared memory ID>" to release the shared memory manually.  Important: This command will kill any programs or processes using the shared memory segment.
We also recommend you apply the following 1-off patch:
Patch:7308349 "1-OFF: MAPPING RULE SETS FAILED W USER REQUESTED TERMINATION SEMGET ERRNO 17"
This patch adds error handling to the Mapping Rule code for all fatal errors related to the acquisition of semaphores and shared memory.


How to Identify Shared Memory Held by Cancelled Mapping Rules [ID 760633.1]

  Modified 31-JUL-2010     Type HOWTO     Status PUBLISHED  

In this Document
  Goal
  Solution




Applies to:

Oracle Profitability Manager - Version: 11.5.10 to 11.5.10.2 - Release: 11.5 to 11.5
Oracle Enterprise Performance Foundation - Version: 11.5.10 to 11.5.10.2   [Release: 11.5 to 11.5]
Information in this document applies to any platform.
Checked for relevance 31-Jul-2010.

Goal

When running a Mapping Rule or a Rule Set containing Mapping Rules, you receive User Requested Termination errors or semget failed errors or other errors indicating a possible server memory issue. How can you check to see if cancelled Mapping Rules are holding shared memory and decreasing the amount of available memory on the server?

Solution

1. Run the following command to list the Process IDs (pid) of processes holding memory:

ipcs -ms -p
Note: The command output will vary by operating system but should not be significantly different.
Example on Linux:
visfs01:/home/globsupt> ipcs -ms -p

------ Shared Memory Creator/Last-op --------
shmid      owner      cpid       lpid
153976832  visisq02   16415      29123
153190407  visfs01    21029      28476
153223176  visfs01    22037      7023
153255945  visfs01    22048      7023
153288714  visfs01    22113      22169
153321483  visfs01    22048      7023
153354252  visfs01    22169      22169
153387021  visfs01    22169      22169
153419803  visfs01    10062      10150
149061660  glog1      9843       28857
153452575  visfs01    10062      10150
153485344  visfs01    10152      10150

2. Use the values in the "cpid" column for the appropriate owner to check for related concurrent requests.
In the example above, to check concurrent requests associated with visfs01, run the following query:
SELECT r.os_process_id unix_pid, r.request_id, r.request_date, r.last_update_date,
r.phase_code, r.status_code, r.logfile_name, p.concurrent_program_name
FROM   fnd_concurrent_requests r, fnd_concurrent_programs p
WHERE  r.concurrent_program_id  = p.concurrent_program_id
AND    r.program_application_id = p.application_id
AND    r.os_process_id in ('21029', '22037', '22048', '22113', '22169', '10062', '10152');
1 row is returned:
 
UNIX_PID REQUEST_ID REQUEST_DATE LAST_UPDATE_DATE PHASE_CODE STATUS_CODE LOGFILE_NAME CONCURRENT_PROGRAM_NAME
10062 2814862 12-Jan-2009 12-Jan-2009 C X xxxxxxx.req FEM_CCE
By reviewing the information above, you can see that an FEM_CCE request is holding two shared memory segments:
153419803
153452575
You can then use an "ipcs -a" command to review the size of the segments:
Example from Linux:
visfs01:/home/globsupt> ipcs -a

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status
0x298f1430 153976832  visisq02  640        415236096  193
0xc8fe0ecc 153190407  visfs01   640        415236096  81
0x00000000 153223176  visfs01   600        1056768    13         dest
0x00000000 153255945  visfs01   600        1056768    13         dest
0x00000000 153288714  visfs01   600        1056768    9          dest
0x00000000 153321483  visfs01   600        8589316    11         dest
0x00000000 153354252  visfs01   600        1056768    8          dest
0x00000000 153387021  visfs01   600        8589316    7          dest
0x00000000 153419803  visfs01   600        20528      0
0xe70c0c4c 149061660  glog1     640        415236096  99
0x010700b6 153452575  visfs01   600        1024       0
0x040700b6 153485344  visfs01   600        4488000    0

------ Semaphore Arrays --------
key        semid      owner      perms      nsems
0xcf090094 89980928   visisq02  640        254
0xd22ddb00 89489410   visfs01   640        204
0x00000000 89522179   visfs01   600        1
0x00000000 89554951   visfs01   600        1
0x4d532ecc 84213768   glog1     640        204
0x020700b6 89587721   visfs01   600        1
0x050700b6 89620490   visfs01   600        1
0x060700b6 89653259   visfs01   600        1
0x070700b6 89686028   visfs01   600        1
0x00000000 89718797   visfs01   600        1
0x00000000 89751566   visfs01   600        1
0x00000000 89784335   visfs01   600        1

------ Message Queues --------
key        msqid      owner      perms      used-bytes   messages
In the example above, 153419803 is holding 20528 bytes while 153452575 is holding 1024 bytes.
If you can confirm the shared memory is associated with a completed FEM_CCE concurrent request, you can use the "ipcrm" command to remove the shared memory segment.  The ipcrm command should only be run by the Unix administrator or a qualified DBA.  Removing the wrong shared memory segment can kill other applications or databases running on the server.

Niciun comentariu:

Trimiteți un comentariu