Wednesday, 14 November 2012

IMPDP and lacking internal import structures

There are number of problems with importing data with impdp depending on the content parameter setting during export. According to few articles on Metalink this is mostly due to the fact, that during import there are created implicitly some structures and this creation failed. Today I just hit something like this. I exported some schemas with content=data_only. When importing I hit:
ORA-39034: Table TABLE_DATA:"TEST"."SCHEDULER$_JOB_ARG" does not exist.
ORA-39126: Worker unexpected fatal error in KUPW$WORKER.UPATE_TD_ROW_IMP [62] 
TABLE_DATA:"TEST"."SCHEDULER$_JOB_ARG"
ORA-31603: object "SCHEDULER$_JOB_ARG" of type TABLE not found in schema "TEST"

ORA-06512: at "SYS.DBMS_SYS_ERROR", line 105
ORA-06512: at "SYS.KUPW$WORKER", line 8364

----- PL/SQL Call Stack -----
  object      line  object
  handle    number  name
0xb7f251c0     19208  package body SYS.KUPW$WORKER
0xb7f251c0      8385  package body SYS.KUPW$WORKER
0xb7f251c0     18770  package body SYS.KUPW$WORKER
0xb7f251c0      4226  package body SYS.KUPW$WORKER
0xb7f251c0      9082  package body SYS.KUPW$WORKER
0xb7f251c0      1688  package body SYS.KUPW$WORKER
0xb791dd40         2  anonymous block
To workaround this I assumed if I create this lacking table I'll be able to go further. I run as sys
create table "TEST"."SCHEDULER$_JOB_ARG" as select * from SCHEDULER$_JOB_ARGUMENT where rownum<1 pre="">
and run import again.
One step forward.
I hit then:
ORA-39034: Table TABLE_DATA:"TEST"."SCHEDULER$_PROGRAM_ARG" does not exist.
ORA-39126: Worker unexpected fatal error in KUPW$WORKER.UPATE_TD_ROW_IMP [62] 
TABLE_DATA:"TEST"."SCHEDULER$_PROGRAM_ARG"
ORA-31603: object "SCHEDULER$_PROGRAM_ARG" of type TABLE not found in schema "TEST"

ORA-06512: at "SYS.DBMS_SYS_ERROR", line 105
ORA-06512: at "SYS.KUPW$WORKER", line 8364

----- PL/SQL Call Stack -----
  object      line  object
  handle    number  name
0xb7f251c0     19208  package body SYS.KUPW$WORKER
0xb7f251c0      8385  package body SYS.KUPW$WORKER
0xb7f251c0     18770  package body SYS.KUPW$WORKER
0xb7f251c0      4226  package body SYS.KUPW$WORKER
0xb7f251c0      9082  package body SYS.KUPW$WORKER
0xb7f251c0      1688  package body SYS.KUPW$WORKER
0xa9d3b2b8         2  anonymous block
Per analogiam I run:
create table "TEST"."SCHEDULER$_PROGRAM_ARG" as select * from SCHEDULER$_PROGRAM_ARGUMENT where rownum<1 pre="">
Next step forward.
Now I hit
ORA-39034: Table TABLE_DATA:"TEST2"."RE$ACTION_IMP_TAB" does not exist.
ORA-39126: Worker unexpected fatal error in KUPW$WORKER.UPATE_TD_ROW_IMP [62] 
TABLE_DATA:"TEST2"."RE$ACTION_IMP_TAB"
ORA-31603: object "RE$ACTION_IMP_TAB" of type TABLE not found in schema "TEST2"

ORA-06512: at "SYS.DBMS_SYS_ERROR", line 105
ORA-06512: at "SYS.KUPW$WORKER", line 8364

----- PL/SQL Call Stack -----
  object      line  object
  handle    number  name
0xb7f251c0     19208  package body SYS.KUPW$WORKER
0xb7f251c0      8385  package body SYS.KUPW$WORKER
0xb7f251c0     18770  package body SYS.KUPW$WORKER
0xb7f251c0      4226  package body SYS.KUPW$WORKER
0xb7f251c0      9082  package body SYS.KUPW$WORKER
0xb7f251c0      1688  package body SYS.KUPW$WORKER
0xb2aae570         2  anonymous block
Now I have no idea where I may find RE$ACTION_IMP_TAB structure, so I wished myself the structure is not important and it is enough to get the properly named object, so I simply created whatever.
create table "TEST2"."RE$ACTION_IMP_TAB" as select * from dual where rownum<1;
The load goes forward - I assume it should be ok. If I import jobs or chains, it would fail, but jobs or chains are a part of metadata, so problem solved.

Of course the best way is to dump with content=ALL (ie. default value), but sometimes one forgets or has no such choice - then the workaround would be useful.

And short update - the manually created objects have to be dropped manually as well.

Monday, 29 October 2012

Zabbix on Oracle - few notes

We migrated Zabbix from MySQL to Oracle.
At the same time we moved to partitioning, here excellent article how to do it on MySQL. On Oracle we partitioned the same tables, created the same indexes, but go with interval partitioning (it is nice to not have to create new partitions manually or program it) based upon weeks (the partitioning key is of NUMBER type and we need an equal interval value for every period, so month is out due to variable number of days).

Almost no problems with upgrade to 2.0.2. We had to change the database however due to national character set - must be UTF-8, while we had previously fixed UTF16 - zabbix on Oracle is based mainly upon NVARCHAR2 type, and upgrade sets some columns to width bigger than 2000.
It is worth to enable CURSOR_SHARING to FORCE (actually I would say this is inevitable).
At the same time I would not recommend to use any automatic memory management (even ASMM). May be this is only our case, but frequently we run into ORA-4031 problems, which eventually ended with instance hangover or crash. As soon as I disabled any automatic memory management completely, the problems gone.

In addition to partition maintenance here there is an action for job, which will drop the HISTORY% tables' old partitions:
 begin
  for cmd in (
    with 
    conf as (select 9+1 part_num_left from dual)
    select 'alter table '||p.table_name||' drop partition '||p.partition_name||' update global indexes'  ddl1 
    from 
      user_tab_partitions p, 
      (select tp.table_name, count(*) l_pnum 
        from user_tab_partitions tp 
        where tp.table_name like 'HISTORY%' 
        group by tp.table_name 
        having count(*)>(select part_num_left from conf)) t
    where 1=1
      and p.table_name=t.table_name 
      and p.table_name like 'HISTORY%' 
      and p.partition_name<>'DONT_DELETE' 
      and p.partition_position between 2 and (t.l_pnum-(select part_num_left from conf)) 
    order by p.table_name, p.partition_position)
  loop
    execute immediate cmd.ddl1;
  end loop;
end;

The partitions labeled DONT_DELETE are partitions on the first position of relevant tables. The label is due to the fact one can not drop the first partition from a table partitioned by interval. It is comfortable to create those partitions before the time we want to store in table in order to keep it empty. The code above drops partitions leaving last n ones (here 9 + DONT_DELETE). The line with condition on DONT_DELETE partition name is actually redundant as the code starts dropping from second partition.

Tuesday, 25 September 2012

Moving subpartitions

In Oracle database MOVE operations are done on the segment level actually. Thus while simple table has its own segment, the partitioned one has none all the segments are bound with partitions. In case of subpartitions the same rule apply. So we move:
ALTER TABLE simple_t MOVE TABLESPACE tblspace1;
ALTER TABLE partitioned_t MOVE PARTITION p1  TABLESPACE tblspace1;
ALTER TABLE subpartitioned_t MOVE SUBPARTITION s1 TABLESPACE tblspace1;
When we move all the segments with must however change the default tablespace value on the table level and possibly on the partition level if it has a different value.
ALTER TABLE partitioned_t MODIFY DEFAULT ATTRIBUTES TABLESPACE tblspace1;
ALTER TABLE subpartitioned_t MODIFY DEFAULT ATTRIBUTES FOR PARTITION p1 TABLESPACE tblspace1;
ALTER TABLE t1 MODIFY DEFAULT ATTRIBUTES... is less known, but can be quite useful allowing for switching on and off Advanced Compression and some other segment parameters.

This applies at least to the version 11.2 (do not check other docs).

Friday, 17 August 2012

Problems with connection

From time to time there happens different problems with connections to Oracle database. Below little report about those I have met.

  • problems on client

    Usually they are trivial and bound with tnsnames.ora. Wrong PATH variable (for example few ORACLE_HOME/bin entries), wrong LD_LIBRARY_PATH, entries in the tnsnames.ora with some unusual blank character, wrong syntax in this file, wrong service_name. For example today 2 different tnsnames.ora files, while part of programs uses the right one, and some the wrong one.
    While trivial, sometimes hard to diagnose, especially by phone.

    Fortunately it is easy to get very detailed trace. We need in sqlnet.ora few entries:
    TRACE_LEVEL_CLIENT=16 #or SUPPORT
    TRACE_FILE_CLIENT=mytrace #useful for identification
    TRACE_DIRECTORY_CLIENT=traces #useful for location
    
    The output is quite detailed and usually helpful in the diagnosis.
  • problems on cman

    Of course if one uses it (we do). I have met 3 major problems with it (and this is relevant for 11g version).

    The first one was with the raise of 11g version. The cman needs one rule to connect to itself. Till the 10g it was for example:
    (RULE=(SRC=10.0.50.10)(DST=127.0.0.1)(SRV=cmon)(ACT=accept))
    
    With 11g it must be (at least on Linux):
    (RULE=(SRC=10.0.50.10)(DST=::1)(SRV=cmon)(ACT=accept))
    
    The Metalink (ID 298916.1) still provides address 127.0.0.1 as a right one, and indeed the cman starts, but one can not connect to it with cmctl. Possibly it works, but one can not control it. I always needed to kill the cman process with kill utility to change anything (and no, we did not use it on production before the problem was solved).

    The second problem was with some older RDBMS versions (ie. before 10g) - we needed to add to simple rule some settings, for example:
    (RULE=(SRC=10.0.50.12)(DST=10.0.60.11)(SRV=*)(ACT=accept)(ACTION_LIST=(mct=0)))
    
    Usually it is enough to end with ACT attribute. This particular case with RDBMS servers on Windows kind os.

    The third one was with error in syntax. Once we loose somewhere in the middle of cman.ora a parenthesis. The cman kept all the rules from the time before error emergence, thus in the rules there were rules located before the point of error as well as those after it. When we added rules before that point and reloaded, everything was fine, if we added after, the rules were ignored. Because we always reloaded, the cman worked and for some time we were unaware of the problem at all.
    The true incident we met, when were trying to restart the cman. It went down and we can not start it up. IIRC the returned error was about problem in the configuration file in the end of it (which of course was due to mess in parentheses). The solution here was to display the config with VIM and color the syntax. Eventually we found the place for lacking parenthesis. The problem was trivial, but not so obvious.

    To those three I add yet another - in our case it was lack of mapping IP->hostname in /etc/hosts, possibly similar situation if no proper name in DNS service available (if used instead of /etc/hosts mapping). In result the cman service returned ORA-12529, even though the cman.ora configuration does not use host names at all (all rules with IP).
    The same happens when the entry in /etc/hosts differs from DST hostname.
  • problems on server

    Both problems on server had something to do with firewalls.
    Once it was MTU parameter, which default value appeared too large after some firewall software update.
    The other problem was with inactivity timeout. Some of our firewalls disconnect connections, on which there is no traffic during specified time. A solution here seems to be enabling dead connection detection (DCD) feature. This feature implementation is based upon sending for time to time some bytes as a heartbeat - thus there is some traffic on a connection, so it is not broken.

Wednesday, 8 August 2012

Automation while suspending the db work

From some time the Oracle RDBMS supports a situation when due to some work lacks the space within db tablespaces. The AFTER SUSPEND trigger is called, so there is a place for kind of callback, one may set the RESUMABLE_TIMEOUT parameter or there is the DBA_RESUMABLE table with entries inserted after supend events.

If I would imagine itself the best way to solve the suspend event it would be to run within the AFTER SUSPEND trigger a code which would add a new file to the tablespace, on which there is a lack of place. There are was even examples with such solution for 9i version, but it is not now at least with 11.2 version (not sure about 10g). So what one may do?

I meet this problem mostly on test/dev environments, so the solution do not need to be a complex one and I do not need to think about any special cases - all added files are of UNLIMITED size, so always the solution is to add a new file (unless the storage is fully used).
  1. I have met with a solution based upon the AFTER SUSPEND trigger. In 11.2 we get ORA-30511 if trying to run some DDL (and adding a file is of such type). The practical usage now I see in the following - setting RESUMABLE_TIMEOUT to some higher value in order to allow a DBA to make a room for the operation in suspend state or/and inform such DBA by SMS or an alert generation
  2. use DBMS_SCHEDULER job to run from time to time a piece of code, which would add a new file
In my case the second solution has much more sense and is a real automation. Because we mostly use Data Pump utilities to load test/dev databases, it usually enables 2h long suspend (or so) on the operation. In such case the best way is to create a procedure like this:
CREATE OR REPLACE
PROCEDURE expand_resumable_tblspace
AS
  l_tblspace VARCHAR2(80);
  l_count pls_integer := 0;
BEGIN
  SELECT
    COUNT(*)
  INTO
    l_count
  FROM
    dba_resumable
  WHERE
    1               =1
  AND SUSPEND_TIME IS NOT NULL
  AND resume_time  IS NULL
  AND status        ='SUSPENDED' ;
  IF l_count        > 0 THEN
    SELECT
      SUBSTR(regexp_substr(ERROR_MSG, 'tablespace [^[:blank:]]*', 1, 1, 'im'),
      12)
    INTO
      l_tblspace
    FROM
      dba_resumable
    WHERE
      1                       =1
    AND SUSPEND_TIME         IS NOT NULL
    AND resume_time          IS NULL
    AND status                ='SUSPENDED'
    AND rownum                <2 ;
    IF SUBSTR(l_tblspace,1,4)!='TEMP' THEN
      EXECUTE immediate 'alter tablespace '||l_tblspace||
      ' add datafile size 1m autoextend on next 8m';
    ELSE
      EXECUTE immediate 'alter tablespace '||l_tblspace||
      ' add tempfile size 128m autoextend on next 128m';
    END IF;
  END IF;
EXCEPTION
WHEN OTHERS THEN
  SYS.DBMS_SYSTEM.KSDDDT;
  SYS.DBMS_SYSTEM.KSDWRT(2, SQLERRM);
  SYS.DBMS_SYSTEM.KSDFLS;
END;
/
and to run a job every let's say 5 minutes, which would call such procedure.

Why take only one row from DBA_RESUMABLE? I assume there is not too much such events at a moment and the second one would be serviced with next job run.

Why cut a tablespace name from ERROR_MSG and not from ERROR_PARAMETERx? I found that in some circumstances (different db version for example) the ERROR_PARAMETERx are set differently or not at all. For 11.2 IIRC the tablespace name was set in ERROR_PARAMETER4.

Tuesday, 7 August 2012

Recycle bin and dropping tablespaces

Today I hit an interesting issue. First the instance have not shut down within an hour. Second I could not drop a schema. The reason seems to be an entry in the recycle bin left after some table, which was bound with a tablespace, which in turn has been dropped apparently some time ago. So no tablespace, while the entry left and dropping the schema or purging recycle bin have failed with error in recursive SQL.

The solution quite simple - creating a new tablespace with the old name, then purging the recycle bin again .

Thursday, 19 July 2012

Log rotation in 11g

Till version 11g the log rotation may been easily done by lsnrctl and set log_file to new name. With advent of ADR if used it blocks this way.
Excellent article on the theme IMHO at http://msutic.blogspot.com/2009/06/truncating-rotating-flushing.html.