Monday 12 May 2014

Zabbix on Oracle RDBMS and ORA-00001

Scenario

We performed today a small experiment with Zabbix - fast database restart with hope to keep the application alive based on the memory buffer and data kept in it during the database outage. This approach has failed even if sometimes we see Zabbix working through some time even with the database disabled.
Just after immediate database restart we observed a whole bunch of errors like the one below:
18264:20140512:134056.833 [Z3005] query failed: [-1] ORA-00001: unique constraint (ZABBIX.EVENTS_PK) violated [insert into 
events (eventid,source,object,objectid,clock,ns,value,value_changed) values (610764154,1,1,157,1399894856,0,1,0)]

Diagnosis

I set the diagnosis completely based on the symptoms, so it may seem not exact, but I suppose it is proper enough. In general it is bound with the mechanism for creating the ids for rows in Zabbix tables. The mechanism looks like the following:
  • an id is built from the value selected from the ids table for a row designated for a particular table increased by 1
  • the new value is used with a new row
  • the nextid value in the ids table is increased by one
Thus the ids table is used as an equivalent of sequence functionality. It seems however, that the values in this table not necessarily are chosen and updated within one transaction. Thus in the scenario like the one below we end up with the nextid value lower than the maximal value in tables to which Zabbix tries to insert new rows and those inserts fail with the error presented above.
In particular the frequent table with which we have trouble is the events table. Nothing strange though

Solution

UPDATE ZABBIX.IDS SET NEXTID=(select max(eventid) from events)+1 WHERE TABLE_NAME='events' AND FIELD_NAME='eventid';
commit;

4 comments:

deshike22 said...

After 4 years I am experiencing the same with zabbix 3.4.4 and Oracle 12c. Does this mean this has been fixed now by Zabbix?

rems said...

if You experience that while using version 3.4.4 (the newest is 3.4.5) I would say that this is still done the same way as previosly.
Maybe nobody notified them about such behaviour.

deshike22 said...

Thanks for the update. How did you fix it? Is this a zabbix problem or Oracle db problem?

rems said...

Well - in the article above there is one line with the solution we used