SDE and DBMS_CRYPTO

starting from ArcGIS version 10.1 while upgrading your ArcSDE Geodatabase in Oracle, one of the pre-requisite required “sde” user privilege is direct execution to the Package DBMS_CRYPTO.

grant execute on SYS.DBMS_CRYPTO to sde;

according to esri documentation: http://resources.arcgis.com/en/help/main/10.1/index.html#//002n0000002v000000

This granting has a relation to st_geometry configuration:

1. Allows the creation of a map member function for the ST_Geometry type, which is called whenever a spatial union or intersection is performed.

2. Upgrade a map member function for the ST_Geometry type, which is called whenever a spatial union or intersection is performed.

the question is: has this privilege has any relation with data encryption ?

Most Probably ‘YES ‘ : according to Oracle Documentation “DBMS_CRYPTO provides an interface to encrypt and decrypt stored data, and can be used in conjunction with PL/SQL programs running network communications. It provides support for several industry-standard encryption and hashing algorithms, including the Advanced Encryption Standard (AES) encryption algorithm.”

So, if you enable data encryption at rest on your database management system, then sde will be able to decrypt data for your geodatabase system.

Oracle Reference Link:http://docs.oracle.com/cd/B19306_01/appdev.102/b14258/d_crypto.htm

how to fetch or extract oracle database operator

To fetch (extract) sql definition of an Oracle Database Operator or Indextype, this can be done through the DBMS_METADATA package. Moreover,  DBMS_METADATA provides a way for you to retrieve metadata from the database dictionary as XML or creation DDL and to submit the XML to re-create the object.

select dbms_metadata.get_ddl(‘OPERATOR’,’operatorname‘) from dual;

operatorname : substitute it with the actual name of the operator

This can be done also for indextype, you can extract the sql definition using the below query:

select dbms_metadata.get_ddl(‘INDEXTYPE’,’indextypename’) from dual;

 

Underlying DBMS error [Layer not found] [Error executing PL/SQL Block db_stgeom_create_indexORA-29833: indextype does not exist]

while upgrading ArcSDE Geodatabase from version 9.3.1 SP2 to version 10 SP5, the below error was faced:

Underlying DBMS error [Layer not found] [Error executing PL/SQL Block db_stgeom_create_indexORA-29833: indextype does not exist]

This error indicates that you are missing an indextype in your ArcSDE Geodatabase in Oracle, the indextype name is: “ST_SPATIAL_INDEX”

when attempting to execute the following Sql Query to create the index type:

*******************************************************************************************************

CREATE OR REPLACE INDEXTYPE “SDE”.”ST_SPATIAL_INDEX” FOR “SDE”.”ST_BUFFER_INTERSECTS” (“SDE”.”ST_GEOMETRY”, “SDE”.”ST_GEOMETRY”, NUMBER), “SDE”.”ST_CROSSES” (“SDE”.”ST_GEOMETRY”, “SDE”.”ST_GEOMETRY”), “SDE”.”ST_ENVINTERSECTS” (“SDE”.”ST_GEOMETRY”, “SDE”.”ST_GEOMETRY”),  “SDE”.”ST_ENVINTERSECTS” (“SDE”.”ST_GEOMETRY”, “SDE”.”ST_GEOMETRY”, VARCHAR2),”SDE”.”ST_ENVINTERSECTS” (“SDE”.”ST_GEOMETRY”, NUMBER, NUMBER, NUMBER, NUMBER, VARCHAR2),  “SDE”.”ST_ENVINTERSECTS” (“SDE”.”ST_GEOMETRY”, NUMBER, NUMBER, NUMBER, NUMBER), “SDE”.”ST_EQUALS” (“SDE”.”ST_GEOMETRY”, “SDE”.”ST_GEOMETRY”), “SDE”.”ST_INTERSECTS” (“SDE”.”ST_GEOMETRY”, “SDE”.”ST_GEOMETRY”), “SDE”.”ST_ORDERINGEQUALS” (“SDE”.”ST_GEOMETRY”, “SDE”.”ST_GEOMETRY”),  “SDE”.”ST_OVERLAPS” (“SDE”.”ST_GEOMETRY”, “SDE”.”ST_GEOMETRY”), “SDE”.”ST_RELATE” (“SDE”.”ST_GEOMETRY”, “SDE”.”ST_GEOMETRY”, VARCHAR2), “SDE”.”ST_TOUCHES” (“SDE”.”ST_GEOMETRY”, “SDE”.”ST_GEOMETRY”),”SDE”.”ST_WITHIN” (“SDE”.”ST_GEOMETRY”, “SDE”.”ST_GEOMETRY”)USING “SDE”.”ST_DOMAIN_METHODS” WITH LOCAL RANGE PARTITION;

**********************************************************************************************************

the following error was received:

ORA-29830: operator does not exist
ORA-29830: operator does not exist

This error indicates that for some reason there are missing operators in your ArcSDE geodatabase, you need to create them.

but, first you need to query the complete list of operators from another ArcSDE geodatabase that is intact and healthy, using the query: select * from user_operators;

Image

compare the list of operators where the ArcSDE geodatabase that has the upgrade issue, and create the missing ones.

In order to extract the sql definition of the operator use the following query as an example:

select dbms_metadata.get_ddl(‘OPERATOR’,’ST_WITHIN’) from dual;

after creating all missing operators, try executing the previous sql query mentioned: “CREATE OR REPLACE INDEXTYPE “SDE”.”ST_SPATIAL_INDEX

then use the following sql, to check indextype was created:

select * from user_indextypes;

ST_SPATIAL_INDEX indextype
ST_SPATIAL_INDEX indextype

Re-Run the Upgrade, and it will be successful.

Query Reference:

select * from user_operators;

select * from user_indextypes;

select dbms_metadata.get_ddl(‘INDEXTYPE’,’ST_SPATIAL_INDEX’) from dual;

select dbms_metadata.get_ddl(‘OPERATOR’,’ST_WITHIN’) from dual;

If this is not the case you are facing….but facing the same error then try the following:

 

SQL> GRANT EXECUTE ON sde.st_spatial_index TO SCHEMA_NAME;

OR

SQL> GRANT EXECUTE ON sde.st_spatial_index TO PUBLIC;

AND

SQL> GRANT CREATE INDEXTYPE TO SCHEMA_NAME;

 

Oracle Temporary Table Space size

As a DBA you may face a problem with your database if TEMP tablespace gets full. Usually, TEMP table space is used for “sorting and aggregate operations“.

One of the reasons your TEMP tablspace get full and expanding, is that the application queries are not well optimized/tuned.

the following queries will eventually help you in diagnosis:

To Find SQL Statements that is causing TEMP SPACE TO BE FILLED:
//——————————————————————–

select s.sid || ‘,’ || s.serial# sid_serial, s.username,

o.blocks * t.block_size / 1024 / 1024 mb_used, o.tablespace,

o.sqladdr address, h.hash_value, h.sql_text
from v$sort_usage o, v$session s, v$sqlarea h, dba_tablespaces t
where o.session_addr = s.saddr
and o.sqladdr = h.address (+)
and o.tablespace = t.tablespace_name
order by s.sid;
/ /———————————————————————
To Find Current TEMP SPACE SIZE:

select * from (select a.tablespace_name,
sum(a.bytes/1024/1024) allocated_mb
from dba_temp_files a
where a.tablespace_name = upper(‘&&temp_tsname’) group by a.tablespace_name) x,
(select sum(b.bytes_used/1024/1024) used_mb,
sum(b.bytes_free/1024/1024) free_mb
from v$temp_space_header b
where b.tablespace_name=upper(‘&&temp_tsname’) group by b.tablespace_name);

Convert BLOB TO CLOB column – METHOD 2

in my previous post, i have provided a way to convert BLOB to CLOB.

There is another way to do the conversion, based on Oracle Metalink DocID: 9876

*** Let us assume that the table name is Z.EMPLOYEES . And, let us assume that the column name that has a ‘BLOB’ data type is BIN_EMP. and the Temp column with ‘CLOB’ data type has the name WTF.

1. take export backup (expdp) of the ORIGINAL table.

 

2. add new temp column:

alter table Z.EMPLOYEES add (WTF CLOB);

3. start initialization:

UPDATE Z.EMPLOYEES SET WTF = empty_clob() WHERE BIN_EMP IS NOT NULL;

4. create the procedure for conversion, the following procedure will perform the conversion:

create or replace procedure Z.blob_to_clob_PROC as
a number;
b number;
c number := 1000;
d number;
offset number := 1;
v_clob clob;
amount binary_integer;
buffer1 raw(1000);
buffer2 varchar2(1000);
cursor c1 is select BIN_EMP, rowid from Z.EMPLOYEES where BIN_EMP is not null;
begin
for i in c1 loop
select WTF into v_clob from Z.EMPLOYEES where rowid = i.rowid for update;
a := dbms_lob.getlength(i.BIN_EMP);
b := ceil(a/c);
for j in 1..b loop
if (c*j <= a) then
d:= c;
else
d:= a – c*(j-1);
end if;
DBMS_LOB.READ ( i.BIN_EMP, d, offset, buffer1);
buffer2 := utl_raw.cast_to_varchar2(buffer1);
DBMS_LOB.WRITEAPPEND(v_clob, d, buffer2);
offset := offset + c;
end loop;
offset := 1;
update Z.EMPLOYEES set WTF = v_clob where rowid=i.rowid;
end loop;
commit;
end;
/

5. execute the following statements:

EXECUTE Z.blob_to_clob_PROC;
commit;

Remark: you can develop Unix Shell script for automating the execution of the procedure to be scheduled.

6. drop the original column with BLOB data type (BIN_EMP)

7.  rename the column WTF to BIN_EMP

 

converting BLOB to CLOB datatype in oracle

Converting BLOB column data type to CLOB is a very tedious task, since you cannot do it directly.

 

First, you need to create an “Oracle Function” under an oracle schema you have access to, the following is the function:

 

CREATE OR REPLACE FUNCTION blob_to_clob (blob_in IN BLOB)

RETURN CLOB

AS

v_clob CLOB;

v_varchar VARCHAR2(32767);

v_start PLS_INTEGER := 1;

v_buffer PLS_INTEGER := 32767;

BEGIN

DBMS_LOB.CREATETEMPORARY(v_clob, TRUE);

 

FOR i IN 1..CEIL(DBMS_LOB.GETLENGTH(blob_in) / v_buffer)

LOOP

 

v_varchar := UTL_RAW.CAST_TO_VARCHAR2(DBMS_LOB.SUBSTR(blob_in, v_buffer, v_start));

 

DBMS_LOB.WRITEAPPEND(v_clob, LENGTH(v_varchar), v_varchar);

v_start := v_start + v_buffer;

END LOOP;

RETURN v_clob;

END blob_to_clob;

/

 

Second, alter the table by adding a temporary column with a name similar to the BLOB column name.

For example, let’s say the BLOB column name is STRING_NAMES, the new CLOB column will be STRING_NAMES2

 

Sql command:

 

Alter table DUMMY.TABLE add (STRING_NAMES2 CLOB);

 

Then, use the “update” statement to populate the column with the converted data:

 

update DUMMY.TABLE  set STRING_NAMES2=blob_to_clob(STRING_NAMES);

 

Once finished drop the original column, then rename the new one to the original name.

 

If your table has huge number of records, you may need to develop a script and schedule it to perform this intensive information.

*** UPDATE:

another way of converting BLOB to CLOB can be found here:

https://geodatamaster.wordpress.com/2014/01/01/convert-blob-to-clob-column-method-2/