CDX RDD question (live usage/compatibility)

[Maurizio la Cecilia]

Sorry for reopening a very old and OT thread, but my needs are quite the same
as Viktor's one and i think something changed after 2005.

I ported successfully a large clipper 5.2e application, working on many
networks, to Harbour 2.0.
Now, before to swap the installation to Harbour version, i would to test the
effectiveness of the job using one or few Harbour workstation concurrently
with old Clipper ones on the same network.

Thus, i changed the Clipper app to use the DBFCDX driver and compiled
Harbour using the same driver and setting the locking scheme to CLIP (not
CLIP53). The adopted code is:

REQUEST DBFCDX
RDDSetDefault("DBFCDX")
RddInfo( RDDI_LOCKSCHEME, DB_DBFLOCK_CLIP )

REQUEST HB_LANG_IT
REQUEST HB_CODEPAGE_IT850
hb_cdpSelect('IT850')
hb_LangSelect('IT')

Apart of a dirty Clipper problem about the compound indexes (i posted
yesterday a message on the google clipper group) solved using the INDEX
command in place of the OrdCondSet()/OrdCreate() pair, the program crashes
only in one function, giving a "lock required" error after a successfully
lock of the same record. The strange thing is that the flow is apparently
the same of other parts of the application working fine.

Thus, i'm thinking that i gone wrong in some configuration option or i'm
ignoring some bug of the adopted implementation.
Ther'is any hint about the DBFCDX driver coexistence? Ther'is another better
path to try (not SQL based, as i need to mantain DBF approach before
quitting old Clipper world).
TIA.
Maurizio

Hi all,
Not strictly a developer question, but could anyone advise
me, if the (x)Harbour CDX RDD is ready/recommended for
heavy-duty live networked usage at the current stage?
I'd like to put some Harbour compiled (win32) systems into
live testing (first with 2-3 concurrent users), and I'd like to
avoid database/index corruption.
Is the locking method in (x)Harbour CDX compatible with
CA-Clipper 5.2e? (I'm using SIXCDX actually)
Any stories and opinions are very welcome.
Brgds,
Viktor
PS: ADS is not an option for me.
_______________________________________________
Harbour mailing list
http://lists.harbour-project.org/mailman/listinfo/harbour

[Maurizio la Cecilia]

I forgotten to say that the same piece of code works fine in Clipper
5.2e+DBFNTX and in Harbour+DBFCDX.
Maybe the opportunistic lock take a rule in that?
TIA.
Maurizio

Post by Maurizio la Cecilia
Sorry for reopening a very old and OT thread, but my needs are quite the
same as Viktor's one and i think something changed after 2005.
I ported successfully a large clipper 5.2e application, working on many
networks, to Harbour 2.0.
Now, before to swap the installation to Harbour version, i would to test
the effectiveness of the job using one or few Harbour workstation
concurrently with old Clipper ones on the same network.
Thus, i changed the Clipper app to use the DBFCDX driver and compiled
Harbour using the same driver and setting the locking scheme to CLIP (not
REQUEST DBFCDX
RDDSetDefault("DBFCDX")
RddInfo( RDDI_LOCKSCHEME, DB_DBFLOCK_CLIP )
REQUEST HB_LANG_IT
REQUEST HB_CODEPAGE_IT850
hb_cdpSelect('IT850')
hb_LangSelect('IT')
Apart of a dirty Clipper problem about the compound indexes (i posted
yesterday a message on the google clipper group) solved using the INDEX
command in place of the OrdCondSet()/OrdCreate() pair, the program crashes
only in one function, giving a "lock required" error after a successfully
lock of the same record. The strange thing is that the flow is apparently
the same of other parts of the application working fine.
Thus, i'm thinking that i gone wrong in some configuration option or i'm
ignoring some bug of the adopted implementation.
Ther'is any hint about the DBFCDX driver coexistence? Ther'is another
better path to try (not SQL based, as i need to mantain DBF approach
before quitting old Clipper world).
TIA.
Maurizio

Hi all,
Not strictly a developer question, but could anyone advise
me, if the (x)Harbour CDX RDD is ready/recommended for
heavy-duty live networked usage at the current stage?
I'd like to put some Harbour compiled (win32) systems into
live testing (first with 2-3 concurrent users), and I'd like to
avoid database/index corruption.
Is the locking method in (x)Harbour CDX compatible with
CA-Clipper 5.2e? (I'm using SIXCDX actually)
Any stories and opinions are very welcome.
Brgds,
Viktor
PS: ADS is not an option for me.
_______________________________________________
Harbour mailing list
http://lists.harbour-project.org/mailman/listinfo/harbour

[Przemysław Czerpak]

On Mon, 25 Jan 2010, Maurizio la Cecilia wrote:

Hi,

Post by Maurizio la Cecilia
Sorry for reopening a very old and OT thread, but my needs are quite the same
as Viktor's one and i think something changed after 2005.
I ported successfully a large clipper 5.2e application, working on many
networks, to Harbour 2.0.
Now, before to swap the installation to Harbour version, i would to test the
effectiveness of the job using one or few Harbour workstation concurrently
with old Clipper ones on the same network.
Thus, i changed the Clipper app to use the DBFCDX driver and compiled
Harbour using the same driver and setting the locking scheme to CLIP (not

CL52 DBFCDX/SIX3 SIXCDX drivers are broken and can corrupt index files.
so I cannot give you any guaranties that it will work. Anyhow if
you want to use it and access the same index files by Harbour then
you should use in Harbour SIXCDX RDD which tries to replicate some
low level SIXCDX behavior. It creates less efficient indexes in some
cases but it's a workaround for some CL52/SIXCDX bugs.
CL53 DBFCDX/COMIX seems to be much better choice for Clipper.
Additionally you have to chose _EXACTLY_ the same locking schemes in
both compilers.
CL52 DBFCDX/SIXCDX uses FP locking. CL53 DBFCDX/COMIX use CL53 locking
but it can be changed to FP locking if you link with your application
cdxlock.obj

Post by Maurizio la Cecilia
REQUEST DBFCDX
RDDSetDefault("DBFCDX")
RddInfo( RDDI_LOCKSCHEME, DB_DBFLOCK_CLIP )

None of Clipper CDX drivers uses such locking scheme so it's wrong.

Post by Maurizio la Cecilia
REQUEST HB_LANG_IT
REQUEST HB_CODEPAGE_IT850
hb_cdpSelect('IT850')
hb_LangSelect('IT')

If you are using HB_CODEPAGE_IT850 in Harbour then you have to link your
Clipper application with ntxita.obj. Otherwise index files will be corrupted
because Clipper and Harbour will use different collation orders.

Post by Maurizio la Cecilia
Apart of a dirty Clipper problem about the compound indexes (i posted
yesterday a message on the google clipper group) solved using the INDEX
command in place of the OrdCondSet()/OrdCreate() pair, the program crashes

Harbour and CL53 DBFCDX/COMIX do not support none compound IDX indexes
so I guess you are using CL52 DBFCDX or SIXCDX driver.

Post by Maurizio la Cecilia
only in one function, giving a "lock required" error after a successfully
lock of the same record. The strange thing is that the flow is apparently
the same of other parts of the application working fine.

In Clipper and Harbour 1022 "Lock required" RT error is logical error.
It means that programmer didn't successfully locked the record. The
physical record lock is not checked when this error is generated.
It means that it's a problem with application code. Check why it's
exploited when you compile it using Harbour. It's possible that setting
valid locks scheme will hide the problem but for sure it will not fix
the code. Such error should not appear in code which always checks
results of RLOCK/FLOCK operations and never tries to write to not locked
record so it's something what you should try to locate and fix in your code.

Post by Maurizio la Cecilia
Thus, i'm thinking that i gone wrong in some configuration option or i'm
ignoring some bug of the adopted implementation.
Ther'is any hint about the DBFCDX driver coexistence? Ther'is another better
path to try (not SQL based, as i need to mantain DBF approach before
quitting old Clipper world).

See above. You have to use exactly the same locking scheme and national
sorting order inb both applications. If you are using CL52 DBFCDX or SIXCDX
then use in Harbour SIXCDX too. If you are using CL53 DBFCDX or COMIX them
use in Harbour DBFCDX.
CL52 DBFCDX and SIXCDX have serious bugs and for me should never be used
in production environment. Anyhow it's possible that your code does not
exploit them very often so you can live with it.

HTH,

best regards,
Przemek

[Maurizio la Cecilia]

Post by Przemysław Czerpak
CL52 DBFCDX/SIX3 SIXCDX drivers are broken and can corrupt
index files.
so I cannot give you any guaranties that it will work. Anyhow if
you want to use it and access the same index files by Harbour then
you should use in Harbour SIXCDX RDD which tries to replicate some
low level SIXCDX behavior. It creates less efficient indexes in some
cases but it's a workaround for some CL52/SIXCDX bugs.
CL53 DBFCDX/COMIX seems to be much better choice for Clipper.

Ok. I realize. No chance to recompile in Cl53, however, as i don't own it.

Post by Przemysław Czerpak
Additionally you have to chose _EXACTLY_ the same locking schemes in
both compilers.
CL52 DBFCDX/SIXCDX uses FP locking. CL53 DBFCDX/COMIX use CL53 locking
but it can be changed to FP locking if you link with your application
cdxlock.obj

Post by Maurizio la Cecilia
REQUEST DBFCDX
RDDSetDefault("DBFCDX")
RddInfo( RDDI_LOCKSCHEME, DB_DBFLOCK_CLIP )

None of Clipper CDX drivers uses such locking scheme so it's wrong.

Ok. My mistake. I based my choice on xHarbour manual saying:
"Setting the locking scheme to 1 will lock the database files like
CA-Clipper 5.2 does. To use CA-Clipper 5.3's locking scheme, set
DBFLOCKSCHEME to 2. This will emulate shared locks using exclusive locks.
Visual FoxPro's locking scheme is selected with DBFLOCKSCHEME set to 3."

Post by Przemysław Czerpak
If you are using HB_CODEPAGE_IT850 in Harbour then you have
to link your
Clipper application with ntxita.obj. Otherwise index files
will be corrupted
because Clipper and Harbour will use different collation orders.

I ever link ntxita.obj, also in DBFNTX build.

Post by Przemysław Czerpak
Harbour and CL53 DBFCDX/COMIX do not support none compound IDX indexes
so I guess you are using CL52 DBFCDX or SIXCDX driver.

Sorry, i was meaning production ".CDX" multitag indexes...

Post by Przemysław Czerpak
In Clipper and Harbour 1022 "Lock required" RT error is logical error.
It means that programmer didn't successfully locked the record. The
physical record lock is not checked when this error is generated.
It means that it's a problem with application code. Check why it's
exploited when you compile it using Harbour. It's possible
that setting
valid locks scheme will hide the problem but for sure it will not fix
the code. Such error should not appear in code which always checks
results of RLOCK/FLOCK operations and never tries to write to
not locked
record so it's something what you should try to locate and
fix in your code.

The problem is ONLY in Cl52+DBFCDX. With app built with Harbour or
Cl52+DBFNTX all works fine.
The Rlock() returns .t. in all the builded app, BUT ONLY in
Cl52+DBFCDX exits a "Lock required" error at subsequent field replace.
This happens also if the Cl52+DBFCDX is the ONLY program running on
the network...
So, i think that the problem is not deriving from my code... Same code
works fine in all other points of app updating a record.

Post by Przemysław Czerpak
See above. You have to use exactly the same locking scheme
and national
sorting order inb both applications. If you are using CL52
DBFCDX or SIXCDX
then use in Harbour SIXCDX too. If you are using CL53 DBFCDX
or COMIX them
use in Harbour DBFCDX.
CL52 DBFCDX and SIXCDX have serious bugs and for me should
never be used
in production environment. Anyhow it's possible that your
code does not
exploit them very often so you can live with it.

The feel i receive from your message is that i'm going in a puzzled
challenge, so i think i'll have two options:
A) To leave the Cl52+DBFNTX app, working well from twenty year ago, and
to change the driver of Harbour to DBFNTX
B) To take the courage of replace with Harbour+DBFCDX all the install

I don't know the effectiveness of the DBFNTX in Harbour, so i ask you
if also the A) choice is dangerous and the pure Harbour path is the only
affordable.
My great thanks for your support and precious feedback.
Best regards.
Maurizio

[Przemysław Czerpak]

Post by Maurizio la Cecilia
"Setting the locking scheme to 1 will lock the database files like
CA-Clipper 5.2 does. To use CA-Clipper 5.3's locking scheme, set
DBFLOCKSCHEME to 2. This will emulate shared locks using exclusive locks.
Visual FoxPro's locking scheme is selected with DBFLOCKSCHEME set to 3."

1 works like CL52/CL53 DBFNTX if you do not link ntxlock.obj with
your code.

Post by Maurizio la Cecilia

Post by Przemysław Czerpak
If you are using HB_CODEPAGE_IT850 in Harbour then you have
to link your
Clipper application with ntxita.obj. Otherwise index files
will be corrupted
because Clipper and Harbour will use different collation orders.

I ever link ntxita.obj, also in DBFNTX build.

The name is confusion. These .obj files change collation order in
Clipper VM and all index formats not omnly NTX.

Post by Maurizio la Cecilia

Post by Przemysław Czerpak
Harbour and CL53 DBFCDX/COMIX do not support none compound IDX indexes
so I guess you are using CL52 DBFCDX or SIXCDX driver.

Sorry, i was meaning production ".CDX" multitag indexes...

And what was your problem?

Post by Maurizio la Cecilia
The problem is ONLY in Cl52+DBFCDX. With app built with Harbour or
Cl52+DBFNTX all works fine.
The Rlock() returns .t. in all the builded app, BUT ONLY in
Cl52+DBFCDX exits a "Lock required" error at subsequent field replace.
This happens also if the Cl52+DBFCDX is the ONLY program running on
the network...
So, i think that the problem is not deriving from my code... Same code
works fine in all other points of app updating a record.

The DBFCDX driver from CL5.2 has few very serious bugs which can cause
memory corruption which can cause any other bad side effects so everything
is possible.

Post by Maurizio la Cecilia
The feel i receive from your message is that i'm going in a puzzled
A) To leave the Cl52+DBFNTX app, working well from twenty year ago, and
to change the driver of Harbour to DBFNTX

It's possible.

Post by Maurizio la Cecilia
B) To take the courage of replace with Harbour+DBFCDX all the install

It's probably safer method. Anyhow as long as some network transport
layer does not cause some separate buffering which can cause problems
when DOS and Windows applications access files concurrently then it
should be safe to share database files and indexes between Clipper
and Harbour. It's enough to set compatible national sorting and locking
schemes.

Post by Maurizio la Cecilia
I don't know the effectiveness of the DBFNTX in Harbour, so i ask you
if also the A) choice is dangerous and the pure Harbour path is the only
affordable.

In Harbour DBFNTX is probably much faster then in Clipper and for sure
they can be used to index realy big tables (I know that some people
use Harbour only to reindex files for Clipper due to huge speed differnce
and the fact that over some size limits Clipper begins to GPF during
indexing). Harbour DBFNTX implementation also supports all CDX extensions
and few others but you will not be able to enable all of them if you want
to share data with Clipper. Compound index, large NTX file support up to
4TB or using RECNO as hidden part of index key to eliminate linear scan
during record updating in indexes uses keys with big number of repeated
values are extensions which change index formats and I implemented them
only in Harbour and xHarbour so only in these languages can operate on
such NTX indexes.
Technically NTX format is much faster then CDX. In the ideal environment
where you have a lot of RAM and you can keep index files in memory CDX
is many times slower then NTX. But we are not leaving in ideal world and
in most of cases indexes are accessed using slow network connections.
It means that the size of index files will determinate the speed. CDX
files store keys in leaf nodes in compressed form what greatly reduces
the total index size and it's the reason of better overall performance
when the cost of reading/writing from/to index file is very big, i.e.
files are stored on file server.

best regards,
Przemek

[Maurizio la Cecilia]

-----Messaggio originale-----
Inviato: martedì 26 gennaio 2010 20.05
A: Harbour Project Main Developer List.
Oggetto: Re: R: [Harbour] CDX RDD question (live usage/compatibility)
1 works like CL52/CL53 DBFNTX if you do not link ntxlock.obj with
your code.

Now it's clear.

The name is confusion. These .obj files change collation order in
Clipper VM and all index formats not omnly NTX.

Yes. I know that.

Post by Maurizio la Cecilia
The feel i receive from your message is that i'm going in a puzzled
A) To leave the Cl52+DBFNTX app, working well from twenty

year ago, and

Post by Maurizio la Cecilia
to change the driver of Harbour to DBFNTX

It's possible.

Post by Maurizio la Cecilia
B) To take the courage of replace with Harbour+DBFCDX all

the install
It's probably safer method.
Anyhow as long as some network transport
layer does not cause some separate buffering which can cause problems
when DOS and Windows applications access files concurrently then it
should be safe to share database files and indexes between Clipper
and Harbour. It's enough to set compatible national sorting
and locking
schemes.

Post by Maurizio la Cecilia
I don't know the effectiveness of the DBFNTX in Harbour, so

i ask you

Post by Maurizio la Cecilia
if also the A) choice is dangerous and the pure Harbour

path is the only

Post by Maurizio la Cecilia
affordable.

In Harbour DBFNTX is probably much faster then in Clipper and for sure
they can be used to index realy big tables (I know that some people
use Harbour only to reindex files for Clipper due to huge
speed differnce
and the fact that over some size limits Clipper begins to GPF during
indexing). Harbour DBFNTX implementation also supports all
CDX extensions
and few others but you will not be able to enable all of them
if you want
to share data with Clipper. Compound index, large NTX file
support up to
4TB or using RECNO as hidden part of index key to eliminate
linear scan
during record updating in indexes uses keys with big number
of repeated
values are extensions which change index formats and I
implemented them
only in Harbour and xHarbour so only in these languages can operate on
such NTX indexes.
Technically NTX format is much faster then CDX. In the ideal
environment
where you have a lot of RAM and you can keep index files in memory CDX
is many times slower then NTX. But we are not leaving in
ideal world and
in most of cases indexes are accessed using slow network connections.
It means that the size of index files will determinate the speed. CDX
files store keys in leaf nodes in compressed form what greatly reduces
the total index size and it's the reason of better overall performance
when the cost of reading/writing from/to index file is very big, i.e.
files are stored on file server.

Your technical analysis is very useful and only now i realize that the
NTX rdd is expanded in Harbour. A quick view on whatsnew.txt told me
that was present yet in version 0.46.
Anyway, the Harbour CDX install on all clients could be the safer choice,
as you say.
Again my great thanks for your support and kindness.
Best regards.
Maurizio