Board index » delphi » TIP: BDE 2.52, Corrupted indexes, international characters, D1

Markku Nevalaine

Delphi Developer
Wed, 18 Jun 1902 08:00:00 GMT

TIP: BDE 2.52, Corrupted indexes, international characters, D1

One newly added Paradox table in an old D1, BDE 2.52 app started to give
"Corrupt index" messages. From Dejanews I found an old thread from last year,
talking about the same thing:
http://x14.dejanews.com/dnquery.xp?search=thread&svcclass=dnserver&threaded=1&ST=PS&CONTEXT=914364718.989069483&HIT_CONTEXT=914364718.989069483&HIT_NUM=252&recnum=%3c34169226.4...@netlife.fi%3e%231/1

The problem occurs if you have international characters (Scandinavian, German,
Spanish etc.) in a field that is a _secondary_ index field. There is no problem
as long as you add new records, but when you delete or edit one, the index
error message appears.
If the field is a primary index field, there are no problems also.

The problem also appears _only_ with Paradox 4.0 version tables, with 5.0
tables everything works fine.

Database Desktop (DBD) is a real CROOK here, that's why it was quite difficult
to find what was wrong. When you use Delphi-1 DBD to create a new Table, and
choose it to be "Paradox 5.0 for Windows" then DBD still creates it to be
Paradox 4.0 level table.

Even if you have a nicely working Paradox 5.0 table, and you use DBD's
"Borrow structure" feature to create a new table, it still makes it
Paradox 4.0 table. This DBD's downgrading feature happens always, if there
are not for instance Logical type Field, that causes the new table to
become Paradox 5.0 version.

Currently, there really seems not be any index corrupting problems even
with international characters and D1 / BDE2.52, as long as you make sure
the Paradox table version is 5.0.

Markku Nevalainen

 

Delphi Developer
Wed, 18 Jun 1902 08:00:00 GMT

Re:TIP: BDE 2.52, Corrupted indexes, international characters, D1


Guy Messel

Delphi Developer
Wed, 18 Jun 1902 08:00:00 GMT

Re:TIP: BDE 2.52, Corrupted indexes, international characters, D1


nope occurs with any version .. 5 .. 7... no fix possible with bde 2.52

Markku Nevalainen <m...@iki.fi> a crit dans l'article
<36802395.3...@iki.fi>...

Quote
> One newly added Paradox table in an old D1, BDE 2.52 app started to give
> "Corrupt index" messages. From Dejanews I found an old thread from last
year,
> talking about the same thing:

http://x14.dejanews.com/dnquery.xp?search=thread&svcclass=dnserver&th...
1&ST=PS&CONTEXT=914364718.989069483&HIT_CONTEXT=914364718.989069483&HIT_NUM=
252&recnum=%3c34169226.4...@netlife.fi%3e%231/1

Quote

> The problem occurs if you have international characters (Scandinavian,
German,
> Spanish etc.) in a field that is a _secondary_ index field. There is no
problem
> as long as you add new records, but when you delete or edit one, the
index
> error message appears.
> If the field is a primary index field, there are no problems also.

> The problem also appears _only_ with Paradox 4.0 version tables, with 5.0
> tables everything works fine.

> Database Desktop (DBD) is a real CROOK here, that's why it was quite
difficult
> to find what was wrong. When you use Delphi-1 DBD to create a new Table,
and
> choose it to be "Paradox 5.0 for Windows" then DBD still creates it to be
> Paradox 4.0 level table.

> Even if you have a nicely working Paradox 5.0 table, and you use DBD's
> "Borrow structure" feature to create a new table, it still makes it
> Paradox 4.0 table. This DBD's downgrading feature happens always, if
there
> are not for instance Logical type Field, that causes the new table to
> become Paradox 5.0 version.

> Currently, there really seems not be any index corrupting problems even
> with international characters and D1 / BDE2.52, as long as you make sure
> the Paradox table version is 5.0.

> Markku Nevalainen

Markku Nevalaine

Delphi Developer
Wed, 18 Jun 1902 08:00:00 GMT

Re:TIP: BDE 2.52, Corrupted indexes, international characters, D1


Quote
Guy Messely wrote:

> nope occurs with any version .. 5 .. 7... no fix possible with bde 2.52

For me it seems to work with 5.0 version. How exactly do you get the
corruption to happen? What language driver version are you using?

It usually happens if you try to delete some record with Intl characters.
I have not got errors any more, not earlier also, and I have a bunch of
5.0 tables out and running, with secondary index fields having international
characters.

Markku Nevalainen

Sundial Service

Delphi Developer
Wed, 18 Jun 1902 08:00:00 GMT

Re:TIP: BDE 2.52, Corrupted indexes, international characters, D1


Quote
Guy Messely wrote:

> nope occurs with any version .. 5 .. 7... no fix possible with bde 2.52

> Markku Nevalainen <m...@iki.fi> a crit dans l'article
> <36802395.3...@iki.fi>...
> > The problem occurs if you have international characters (Scandinavian,
> German,
> > Spanish etc.) in a field that is a _secondary_ index field. There is no
> problem
> > as long as you add new records, but when you delete or edit one, the
> index
> > error message appears.
> > If the field is a primary index field, there are no problems also.

As near as I can tell, Paradox is applying an ASCII-oriented algorithm
here.  It looks to me like it happens only when the index is "not case
sensitive" but I'd like someone to confirm or deny this.
Guy Messel

Delphi Developer
Wed, 18 Jun 1902 08:00:00 GMT

Re:TIP: BDE 2.52, Corrupted indexes, international characters, D1


it happens to me with index as "case sensitive" as well ... but occurs only
so far with accentuated caracters

Sundial Services <i...@sundialservices.com> a crit dans l'article
<3681394E.5...@sundialservices.com>...

Quote
> Guy Messely wrote:

> > nope occurs with any version .. 5 .. 7... no fix possible with bde 2.52

> > Markku Nevalainen <m...@iki.fi> a crit dans l'article
> > <36802395.3...@iki.fi>...
> > > The problem occurs if you have international characters
(Scandinavian,
> > German,
> > > Spanish etc.) in a field that is a _secondary_ index field. There is
no
> > problem
> > > as long as you add new records, but when you delete or edit one, the
> > index
> > > error message appears.
> > > If the field is a primary index field, there are no problems also.

> As near as I can tell, Paradox is applying an ASCII-oriented algorithm
> here.  It looks to me like it happens only when the index is "not case
> sensitive" but I'd like someone to confirm or deny this.

Guy Messel

Delphi Developer
Wed, 18 Jun 1902 08:00:00 GMT

Re:TIP: BDE 2.52, Corrupted indexes, international characters, D1


this is an old problem to exist and be known for a while in 16 bits BDE and
never been fixed.
when my customers get the index so{*word*222}up it needs to be rebuilted
because all the accentuated caracters are no more sorted properly ... then
i need to rebuilt the table in a 32 bits environment because no tool exists
in 16 bits to do it. This is right now on a paradox 5 version with Paradox
Intl850 driver. (Secondary index on names)
Unfortunately i will not carry any BDE 2.52 16 bits support anymore soon so
i will be free from that extra work frustrating since long time now.

Markku Nevalainen <m...@iki.fi> a crit dans l'article
<3680F031.1...@iki.fi>...

Quote
> Guy Messely wrote:

> > nope occurs with any version .. 5 .. 7... no fix possible with bde 2.52

> For me it seems to work with 5.0 version. How exactly do you get the
> corruption to happen? What language driver version are you using?

> It usually happens if you try to delete some record with Intl characters.
> I have not got errors any more, not earlier also, and I have a bunch of
> 5.0 tables out and running, with secondary index fields having
international
> characters.

> Markku Nevalainen

Markku Nevalaine

Delphi Developer
Wed, 18 Jun 1902 08:00:00 GMT

Re:TIP: BDE 2.52, Corrupted indexes, international characters, D1


Quote
Sundial Services wrote:

> As near as I can tell, Paradox is applying an ASCII-oriented algorithm
> here.  It looks to me like it happens only when the index is "not case
> sensitive" but I'd like someone to confirm or deny this.

Yes, corruption happens only with non case sensitive indexes, and perhaps
that's why never happens with primary indexes, that is always case
sensitive.

Markku Nevalainen

Markku Nevalaine

Delphi Developer
Wed, 18 Jun 1902 08:00:00 GMT

Re:TIP: BDE 2.52, Corrupted indexes, international characters, D1


Quote
Guy Messely wrote:

> when my customers get the index so{*word*222}up it needs to be rebuilted
> because all the accentuated caracters are no more sorted properly ... then
> i need to rebuilt the table in a 32 bits environment because no tool exists
> in 16 bits to do it.

What is that 32-bit repair tool? I can't (easily) create a test situation
with accent characters data having corrupted the database. But on a daily
basis I use the standard TUTILITY to fix the database files. And if the index
files are in a totally useless state, let the free Super Page utility FIXIT
rebuild them from scratch.

Quote
> Intl850 driver. (Secondary index on names)

I noticed that I have used mainly Paradox Ascii, but also SWEDFIN,
and both seem to work ok, database does not get corrupted. I'm not
sure if the different Language Drivers make much difference, than
being able to sort the database in right order.

Markku Nevalainen

Guy Messel

Delphi Developer
Wed, 18 Jun 1902 08:00:00 GMT

Re:TIP: BDE 2.52, Corrupted indexes, international characters, D1


database does not really get corrupted totally, but its secondary index on
that alpha field where you would use accentuated caracters becomes easily
screwed up.
Just look at the table using its secondary index after rebuilting the table
with bde2.52 and tutility, you should see the misplacement of the words. At
first it seems to work but later with more data you will see it. Example:
all "Ba" get after "Beb" ...

Markku Nevalainen <m...@iki.fi> a crit dans l'article
<3683F240.7...@iki.fi>...

Quote
> Guy Messely wrote:

> > when my customers get the index so{*word*222}up it needs to be rebuilted
> > because all the accentuated caracters are no more sorted properly ...
then
> > i need to rebuilt the table in a 32 bits environment because no tool
exists
> > in 16 bits to do it.

> What is that 32-bit repair tool? I can't (easily) create a test situation
> with accent characters data having corrupted the database. But on a daily
> basis I use the standard TUTILITY to fix the database files. And if the
index
> files are in a totally useless state, let the free Super Page utility
FIXIT
> rebuild them from scratch.

> > Intl850 driver. (Secondary index on names)

> I noticed that I have used mainly Paradox Ascii, but also SWEDFIN,
> and both seem to work ok, database does not get corrupted. I'm not
> sure if the different Language Drivers make much difference, than
> being able to sort the database in right order.

> Markku Nevalainen

Sundial Service

Delphi Developer
Wed, 18 Jun 1902 08:00:00 GMT

Re:TIP: BDE 2.52, Corrupted indexes, international characters, D1


Quote
Guy Messely wrote:

> database does not really get corrupted totally, but its secondary index on
> that alpha field where you would use accentuated caracters becomes easily
> screwed up.
> Just look at the table using its secondary index after rebuilting the table
> with bde2.52 and tutility, you should see the misplacement of the words. At
> first it seems to work but later with more data you will see it. Example:
> all "Ba" get after "Beb" ...

> > I noticed that I have used mainly Paradox Ascii, but also SWEDFIN,
> > and both seem to work ok, database does not get corrupted. I'm not
> > sure if the different Language Drivers make much difference, than
> > being able to sort the database in right order.

> > Markku Nevalainen

As far as I can see, the secondary-index files (which, in fact, consist
of a table (Xnn) and a primary-index to that table (Ynn)) are built
using the Ascii sort order even when the table they are indexing uses an
international language-driver.  In cases where the Ascii sort-order does
not match that of the international driver, namely(!) where diacriticals
characters are used, the sort-order obtained by following the index is
not the proper international collation but an Ascii-bastardized
byte-wise version of it.

Since I am right now heavily involved in working on a new product
(ChimneySweep 3.0) that verifies and works with index files directly
{doing so 25 to 100 times faster than TUtility!!}, I have an enormous
stake in understanding this problem and getting a handle on it.

People who have small(!) files which demonstrate this problem are
invited to e-mail them to "be...@sundialservices.com" as ZIP-files.
Please describe the files and their contents.  Rest assured that all
data will be held strictly confidential -- and pay attention that you do
not inadvertantly post them to a newsgroup!  :-)

Delphi Developer
Wed, 18 Jun 1902 08:00:00 GMT

Re:TIP: BDE 2.52, Corrupted indexes, international characters, D1


Markku Nevalaine

Delphi Developer
Wed, 18 Jun 1902 08:00:00 GMT

Re:TIP: BDE 2.52, Corrupted indexes, international characters, D1


Quote
Markku Nevalainen wrote:

> The problem also appears _only_ with Paradox 4.0 version tables, with 5.0
> tables everything works fine.
>...
> Currently, there really seems not be any index corrupting problems even
> with international characters and D1 / BDE2.52, as long as you make sure
> the Paradox table version is 5.0.

It seems that I have to correct my earlier sayings. The corrupting problem
with international characters sets and BDE 2.52 is _not_ tied to Paradox
4.0 version, and using 5.0 version also can't correct it.

You can easily corrupt any 4.0 or 5.0 Paradox table that has international
characters as non-case sensitive, secondary index. Just drop the index,
and then re-create it:

  Parts.DeleteIndex('ByDescription');
  Parts.AddIndex('ByDescription', 'Description', [ixCaseInsensitive]);

That's it. The PARTS.DB index that was OK one second ago is now smashed
and gives 'corrupt index' error message.

The problem really seems to be tied to recreating indexes to tables
that already have data in with international characters. If you start
with empty table with indexes, you can add any international data you
want, and indexes and everything works fine.
The problems start only if you have to recreate the indexes. BDE AddIndex
somehow treats the index file differently that the usual BDE run time
indexing.

This is my _current_ opinion about this case. It may be it changes later,
if I still find something new.

Markku Nevalainen

Sundial Service

Delphi Developer
Wed, 18 Jun 1902 08:00:00 GMT

Re:TIP: BDE 2.52, Corrupted indexes, international characters, D1


Quote
Markku Nevalainen wrote:
> You can easily corrupt any 4.0 or 5.0 Paradox table that has international
> characters as non-case sensitive, secondary index. Just drop the index,
> and then re-create it:

>   Parts.DeleteIndex('ByDescription');
>   Parts.AddIndex('ByDescription', 'Description', [ixCaseInsensitive]);

> That's it. The PARTS.DB index that was OK one second ago is now smashed
> and gives 'corrupt index' error message.

> The problem really seems to be tied to recreating indexes to tables
> that already have data in with international characters. If you start
> with empty table with indexes, you can add any international data you
> want, and indexes and everything works fine.
> The problems start only if you have to recreate the indexes. BDE AddIndex
> somehow treats the index file differently that the usual BDE run time
> indexing.

> This is my _current_ opinion about this case. It may be it changes later,
> if I still find something new.

As you imply in this most excellent post, I believe that the root cause
of this problem is that there is some part of 16-bit BDE that is not
correctly internationalized.  It appears to be using ASCII-specific
comparison rather than employing the language-driver as it should be.
And yes, it does appear that other parts of the selfsame BDE work
correctly!

I am sincerely hoping that Borland/InPrise will NOT consider 16-bit BDE
as "a dead horse not worth fixing," but will recognize how heavily it
continues to be used.  I believe that the company has an obligation,
indeed, to update the product to correct the Y2K-related issues that
have surfaced, and I believe that this indexing issue should be
addressed as well.

No, for InPrise to furnish these updates:  (a) might not be palatable to
the 32-bit-ized programmers; and (b) might not produce any additional
revenue at all...  but nonetheless, "it's the right thing to do."  (And
soon.)  The impact to all of us developers who rely upon this software
-- and who are powerless to do more than to request the change -- is
considerable.

Markku Nevalaine

Delphi Developer
Wed, 18 Jun 1902 08:00:00 GMT

Re:TIP: BDE 2.52, Corrupted indexes, international characters, D1


Quote
Sundial Services wrote:

> I believe that the company has an obligation,
> indeed, to update the product to correct the Y2K-related issues that
> have surfaced,

Yuk! What's that Y2K-problem, in here, precisely?

Quote
>  "it's the right thing to do."  

Well, I can't say I would be against this:)  I wonder who is the right
person at Inprise to send these wishes.

Markku Nevalainen

Go to page: [1] [2]

Other Threads

1. Bug: BDE 2.52 and international characters

2. international bde 2.52 available?

3. BDE + Paradox-probs D1, BDE 2.52 and NETWORK

4. BDE 2.52 , german version for D1/D2 wanted

5. CREATE INDEX is case insensitive in BDE 2.52???

6. D1, MaskEdit and international characters

7. BDE 5.0 SQL on Paradox tables so slow in comparisont to BDE 2.52

8. BDE 2.52 and BDE 4.51 Concurrency Question

9. BDE And Interbase slow on international character sets

10. BDE IntrBase Searches Slow on international character sets