Board index » off-topic » DBase File size limitation?

DBase File size limitation?


2005-05-24 03:49:05 PM
off-topic4
Hi,
I have an old application that stores files and scanned images for
document archival purposes. Since I developed it a few years back the
application usees BDE and Dbase files.
One of my customers have reached a file size of about 2GB, and is
having problems with a table being corrupted. I restored it deleting
invalid records, but after a few days the problem reappeared.
The table is made of 3 fields, one of which is a blob field holding the
binary data for the stored file, so its structure is really simple.
Any ideas?
Thanks.
Fabio Dalle Ave
 
 

Re:DBase File size limitation?

In article <4292dc71$ XXXX@XXXXX.COM >,
XXXX@XXXXX.COM says...
Quote
Any ideas?
could be various reasons. I doubt that 2 GB in size alone would make up
a problem.
Which level do the tables have? 4,5,7?
'a few days later' does not help so much to track it down.
More important is a statement like 'after inserting/updating/ about
xxxxx records'.
'2GB size' also is missing one important information: how are the sizes
in the files? (you will most likely have a *.dbf, *.mdx and a *.dbt).
general hints:
with BLOBs in dBase make sure that you have a primary index on a key
field. If you don't have, make a copy of the table before you are
touching this option.
if a lot of updating and deleting takes place in tables with BLOb-
fields, (inserting is not a problem) it is always a good idea to have
some automatic housekeeping-tool running (to PACK the tables).
--
ciao,
Andr?
 

Re:DBase File size limitation?

A few clarifications:
The database has about 6350 records in it, with the space being taken
up I'm not really sure how to check table levels... I can't
restructure the table because I get an "Insufficient disk space" when I
try it (but I have plenty of space).
The problem occurs when inserting data in the table, and when it
happens it is not possibile to add any more data to it. It's still
possibile to view existing records, and any insert/update operation
ends up with a "Read failure on file..." error message.
*Lysander* wrote:
Quote
In article <4292dc71$ XXXX@XXXXX.COM >,
XXXX@XXXXX.COM says...

>Any ideas?
could be various reasons. I doubt that 2 GB in size alone would make
up a problem.

Which level do the tables have? 4,5,7?

'a few days later' does not help so much to track it down.
More important is a statement like 'after inserting/updating/ about
xxxxx records'.

'2GB size' also is missing one important information: how are the
sizes in the files? (you will most likely have a *.dbf, *.mdx and a
*.dbt).

general hints:
with BLOBs in dBase make sure that you have a primary index on a key
field. If you don't have, make a copy of the table before you are
touching this option.

if a lot of updating and deleting takes place in tables with BLOb-
fields, (inserting is not a problem) it is always a good idea to have
some automatic housekeeping-tool running (to PACK the tables).
 

{smallsort}

Re:DBase File size limitation?

In article <4292f064$ XXXX@XXXXX.COM >,
XXXX@XXXXX.COM says...
Quote
The database has about 6350 records in it,
That is definitely not too much, depending of course on the size of the
binary-field.
Tell your customer to look for a file with the same name as your table,
but with suffix "dbt". This file holds the BLOb-Data for your table
("dbf"), while indexes are kept in "mdx". Size of dbt will be important.
Quote
up I'm not really sure how to check table levels...
if the table is created on the site of your customer, it will most
likely have the level which is set as default in the BDE.
In BDEAdmin.exe, check for "configuration/drivers/native/dbase: LEVEL"
Quote
restructure the table because I get an "Insufficient disk space" when
check also if you are victim of the 4GB-Bug in BDE, search this NG for
topic "4GB"
Quote
The problem occurs when inserting data in the table,
So, records and BLObs only get appended, not deleted, and most of all:
not changed?
When changing a BLOb, the old value does not "vanish". The new value is
usually added at the end of the file; the header (for the main record)
will get a new pointer to the new location. The garbage is cleaned ONLY
by copying and reappending the records, or by using PACK.
Deleting will not help, because dBase-tables have a deleted flag for
every record and you can "recover" deleted records until the next
housekeeping-run.
How many users do you have on that "problem"-site?
Network-locking-conflicts can lead to so-called "Bloats" of BLOb-fields,
where size of dbt-files is increasing overproportionally.
--
ciao,
Andr?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Deutsche dBase-Konferenz dBKon 2005 ::: 11. bis 13. November 2005
Deutschsprachige Konferenz zu dBase und anderen Datenbanken
Info: www.dbase-konferenz.de
 

Re:DBase File size limitation?

Quote

Tell your customer to look for a file with the same name as your
table, but with suffix "dbt". This file holds the BLOb-Data for your
table ("dbf"), while indexes are kept in "mdx". Size of dbt will be
important.
The DBT file holds the 2GB worth of data, while the other 2 files (.DBF
and .MDX) are less that 500K combined. The data mainly gets appended:
I would say that less than 1% of the operations are updates and deletes
(only when a document is archived by mistake).
This customer has just 3 users using the software, but has a lot of
documents being inserted/scanned (i have no idea on how many a day
though).
Does anyone know where I can find a free table analyzer / fixer (much
as DTUtil32.exe for paradox tables) ?
Thanks for any clue. ;)
*Lysander* wrote:
Quote

>up I'm not really sure how to check table levels...
if the table is created on the site of your customer, it will most
likely have the level which is set as default in the BDE.
In BDEAdmin.exe, check for "configuration/drivers/native/dbase: LEVEL"

How can I eventually change the table level?? And what good would that
do? I already built the 4GB bug fix into the application. Is Database
Desktop that throws the error when restructuring the table
Quote
>restructure the table because I get an "Insufficient disk space"
>when
check also if you are victim of the 4GB-Bug in BDE, search this NG
for topic "4GB"

>The problem occurs when inserting data in the table,
So, records and BLObs only get appended, not deleted, and most of
all: not changed?
When changing a BLOb, the old value does not "vanish". The new value
is usually added at the end of the file; the header (for the main
record) will get a new pointer to the new location. The garbage is
cleaned ONLY by copying and reappending the records, or by using PACK.
Deleting will not help, because dBase-tables have a deleted flag for
every record and you can "recover" deleted records until the next
housekeeping-run.

How many users do you have on that "problem"-site?
Network-locking-conflicts can lead to so-called "Bloats" of
BLOb-fields, where size of dbt-files is increasing overproportionally.
 

Re:DBase File size limitation?

In article <42931b8d$ XXXX@XXXXX.COM >,
XXXX@XXXXX.COM says...
Quote
Does anyone know where I can find a free table analyzer / fixer (much
as DTUtil32.exe for paradox tables) ?
I am not using any such tool, because I have the dBase-runtime on all
sites where I have dBase-tables also. The dBase runtime does have a
built-in clean-up utility for this.
I have heard about a tool "fixmemo" which is meant for memos, but should
also handle other types of BLOb-corruption.
Try googling for "fixmemo".
If the reason for the problems is not the 4GB-Bug then you must indeed
install some house-keeping routine.
--
ciao,
Andr?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Deutsche dBase-Konferenz dBKon 2005 ::: 11. bis 13. November 2005
Deutschsprachige Konferenz zu dBase und anderen Datenbanken
Info: www.dbase-konferenz.de
 

Re:DBase File size limitation?

On 24 May 2005 05:18:21 -0700, "Fabio Dalle Ave"
< XXXX@XXXXX.COM >wrote:
Quote
>
>Tell your customer to look for a file with the same name as your
>table, but with suffix "dbt". This file holds the BLOb-Data for your
>table ("dbf"), while indexes are kept in "mdx". Size of dbt will be
>important.

The DBT file holds the 2GB worth of data, while the other 2 files (.DBF
and .MDX) are less that 500K combined. The data mainly gets appended:
I would say that less than 1% of the operations are updates and deletes
(only when a document is archived by mistake).
I also have problems with a client when the .dbt table gets close to
2Gb (dBase 3). The memo fields get scrambled and do not point to the
original record. I now keep an eye on table size and when approaching
the problem area, archive to table 2 or 3 or whatever, zap the
original and start again. Do not know if this is possible in your
situation but it works for me and may be of some help to you.
---------
Skil-Phil
 

Re:DBase File size limitation?

I already sent my proposal to upgrade to a new version of my software
which can use both SQL Server or Firebird, using DBExpress, which
should avoid the problems. I'd rather sell the new version that do a
quick hack to overcome a limitation.
However I thought about splitting the table every year, which would
also make backups a lot less painful.
Thanks for the tip!
Phil Hansen wrote:
Quote
On 24 May 2005 05:18:21 -0700, "Fabio Dalle Ave"
< XXXX@XXXXX.COM >wrote:

>>
>>Tell your customer to look for a file with the same name as your
>>table, but with suffix "dbt". This file holds the BLOb-Data for
your>>table ("dbf"), while indexes are kept in "mdx". Size of dbt
will be>>important.
>
>The DBT file holds the 2GB worth of data, while the other 2 files
>(.DBF and .MDX) are less that 500K combined. The data mainly gets
>appended: I would say that less than 1% of the operations are
>updates and deletes (only when a document is archived by mistake).

I also have problems with a client when the .dbt table gets close to
2Gb (dBase 3). The memo fields get scrambled and do not point to the
original record. I now keep an eye on table size and when approaching
the problem area, archive to table 2 or 3 or whatever, zap the
original and start again. Do not know if this is possible in your
situation but it works for me and may be of some help to you.

---------
Skil-Phil