Board index » delphi » Loading in large, and I mean large, files

bl...@torfree.net (Arlo Smith)

Delphi Developer
Tue, 29 May 2001 03:00:00 GMT

Loading in large, and I mean large, files

Hello. I'm attempting to write a PGNreader and you really don't need to
know more about it other than PGN files are Chess Game text files
routinely over 800k, in PGN format. Now, an existing PGNreader I have can
load in a 800k game file easily ... the programmer can't be using what I'm
using. And what am I using? Well, StringListname.loadfromfile(), for one,
which basically can't handle the 800k file on my computer (I also tried
AssignFile, Reset, Readln() and it's no better). Seems to load in a 300k
file okay, but 800k makes the HD grind away getting nowhere. What is this
OTHER programmer doing, er, using, since his program works fine on my
computer?

Definitely he loads the whole file--I moved the file while the program was
still running and his program still continues to access the games. There
must be more efficient ways of loading in these files, ways I haven't a
clue about!

--
     I've seen your repulsion and it looks real good on you  
     I've seen your repulsion and I know what it can do      

 
Sundial Service

Delphi Developer
Tue, 29 May 2001 03:00:00 GMT

Re:Loading in large, and I mean large, files


Quote
Arlo Smith wrote:

> Hello. I'm attempting to write a PGNreader and you really don't need to
> know more about it other than PGN files are Chess Game text files
> routinely over 800k, in PGN format. Now, an existing PGNreader I have can
> load in a 800k game file easily ... the programmer can't be using what I'm
> using. And what am I using? Well, StringListname.loadfromfile(), for one,
> which basically can't handle the 800k file on my computer (I also tried
> AssignFile, Reset, Readln() and it's no better). Seems to load in a 300k
> file okay, but 800k makes the HD grind away getting nowhere. What is this
> OTHER programmer doing, er, using, since his program works fine on my
> computer?

> Definitely he loads the whole file--I moved the file while the program was
> still running and his program still continues to access the games. There
> must be more efficient ways of loading in these files, ways I haven't a
> clue about!

Well, Arlo, think about how TStringList might be implemented.  In D1,
for example, it's a block of text allocated in a single chunk along with
a pointer-matrix to find the elements; all of which is limited to 32K
memory and a lot less elements.  It's fast and convenient but certainly
not the only way to do it.

ReadLn and their cousins expect to work with text files, delimited by
CR/LFs, and they are mainly concentrating on "being compatible with
Standard Pascal."

You can use, for example, a TFileStream to open the file, position
anywhere in it, and read bytes.  In 32-bit mode you could snarf the
whole 800K into memory in one slurp (although it's usually better to
store things like that in smaller, discontiguous blocks... the memory
manager is used to dealing with lots of small things rather than a few
big things, and of course in 16-bit it's a requirement).

Once you have read your file into memory, in one or more pieces, you can
use TLists (not TStringLists) to store a list of pointers to the start
of individual "items of interest" whereever you have stored them in
memory.  You can also use trees, linked-lists and other structures.

Mark Shapi

Delphi Developer
Wed, 30 May 2001 03:00:00 GMT

Re:Loading in large, and I mean large, files


When loading large files into memory, I have found TMemoryStream to be
consistently faster than TFileStream.
jph

Delphi Developer
Sun, 03 Jun 2001 03:00:00 GMT

Re:Loading in large, and I mean large, files


hmmm.... i posted the following for someone else a few days ago in another
thread...

I don't know for sure if it applies, but I was JUST on the web and ran
across a unit by Primoz Gabrijelcic called TGpHugeFile 2.1, which he
describes as follows:

Encapsulation of Windows file-handling routines which allows work with >2GB
files.
Included is support for FILE_FLAG_NO_BUFFERING access and buffering for
sequentially accessed files. Tested with D3 and D4.

Sounds like it might help you, and it is full-source freeware.  Check it
out!

http://www.eccentrica.org/gabr/gp/index.htm

Quote
Arlo Smith wrote:

> Hello. I'm attempting to write a PGNreader and you really don't need to
> know more about it other than PGN files are Chess Game text files
> routinely over 800k, in PGN format. Now, an existing PGNreader I have can
> load in a 800k game file easily ... the programmer can't be using what I'm
> using. And what am I using? Well, StringListname.loadfromfile(), for one,
> which basically can't handle the 800k file on my computer (I also tried
> AssignFile, Reset, Readln() and it's no better). Seems to load in a 300k
> file okay, but 800k makes the HD grind away getting nowhere. What is this
> OTHER programmer doing, er, using, since his program works fine on my
> computer?

> Definitely he loads the whole file--I moved the file while the program was
> still running and his program still continues to access the games. There
> must be more efficient ways of loading in these files, ways I haven't a
> clue about!

> --
>      I've seen your repulsion and it looks real good on you
>      I've seen your repulsion and I know what it can do

Primo? Gabrijelèi

Delphi Developer
Sun, 03 Jun 2001 03:00:00 GMT

Re:Loading in large, and I mean large, files


Quote
jph wrote in message <367743DE.7613E...@InternetAddress.com>...
>I don't know for sure if it applies, but I was JUST on the web and ran
>across a unit by Primoz Gabrijelcic called TGpHugeFile 2.1, which he
>describes as follows:

Please, don't use it! I'll upload version 2.2 today. It should fix one {*word*193}
Win32 API related bug!

Primoz

Other Threads

1. How to load/store large PCX/BMP files??

2. Loading / Sorting large data files.

3. Largest Image file possible to load

4. load a PART of a large file

5. loading LARGE bitmaps, stitching them, then saving result in 1 file w/o tbitmap

6. Load large file in a DBGrid with D6

7. How to display large array of large images ...

8. Would Large Delphi Application = large executable ?

9. large databasis of 30 Gb or larger

10. Large integer field type won't work with really larger integers