Board index » delphi » Reading binary files using TFileStream

Reading binary files using TFileStream


2005-12-01 03:24:55 AM
delphi138
Using D7, I am trying to read a binary file that I (sort of)
know the structure to using TFileStream. The structure is
as follows:
{struct file {
char hdr[64];
int nruns;
local int i = 0;
for (i=0; i<nruns;i++) {
struct run runs;
}
}
struct run {
int nlaps;
struct lap laps[nlaps];
char dunno[7];
short ntracks;
char dunno2[3];
struct trackpoint tp[ntracks];
};
struct trackpoint {
unsigned int x;
unsigned int y;
double alt;
char char1;
time_t date;
char zeros[19];
};
struct lap {
time_t start;
int dur;
struct posns {
short s1;
unsigned int px;
unsigned int py;
char zeroes[8];
} pos[2];
float len;
int cal;
};
So, what I am trying to do is read the first 64 bytes of the
file and then read the next four to determine the number of
runs. I will then read the four to get the number of laps for
the first run and so on and so forth until the data for the
first run is logged. I will then loop and do it again.
The problem I am having is actually reading the data.
Whenever I try to access the header information, I get an
access violation (see snippet below). (I haven't even tried
to get to the number of runs.)
var
binStream: TFileStream;
binString: string;
iHeadLength : integer;
sHeader : Word;
begin
try
//stream from the binary file
binStream := TFileStream.Create('c:\logbook.bin',
fmOpenRead or fmShareDenyNone);
try
binStream.Position := 0;
binStream.ReadBuffer(sHeader, 64);
ShowMessage(PChar(sHeader));
finally
binStream.Free;
end;
except
on e : Exception do begin
ShowMessage(e.Message);
end;
end;
end;
I've searched high and low for examples of how to do
something like this, but haven't found anything that
correlates to what I am trying to do. Any suggestions?
 
 

Re:Reading binary files using TFileStream

Ronaldo Souza <XXXX@XXXXX.COM>wrote in news:XXXX@XXXXX.COM:
Quote
You are trying to read 64 bytes into a word var (2 bytes). Create a 64
byte char buffer and read hdr into it:

Thanks. I am now able to read the header info and get to the number of runs
and number of laps for the first run using:
var
...
iRunCtr : integer;
iLapCtr : integer;
iCtr : integer
try
...
binStream.Read(iRunCtr, SizeOf(integer));
for iCtr = 1 to iRunCtr do begin
binStream.Read(iLapCtr, SizeOf(integer));
end;
...
Now I am stuck in trying to figure the rest of the information for the
indiviaual lap info (e.g., the start time, duration and trackpoints).
Quote
Another potential pitfall: do you know if the file was created using
packed records?
I have no idea. I don't have access to the developer who wrote the file.
Is there a way to tell via the header information?
 

Re:Reading binary files using TFileStream

SamFelis writes:
Quote
var
sHeader : Word;
[SNIP]
binStream.ReadBuffer(sHeader, 64);
You are trying to read 64 bytes into a word var (2 bytes). Create a 64
byte char buffer and read hdr into it:
var
sHeader : array[0..63] of char;
. . .
binStream.ReadBuffer(sHeader, 64);
. . .
Another potential pitfall: do you know if the file was created using
packed records?
HTH,
Ronaldo
 

Re:Reading binary files using TFileStream

SamFelis writes:
Quote
Thanks. I am now able to read the header info and get to the number of runs
and number of laps for the first run using:

var
...
iRunCtr : integer;
iLapCtr : integer;
iCtr : integer
try
...

binStream.Read(iRunCtr, SizeOf(integer));
for iCtr = 1 to iRunCtr do begin
binStream.Read(iLapCtr, SizeOf(integer));
end;

...

Now I am stuck in trying to figure the rest of the information for the
indiviaual lap info (e.g., the start time, duration and trackpoints).
You have to translate the C structs to Delphi records:
type
TPosns = packed record //"posns" struct, embedded in "lap"
s1 : SmallInt;
px : LongWord;
py : LongWord;
zeroes : array[0..7] of Char;
end;
TLap = packed record //"lap" struct
start : LongWord; //time_t: see tip #1
dur : Integer;
pos : array[0..1] of TPosns;
len : Single;
cal : Integer;
end;
. . .
After the translation, you will have a TRun type, so your code should look
something like:
Quote
var
...
iRunCtr : integer;
iCtr : integer
ARun : TRun;
Quote
try
...

read header
Quote
binStream.Read(iRunCtr, SizeOf(integer));
for iCtr = 1 to iRunCtr do begin
binStream.Read(ARun, SizeOf(ARun));
process run info
Quote
end;

...
Here are some tips for the conversion:
1) time_t is an unsigned long integer which holds the number of seconds
elapsed since 01/01/1970. If all you are interested in is the
difference between 2 time_t vars it is ok, but if you need the date &
time values, search www.tamaracka.com for "delphi time_t to DateTime"
2) Type conversions:
---------------------------------------------------------
Windows API Delphi C VB
---------------------------------------------------------
SHORT SmallInt short Integer
LONG Integer int Long
DWORD DWORD ulong Long
Single float Single
Double double Double
LPSTR/LPCSTR PChar char* ByVal S As String
<type>* Var X:<type><type>* Byref X As <type>
BOOL LongBool int Long
---------------------------------------------------------
3) Are you sure about these declarations?
struct run {
int nlaps;
struct lap laps[nlaps]; <<<<<<<<<<<
char dunno[7];
short ntracks;
char dunno2[3];
struct trackpoint tp[ntracks]; <<<<<<<<<<<
};
Quote
>Another potential pitfall: do you know if the file was created using
>packed records?

I have no idea. I don't have access to the developer who wrote the file.
Is there a way to tell via the header information?
Nope. Let's assume he/she knew what they were doing and created the file
with packed structs. For more info, check the help for "packed" and "$A".
HTH,
Ronaldo
 

Re:Reading binary files using TFileStream

Thanks for the initial translation (as well as the translation table
between C and Delphi variable types). This has started me in the right
direction and I have started getting valid data out of the file...well,
sort of.
The one thing that is causing problems is the type of the Duration value.
I haven't been able to retrieve a valid (as in recognizable) value yet
with integer or short int. I am going to keep trying with others.
Quote
Here are some tips for the conversion:
1) time_t is an unsigned long integer which holds the number of
seconds
elapsed since 01/01/1970. If all you are interested in is the
difference between 2 time_t vars it is ok, but if you need the date
& time values, search www.tamaracka.com for "delphi time_t to
DateTime"
Thanks for the explanation. With that I was able to get the date/time
of the run by dividing the start value by 86400 (the number of seconds
in a day) and then adding it to 25569 (the value of 01/01/1970
00:00:00.00).
Quote
3) Are you sure about these declarations?
struct run {
int nlaps;
struct lap laps[nlaps]; <<<<<<<<<<<
char dunno[7];
short ntracks;
char dunno2[3];
struct trackpoint tp[ntracks]; <<<<<<<<<<<
};
I received this structure from a message board that said this was the
structure of the file and that the poster was able to recreate the
output using this structure definition. Based on previous posts from
him and others, I would say the structure is legitimate. (The board is no
longer active, otherwise I'd go to the source.)
It's also why I am having problems getting the data out. I am not sure if
all the run data is at the beginning of the file, followed by the lap
data, followed by the trackpoint data or if the lap and trackpoint
information is sprinkled between the run data. The way I read the
structures, it appears all data associated with a run is written before
the next run's data is written.
Regardless, you've definitely helped me out here. At least now I can
get SOME data out of the file. I foresee a long weekend of trial and
error ahead of me. ;-)
-sam
 

Re:Reading binary files using TFileStream

SamFelis writes:
Quote
>3) Are you sure about these declarations?
>struct run {
>int nlaps;
>struct lap laps[nlaps]; <<<<<<<<<<<
>char dunno[7];
>short ntracks;
>char dunno2[3];
>struct trackpoint tp[ntracks]; <<<<<<<<<<<
>};

I received this structure from a message board that said this was the
structure of the file and that the poster was able to recreate the
output using this structure definition. Based on previous posts from
him and others, I would say the structure is legitimate. (The board is no
longer active, otherwise I'd go to the source.)
OK. I thought this was the definition of a struct, which didn't make
much sense.
Quote
It's also why I am having problems getting the data out. I am not sure if
all the run data is at the beginning of the file, followed by the lap
data, followed by the trackpoint data or if the lap and trackpoint
information is sprinkled between the run data. The way I read the
structures, it appears all data associated with a run is written before
the next run's data is written.
Try something like this:
type
TPosns = packed record
s1 : SmallInt;
px : LongWord;
py : LongWord;
zeroes : array[0..7] of Char;
end;
TLap = packed record
start : LongWord;
dur : Integer;
pos : array[0..1] of TPosns;
len : Single;
cal : Integer;
end;
TLapArray = packed array[0..(MaxInt div SizeOf(TLap))-1] of TLap;
TLapArrayPtr = ^TLapArray;
TTrackPoint = packed record
x : LongWord;
y : LongWord;
alt : Double;
char1 : char;
date : LongWord;
zeros : array[0..18] of char;
end;
TTrackPointArray = packed array[0..(MaxInt div
SizeOf(TTrackPoint))-1] of TTrackPoint;
TTrackPointArrayPtr = ^TTrackPointArray;
TRun = record //in memory use only
nlaps : Integer;
laps : TLapArrayPtr; //to be allocated with
nlaps*SizeOf(TLap)
dunno : array[0..6] of char;
ntracks : SmallInt;
dunno2 : array[0..2] of char;
tp : TTrackPointArrayPtr; // to be allocated with
nlaps*SizeOf(TTrackPoint)
end;
procedure TForm1.ButtonReadRunFile(Sender: TObject);
var
FS : TFileStream;
Hdr : array[0..63] of byte;
NRuns,i : integer;
Run : TRun;
begin
FS := TFileStream.Create('C:\MyRunFile.dat',fmOpenRead);
try
//Read Header
FS.Read(Hdr,SizeOf(Hdr));
//Read NRuns
FS.Read(NRuns,SizeOf(NRuns));
//You should do some checks on the integrity of NRuns:
//0 < NRuns <= MaxRuns - abort in case of an unexpected value
for i := 1 to NRuns do
begin
//Make sure var Run is empty
FillChar(Run,SizeOf(Run),#0);
//At this point we should be at the first run ->read NLaps
FS.Read(Run.NLaps,SizeOf(Run.NLaps));
//You should do some checks on the integrity of NLaps:
// 0 < NLaps <= MaxLaps - abort in case of an unexpected value
//Allocate memory for the laps
GetMem(Run.Laps,Run.NLaps*SizeOf(Run.Laps^[0]));
//Read data for all laps
FS.Read(Run.NLaps,Run.NLaps*SizeOf(Run.Laps^[0]));
//Read dunno
FS.Read(Run.Dunno,SizeOf(Run.Dunno));
//Read NTracks
FS.Read(Run.NTracks,SizeOf(Run.NTracks));
//You should do some checks on the integrity of NTracks:
//0 < NTracks <= MaxTracks - abort in case of an unexpected value
//Read dunno2
FS.Read(Run.Dunno2,SizeOf(Run.Dunno2));
//Allocate memory for the trackpoints
GetMem(Run.TP,Run.NTracks*SizeOf(Run.TP^[0]));
//Read data for all trackpoints
FS.Read(Run.TP,Run.NTracks*SizeOf(Run.TP^[0]));
{*******************}
{* PROCESS THE RUN *}
{*******************}
//Deallocate trackpoints info
FreeMem(Run.TP);
//Deallocate laps info
FreeMem(Run.Laps);
end; //for
finally
FS.Free
end;
end;
Quote
Regardless, you've definitely helped me out here.
Glad I could help... :-)
Quote
I foresee a long weekend of trial and error ahead of me. ;-)
Let's hope not! :-)
Good luck,
Ronaldo
 

Re:Reading binary files using TFileStream

Ronaldo --
Thanks for the detailed response. As it turns out, the structure I
provided above is inaccurate...to a certain point. The "dunno"
structures are notes and can be dynamic in nature. I was able to figure
out how to read the file based on your responses, searching the web and
a dash of trial and error tossed in for good measure.
I’m able to read the entire file except for one piece of data. The
value I’m trying to read from the file is the number of trackpoints.
According to the XML file I have (output by the program in question as a
way of exporting the data; I am trying to read directly from the binary
so that I don't have to export a file in order to read the data), there
should be 436 trackpoints whereas I keep getting 257. If I hardcode 436
as the number of trackpoints, I’m able to successfully read the entire
record and move to the next record (however, I then get stopped at the
second run b/c there are either more or less than 436 trackpoints).
During my search, I came across a Perl script that does what I am trying
to do in Delphi. In Perl, the instructions are to retrieve the
following (where $data is the record from the file that stores the
number of trackpoints, as well as extraneous data):
($u1,$ntp,$u2) = unpack("h12Sh6",$data);
From the perl.com help page on PACK:
h A hex string (low nybble first) (the number indicates
the number of bits to be read)
S An unsigned short value. This 'short' is _exactly_ 16
bits
Which I *believe* translates into something like:
- read the first four bits and toss them to the wayside, read the second
12 bits and save it to $u1 (making 2 bytes total)
- read the next 2 bytes and save them to $ntp
- read the next six bits and save them to $u2 (the last two bits of the
record are ignored)
However, this only gives me 5 bytes. I know I have to read in 8 bytes
at this location, as whenever I read more (or less), the file position
is incorrect downstream as evidenced by the data I retrieve. When I
read the file following this thought (e.g., getting the 5 bytes and
tossing away that last 3), the value written to $ntp is incorrect (it's
always a 1).
You wouldn't by chance have an intimate knowledge of Perl and could
suggest what should be done within Delphi, would you?
 

Re:Reading binary files using TFileStream

SamFelis writes:
Quote

($u1,$ntp,$u2) = unpack("h12Sh6",$data);

[SNIP]
You wouldn't by chance have an intimate knowledge of Perl and could
suggest what should be done within Delphi, would you?
Most definitely not... :-(
From the *very confusing* help on perl.com, I think "h" and "H" work
with nibbles (groups of 4 bytes). So you should read 6+2+3 bytes, which
goes against what you see in the file (8 bytes). To solve this
"mystery", I would play with the structure below to see how things go:
type
TNumTrackPtsRec = record
u1 : array[1..5] of byte;
ntp : word;
u2 : array[0..3] of byte;
end;
Play with the size of "u1" until you can read the correct number of
trackpoints in the first run. Play with the size of "u2" until you can
get aligned with the first byte of the second run, after you read the first.
Good luck,
Ronaldo
 

Re:Reading binary files using TFileStream

SamFelis writes:
Quote
Thanks for the detailed response. As it turns out, the structure I
provided above is inaccurate...to a certain point. The "dunno"
structures are notes and can be dynamic in nature. I was able to figure
out how to read the file based on your responses, searching the web and
a dash of trial and error tossed in for good measure.
Try adding the packed in uppercase to the typedefs and changing the
length of TRun.dunno to 0..5:
type
TPosns = packed record
s1 : SmallInt;
px : LongWord;
py : LongWord;
zeroes : ***PACKED*** array[0..7] of Char;
end;
TLap = packed record
start : LongWord;
dur : Integer;
pos : ***PACKED*** array[0..1] of TPosns;
len : Single;
cal : Integer;
end;
TLapArray = packed array[0..(MaxInt div SizeOf(TLap))-1] of TLap;
TLapArrayPtr = ^TLapArray;
TTrackPoint = packed record
x : LongWord;
y : LongWord;
alt : Double;
char1 : char;
date : LongWord;
zeros : ***PACKED*** array[0..18] of char;
end;
TTrackPointArray = packed array[0..(MaxInt div
SizeOf(TTrackPoint))-1] of TTrackPoint;
TTrackPointArrayPtr = ^TTrackPointArray;
TRun = record //in memory use only
nlaps : Integer;
laps : TLapArrayPtr; //to be allocated with
nlaps*SizeOf(TLap)
dunno : ***PACKED*** array[0..6] of char; <SHOULD BE 0..5?
ntracks : SmallInt;
dunno2 : ***PACKED*** array[0..2] of char;
tp : TTrackPointArrayPtr; // to be allocated with
nlaps*SizeOf(TTrackPoint)
end;
I suspect dunno handles "h12" and dunno2 handle "h6"... JUST A GUESS...
Good luck,
Ronaldo
 

Re:Reading binary files using TFileStream

"SamFelis" wrote
Quote
struct run {
int nlaps;
struct lap laps[nlaps];
char dunno[7];
short ntracks;
char dunno2[3];
struct trackpoint tp[ntracks];
};
Sam, This looks similar to a problem that I was helping
my nephew with last week. Are you in the track and
field business too? Regards, JohnH
 

Re:Reading binary files using TFileStream

"John Herbster" <herb-sci1_at_sbcglobal.net>wrote in news:43960fdd$1
@newsgroups.borland.com:
Quote
Sam, This looks similar to a problem that I was helping
my nephew with last week. Are you in the track and
field business too? Regards, JohnH
I do a fair bit of running, but I wouldn't call it a business. it is more
of a hobby to keep me sane. ;) As for the data structures, this is
*supposedly* what the GPS data binary looks like that I am trying to read
directly (but that is giving me fits as the above indicates).
 

Re:Reading binary files using TFileStream

Ronaldo Souza <XXXX@XXXXX.COM>wrote in
Quote
From the *very confusing* help on perl.com, I think "h" and "H" work
with nibbles (groups of 4 bytes). So you should read 6+2+3 bytes,
which goes against what you see in the file (8 bytes). To solve this
"mystery", I would play with the structure below to see how things go:
<snip>
Quote
Play with the size of "u1" until you can read the correct number of
trackpoints in the first run. Play with the size of "u2" until you can
get aligned with the first byte of the second run, after you read the
first.

You hit the nail on the head...well, almost. The u2 value isn't necessary
at all. Everything else lines up and I am now able to read the file in its
entirety.
Thanks so much for your assistance. Without it, I don't think I could've
solved this riddle.