Board index » delphi » Example of inconsistent data state with OPF

Example of inconsistent data state with OPF


2005-11-01 03:33:19 PM
delphi248
Jim Cooper writes:
Quote
Oh no, two lines of code!!
Jim, I don't think you are reading my posts carefully enough. At first
you /disputed/ the need for two separate processes to achieve the
"update" with an OPF. When I clearly demonstrated to you of the
existence of such a situation, you begin to exclaim that two lines of
code are no big deal. :(
It is quite possible that you /really/ don't understand the /impact/ of
the two-step process of achieving the update.
Let me explain again, with some examples.
The best way to show this is to use XML representation of the classes
in the OPF scenario:
<BOClass1>
<DetailClass1 detailAttr="attrd3"/>
<DetailClass2 detailAttr="attr32"/>
<DetailClass3 detailAttr="attr23"/>
</BOClass1>
<BOClass2>
<DetailClass4 detailAttr="attrf4r"/>
<DetailClass5 detailAttr="attr3ed"/>
<DetailClass6 detailAttr="x"/>
</BOClass2>
So, this is your typical containment scenario (BOClass contains a list
of DetailClasses). Great.
Now suppose you want to associate DetailClass6 with BOClass1. In an
OPF scenario, you will have to achieve this with TWO distinct steps:
1. Remove DetailClass6 as a child node of BOClass2; then
2. Insert DetailClass6 as a child node of BOClass2;
If 1. succeeds but 2. fails, you have inconsistent data state requiring
explicit management to recover from! (The very idea we pay such huge
amounts of money for RDBM technology. <g>) This is how the data could
look in an inconsistent state:
<BOClass1>
<DetailClass1 detailAttr="attrd3"/>
<DetailClass2 detailAttr="attr32"/>
<DetailClass3 detailAttr="attr23"/>
</BOClass1>
<BOClass2>
<DetailClass4 detailAttr="attrf4r"/>
<DetailClass5 detailAttr="attr3ed"/>
<-- missing data!!!!
</BOClass2>
However, in a DataSet scenario, the following is the situation:
BOClass_DataSet:
---------------------
BOClass1|
BOClass1|
BOClass1|
BOClass2|
BOClass2|
BOClass2|
DetailClass_DataSet:
---------------------
DetailClass1|BOClass1|attrd3
DetailClass2|BOClass1|attr32
DetailClass3|BOClass1|attr23
DetailClass4|BOClass2|attrf4r
DetailClass5|BOClass2|attr3ed
DetailClass6|BOClass2|x <-- Update this only. (One step).
(The above BOClass_DataSet remains untouched.)
So, to recap: what have we done with the dataset scenario? We have
simply /borrowed/ the good work of sound data modeling techniques from
our database so that data consistency can be maintained in our
"process" tiers too. Fantastic!
Quote
>Clearly that there are two distinct steps involved here:

Sure, could be.
/Could/ be? What is this? Now you see it, now you don't? :)
Quote
>Now suppose that 1) succeeds and 2) fails? What happens to our
>collection of MasterBO's in memory?


Exactly what you would expect, I assume.
Yes, inconsistent data - the cardinal sin of data management!
Quote
>Isn't it /inconsitent/?


MIght be. It depends on the problem domain.
No, not "might" - it /is/.
Quote
Of course, you can always
trap an exception and do something if you want, the same way you
would in any other code where things didn;t go according to plan.
Trap an exception? Pray-tell, please demonstrate your idea of
recovering from such a state? No doubt, you will have to introduce
pseudo-transactional semantics or some sort of compensating transactions
in order to keep your in-memory state consistent!
An in-memory normalised dataset would avoid this completely, because it
would not entertain the OO ideas of containment or inheritance. Like
properly designed data models, datasets can take advantage of the exact
same modeling consistency to their advantage!
Quote
>That's the point.


What point?
You're joking, right? The point was to demonstrate how an OPF example
could result in inconsistent data in memory. We've proved it. /That/
is the point.
 
 

Re:Example of inconsistent data state with OPF

Quote
1. Remove DetailClass6 as a child node of BOClass2; then
2. Insert DetailClass6 as a child node of BOClass2;
Correction, that should be:
1. Remove DetailClass6 as a child node of BOClass2; then
2. Insert DetailClass6 as a child node of /BOClass1/.
(Sorry.)
 

Re:Example of inconsistent data state with OPF

Quote
If 1. succeeds but 2. fails, you have inconsistent data state requiring
explicit management to recover from!
OK, I see what you're getting at. The situation is worse than you imagine, then :-)
The exact same situation can arise in a database update though. Suppose instead
of your structure you have a link table instead (eg so that a given detail can
be associated with several masters). Then it is a two step update there too. In
**any** situation where there might be failure (regardless of whether it is OPF
code, database code, or any other sort of code) you have to deal with it
somehow. it is just a question of how you want to do that. And that is probably
the most complicated part of writing an OPF.
You are right that it is something you may need to deal with. (That particular
example does not actually arise much in practice unless you have the
many-to-many links though.) You are incorrect to think it has affected your
database though, because you haven't saved anything yet. The more likely issue
is that the two objects states does not reflect the state of the database. But
then that is always going to happen with an OPF even if things don't go wrong.
Change any property value and it is inconsistent with what's in the database
(this is also the case for TDataset-based BOs, right?)
The situation is actually more complicated than that,though.
In my first OPF, I modelled the database the way you did, with a master ID
column in the detail table. However, that turns out to mean that you can only
have one collection of type detail in the master class, because if you had more
than one you don't have any way to tell which collection a given detail should
go in. So if you want to be able to handle that situation you need to model that
differently in your database - each collection needs an ID too. That's
complication 1.
Then there is the issue of validation/errors. Suppose we are just dealing with
MasterBO1.Details.Remove(DetailBO);
MasterBO2.Details.Add(DetailBO);
Suppose one of these fails (it doesn't matter which for the moment). Do you need
to deal with it? It depends on the mode of failure, right? If there is some
system level error like running out of memory, we will just let it fail, yes?
Because we are then having very serious problems we cannot really deal with :-)
Most likely we are only going to be concerned if we are doing some sort of
validation, checking whether removing DetailBO from MasterBO1 is legal, and
adding it to MasterBO2 is legal.
So, if you do have Remove and/or Add doing validation, you may need to deal with
it. It might be perfectly legitimate to have one fail and the other succeed, but
in the general case something needs to be done. The question is what?
1. Do nothing
2. Ask the user
3. Manually rollback (ie explicitly write code to do the opposite thing)
4. Automatically rollback (ie implement transactions at the object level)
5. Something else I haven't thought of.
With an OPF there are actually more opportunities for validation, because the
code above has left code out - there has been no saving to the database yet, and
that could also fail. What do you do if the code above works, and the database
persistence fails? Supposing the database needs two updates too - should a
transaction be used and how do you know that, because there will be two calls to
save?
I guess essentially we are dealing with the concept of transactions here. There
are a number of techniques that can be used, but if you want them at the object
level there will be more code somewhere.
In my first OPF based app (actually set of apps) in turned out we did not need
them, as it happens. But I accept that it is not the general case.
Will this also be a problem in TDataset-based BOs? Well, probably yes, it will.
If you have one TDataset-based BO for MasterBO1, and another for MasterBO2, then
exactly the same issues arise. How are you modelling the two collections of
detailBOs? I think you have made an error your example, as you assume both
collections will be represented by one BO, but will that really be the case? As
soon as you have two objects you have the same problems. The way you solve them
might be different, but you **have** to have the problem.
If you have any two objects (even if they are not business objects and are never
going to be persisted anywhere), and you want them to be consistent with each
other, you have the same issue. it is not particularly an OPF issue, it is a
general coding issue. Imagine the situation where you have two listboxes, and
you can transfer strings from one to the other. You need to keep the two
consistent, yes? You don't want to lose a string.
Cheers,
Jim Cooper
__________________________________________
Jim Cooper XXXX@XXXXX.COM
Skype : jim.cooper
Tabdee Ltd www.tabdee.ltd.uk
TurboSync - Connecting Delphi to your Palm
__________________________________________
 

Re:Example of inconsistent data state with OPF

Jim Cooper writes:
Quote
OK, I see what you're getting at. The situation is worse than you
imagine, then :-)
You have no idea! <g>
Quote
The exact same situation can arise in a database update though.
You see, you are assuming that the TDataset scenario /necessarily/
involves direct connection to the database. That assumption is wrong.
I am talking about a Dataset in memory - a disconnected one that
operates in the same fashion as the OPF when it comes to "connectivity".
Quote
Suppose instead of your structure you have a link table instead (eg
so that a given detail can be associated with several masters). Then
it is a two step update there too.
No. Again, you assume wrong. In that situation, which involves a many-
to-many relationship, you would mimic the exact same normalised
structure of the data model directly in memory. So, this allows you
to manage the data in exactly the same way that the RDBMS is going to do
it for /permanent/ storage.
So, in both the situations (the OPF model and the disconnected TDataset
model) we have in-memory representation of the data. In both situations
we have a /separate/ update process to the database (via SQL or
whatever). In both situations we need to keep two things consistent:
1) The data we have in memory; and
2) The data that is stored in the database;
The /difference/ between the two approaches is the /way in which/ we
represent the data in memory. With the OPF approach, which involves
associating explicitly child objects to "parent" objects we have the
/added/ complexity of in-memory inconsistency. Jim, this is a
fundamental point - you have to understand this before we can continue
in a meaningful discussion. The issue is not about inconsistencies
between the in-memory representation of the data and the data as it sits
in the database; the issue is about data inconsistencies /directly/
within the in-memory representation of the data! (Big difference! :) )
Quote
In **any** situation where there
might be failure (regardless of whether it is OPF code, database
code, or any other sort of code) you have to deal with it somehow.
True, but that was not we are concerned about here. (See above.)
Quote
You are right that it is something you may need to deal with. (That
particular example does not actually arise much in practice unless
you have the many-to-many links though.)
The example we used involves a 1-to-many relationship. That sort of
situation rears its head /everyday/!
Quote
You are incorrect to think
it has affected your database though,
I never said that. <g>
Quote
because you haven't saved
anything yet. The more likely issue is that the two objects states
does not reflect the state of the database.
I have explained this - this is not the issue. The issue is of
inconsistent data state in memory!
Quote
But then that is always
going to happen with an OPF even if things don't go wrong. Change any
property value and it is inconsistent with what's in the database
(this is also the case for TDataset-based BOs, right?)
(Hence, not relevant.)
Quote
The situation is actually more complicated than that,though.
Agreed! :-D
Quote
In my first OPF, I modelled the database the way you did, with a
master ID column in the detail table. However, that turns out to mean
that you can only have one collection of type detail in the master
class, because if you had more than one you don't have any way to
tell which collection a given detail should go in. So if you want to
be able to handle that situation you need to model that differently
in your database - each collection needs an ID too. That's
complication 1.
If I understand you correctly, that "complication" is called
normalisation. Can you please post a DDL of your approach, I'd very
much like to see it.
Quote
Then there is the issue of validation/errors. Suppose we are just
dealing with
Validation can be handled easily with the event models of the
disconnected datasets. Easy peasy.
Quote
MasterBO1.Details.Remove(DetailBO); MasterBO2.Details.Add(DetailBO);

Suppose one of these fails (it doesn't matter which for the moment).
Do you need to deal with it? It depends on the mode of failure,
right? If there is some system level error like running out of
memory, we will just let it fail, yes? Because we are then having
very serious problems we cannot really deal with :-)
In a dataset scenario, if the update fails (say for reasons of
validation) the database is not even called into the picture, BUT! our
local in memory data representation is NOT inconsistent and no further
work is required to recover. In the OPF scenario, where you require the
two steps, the database is ALSO not called into action BUT the data in
memory is now /inconsistent/ . We have to do something about the first
action to bring it back to the original state. Or we have to flush the
memory and update it with fresh data from the database to overcome the
inconsistency.
Quote
Most likely we are only going to be concerned if we are doing some
sort of validation, checking whether removing DetailBO from MasterBO1
is legal, and adding it to MasterBO2 is legal.
Yes.
Quote
So, if you do have Remove and/or Add doing validation, you may need
to deal with it. It might be perfectly legitimate to have one fail
and the other succeed, but in the general case something needs to be
done. The question is what?
Perfect. With a normalised scenario, we don't need these extra
complications to maintain data consistency. (Interestingly, a badly
designed data model also creates the need to manage inconsistencies in
the application logic. The same is happening here.)
Quote
1. Do nothing
Terrible solution. Inconsistent data in your mission critical
applications is a failure of the highest magnitude. Doing nothing is
the worst option.
Quote
2. Ask the user
What will the user say? This has nothing to do with the user. it is an
/internal/ data consistency issue.
Quote
3. Manually rollback (ie explicitly
write code to do the opposite thing)
4. Automatically rollback (ie
implement transactions at the object level)
Aah, now we are talking. We can not rely on automatic transactions (ie.
depending on the database technology) simply because nothing is
committed to the database yet. We /have/ to wire in our own home-grown
transactional system. Do you have any idea how not-so-trivial that is?
<g>
Quote
5. Something else I
haven't thought of.
(Read my mind.) :-D
Quote
With an OPF there are actually more opportunities for validation,
because the code above has left code out - there has been no saving
to the database yet, and that could also fail.
I think I have discussed this sufficiently above.
Quote
What do you do if the
code above works, and the database persistence fails? Supposing the
database needs two updates too - should a transaction be used and how
do you know that, because there will be two calls to save?
This is an altogether different matter. In this scenario you /need/
database transactions. In our earlier example where we were simply
updating ONE value, the issue of database transactions were not even
considered. (Please don't complicate the issue, we are dealing with
a simple example, let's flesh that one out first before we start
tackling much more complicated scenarios. I assure you the issues
against the need to model OO classes for your database tables with
containment and hierarchies become even more compelling.)
Quote
I guess essentially we are dealing with the concept of transactions
here. There are a number of techniques that can be used, but if you
want them at the object level there will be more code somewhere.
Not relevant (yet). See above.
Quote
Will this also be a problem in TDataset-based BOs? Well, probably
yes, it will. If you have one TDataset-based BO for MasterBO1, and
another for MasterBO2, then exactly the same issues arise. How are
you modelling the two collections of detailBOs? I think you have made
an error your example,
:-D /ja-well-no-fine/
Quote
as you assume both collections will be
represented by one BO, but will that really be the case? As soon as
you have two objects you have the same problems. The way you solve
them might be different, but you **have** to have the problem.
No, you are conflating the issues here. The issue is about the
different approaches:
1) One (OPF) uses explicit object instances to do the association;
2) The second (TDataset) uses /real/ references to do the association;
In the former approach, you have two explicit realities:
i) The association of MasterObject2 to DetailObject6;
ii) The absence of the association of DetailObject6 to MasterObject1;
As daft as that second statement is, the reality is that we have to
contend with it when there is an UPDATE involved - or more particularly
when an update could fail as a result of our /contrived/ two-step
process!
 

Re:Example of inconsistent data state with OPF

Quote
You see, you are assuming that <snip>
No I am not.
Quote
The /difference/ between the two approaches is the /way in which/ we
represent the data in memory.
While that is indeed different, it does not fundamentally alter the problem. If
you are running your TDatasets disconnected then you also have data in memory
and data in the database. If you have two of your type of BOs then the in-memory
data can also become inconsistent.
Quote
The example we used involves a 1-to-many relationship. That sort of
situation rears its head /everyday/!
Well, my experience varies from yours considerably. One-to-many relationships
are common. Changing the master of a given detail item is not, IME. But it
doesn't matter really, it is the principle of the thing.
Quote
If I understand you correctly, that "complication" is called
normalisation.
Not really. My point there was that there are different ways to model even
simple looking collections of BOs when it comes to persistence. Since you are
only concerned about in-memory stuff, this is a red herring, so let's ignore it.
Let's instead look at the bigger picture of keeping in-memory data consistent.
Quote
Validation can be handled easily with the event models of the
disconnected datasets. Easy peasy.
Aha, back to the evils of putting business logic in event handlers eh? :-) In
any case, any time you update two BOs and want either both updates to work or
nothing to happen, it is no simpler using events than try..except blocks.
Quote
In the OPF scenario, where you require the two steps
**Any** time you want two method calls to work you have this problem. It is not
an OPF-specific problem. Your type of BOs will also suffer from it.
Quote
Perfect. With a normalised scenario, we don't need these extra
complications to maintain data consistency.
Only in the case you assume your BOs can do this sort of updating in one step
and mine can't. If yours can not then you are in exactly the same situation.
In your example code, you have one BO holding all the master records, and one
holding all the detail records. But what if I am allowed to do this (I've
mentioned this already) :
DetailBO.Owner := MasterBO2;
There is nothing stopping me doing that. it is effectively the same as you are
suggesting doing with your BOs, yes?
So this particular issue is a red herring. There *is* an issue with keeping
in-memory data consistent, but your type of BO has no advantages in doing that
that I can see. The issues are in fact identical for both.
Quote
Terrible solution. Inconsistent data in your mission critical
applications is a failure of the highest magnitude. Doing nothing is
the worst option.
Terrible assumption. It may be perfectly valid to have the remove work and the
add fail. You cannot possibly know without knowing the problem domain. It is
**usually** not a good idea, not **always**.
Quote
What will the user say? This has nothing to do with the user. it is an
/internal/ data consistency issue.
Again, you are making assumptions about what is allowed and what is not. It
**depends**. It may indeed require a human level decision.
Quote
Do you have any idea how not-so-trivial that is? <g>
Seeing as I told you it was difficult, and I have likely spent a lot more time
thinking about this in the context of an OPF than you, then yes I do.
But, and it is a big but, if you want object level transactions, then as you say,
your BOs are of no help either.
Quote
This is an altogether different matter. In this scenario you /need/
database transactions.
Agreed. I was trying to point out that there is a potential need for handling
two types of transactions. One at the database level, and one at the object
level. When in an OPF there is the additional need to keep in-memory objects in
sync with the database, which *is* an extra thing an OPF needs code for, that
your approach probably makes easier.
Quote
As daft as that second statement is, the reality is that we have to
contend with it when there is an UPDATE involved - or more particularly
when an update could fail as a result of our /contrived/ two-step
process!
Like I keep saying to you **any** time there is a two step update process
involved we have this problem. You have artificially constructed a situation
where my BOs have to do two steps and yours only has to do one. But your example
is covering up the fact that you will probably also have to do 2 steps in the
same situation. The two approaches are not significantly different in this
respect. To deal with failure of multi-step updates of in-memory objects
requires extra work somewhere.
Cheers,
Jim Cooper
__________________________________________
Jim Cooper XXXX@XXXXX.COM
Skype : jim.cooper
Tabdee Ltd www.tabdee.ltd.uk
TurboSync - Connecting Delphi to your Palm
__________________________________________
 

Re:Example of inconsistent data state with OPF

Abdullah Kauchali writes:
Quote
Trap an exception? Pray-tell, please demonstrate your idea of
recovering from such a state? No doubt, you will have to introduce
pseudo-transactional semantics or some sort of compensating
transactions in order to keep your in-memory state consistent!
How does using BOs complicate this? In *any* application that deals with a
database, there is the possibility of data in memory becoming inconsistent
with what's in the database due to some kind of application error. Dealing
with such issues is far older than the concept of business objects or even
OO in general. Trying to assert that the use of BOs somehow makes this more
complicated or more error prone is a red-herring.
The example you give is no more likely to happen than similar code that
subtracts a value from a row in one dataset and adds the value to a row in
another dataset (e.g. subtracting from inventory and adding to a filled
orderline). What do you do if the inventory dataset has been updated and the
orderline dataset suffers an error? How is it any different, any safer, any
easier to manage? By you logic, memory datasets must *equally* be banned and
all code must be written purely against a live database, anything else
introduces another possible point of failure that must be handled outside an
RDBMS transaction. Avoiding BOs in favour of procedural code does not in any
way avoid or solve such an issue.
--
Wayne Niddery - Logic Fundamentals, Inc. (www.logicfundamentals.com)
RADBooks: www.logicfundamentals.com/RADBooks.html
"The two most abundant elements in the universe are hydrogen and
stupidity." - Harlan Ellison
 

Re:Example of inconsistent data state with OPF

Abdullah Kauchali writes:
Quote

You see, you are assuming that the TDataset scenario /necessarily/
involves direct connection to the database. That assumption is wrong.
I am talking about a Dataset in memory - a disconnected one that
operates in the same fashion as the OPF when it comes to
"connectivity".
Which, as soon as you have more than a single dataset, has the same
potential "transaction" problem of one getting updated and the other
failing, you've solved nothing here.
Quote
No. Again, you assume wrong. In that situation, which involves a
many- to-many relationship, you would mimic the exact same normalised
structure of the data model directly in memory. So, this allows you
to manage the data in exactly the same way that the RDBMS is going to
do it for /permanent/ storage.
One of your claims is that BO's cannot handle transactions (or that if they
do they are reinventing the RDBMS wheel), but memory datasets also do not
have transaction support so that argument fails.
Quote
the issue is about data inconsistencies
/directly/ within the in-memory representation of the data! (Big
difference! :) )
Red-herring since datasets have the same potential problem.
--
Wayne Niddery - Logic Fundamentals, Inc. (www.logicfundamentals.com)
RADBooks: www.logicfundamentals.com/RADBooks.html
"In a tornado, even turkeys can fly." - unknown
 

Re:Example of inconsistent data state with OPF

"Wayne Niddery [TeamB]" <XXXX@XXXXX.COM>, haber iletisinde žunlar?
yazd?XXXX@XXXXX.COM...
Quote

One of your claims is that BO's cannot handle transactions (or that if
they do they are reinventing the RDBMS wheel), but memory datasets also do
not have transaction support so that argument fails.

kbmMemTable supports transactions against its data (in memory) , and this
feature is quite useful.
Gokhan Ersumer