[IRCServices] Closer looks at services databases?

[&quot]

This message has been sent to the coding mailing list. Please can ALL
dicussion take place there from now on.

[snip]
> Just for interest sakes (I know we have talked about this *MUCH* in the
> past), but just what exactly is ment by services "reading once,
> and writing
> many".  I totally like do not understand that interpretation of
> what exactly
> services is doing, and howcome this is the case in the scenario of a "data
> update" or "data retrieval".

This topic relates to Services writing its data to its database (i.e. the
flat binary files it currently uses.)

> If a nickname is registered, read not at all, write many
> If a nick is changed, read once, write perhaps more than which is
> required?
> If a memo is read, read ALLOT, write one bit?
> etc.

At startup, Services "reads" (loads) the databases, in their entirity, into
memory. From then on all data manipulation takes place in memory. Evey 30
mins or so that data is written back to the data files. This means that any
changes made to those files will be overwritten and will not be reflected in
Services. So, if the load/save routines were replaced with routines that
read/wrote data to/from a SQL database, rather than flat files, people would
still be unable to modify it from a another application. This whole thread
stems from the suggestion that making Services use a SQL database would
allow other applications to modify it. My response simply pointed out that
changes could not be made without making major modifications to Services so
that it re-read the data from the SQL server every time it needed to use it.

> Shouldn't there perhaps rather be looked into the ammount of information
> services writes to the databases?  I admit yes, I have not looked at the
> code seeing that I don't have it at the moment, but from what I remember
> back the last time I looked at it, yes, the database code was a mess, and
> because of that, I don't understand it.  Which brings me back to the
> question of what exactly does services do with its database?

See above. I think that explains it.

> A quick example (I am talking under speculation here - as I said before, I
> did not look at the code, and I wont bother looking at the code,
> because its
> to compicated for a avarage programmer which I hope I can be seen as).

[snip]
> My question is this:  How exactly does services process this update?

Again, see above.

[snip]
> In the past, I made (I think), quite a big effort on getting services to
> support SQL based databases, and it is exactly for this reason.  As far as
> physical searches (locating of the user's registration details)
> through the
> database goes, they are 1) accellerated, and 2) they are cached.  This is
> what makes SQL databases good for high-information stores (allot of
> records).

I think you're missing the point. No one is denying that SQL databases are
quick. But making calls outside of the current process, in a single threaded
manner, is no where near as quick as locating it within the process's
memory; this only holds true if the lookup functions are well written and
the data in memory well hashed/indexed - currently this is not so but I am
working to improve it (read: I know how to make it better and I am currently
recoding the necessary parts to fix it.)

[snip]
> The point at which I am getting, is that services is allocating allot of
> bytes in its tables to chars which is not needed, and therefor, also needs
> to update more than it needs to, aswell as send more information over the
> IRC network than what is needed.

Please provide examples of variables that use (char) instead of numeric
types. As for sending more data over the network than is needed, please
explain how using different types is going to speed things up on a text
based protocol.

> Secodly, by utilising a identifier (such as NickID set in my
> example), your
> updates, sending of memos, and various other procedures related to a
> nickname (for example), is ALLOT faster.

You keep saying you don't understand the code. I'm beginning to agree with
you. ALL commands use nicknames as targets and sources. That means I still
have to lookup the nickname of the source/target of the message to find the
ID. Once I've done that I already have all the information I need to perform
99% of the tasks Services does. In all other cases, Services uses pointers
to locate related structures, not the "names" of those structures.

> Say, I issue UPDATE table SET SetKillImmed='1'  WHERE NickID='12'
> (The user
> does a /msg nickserv set kill immed).  This is 50 (If I counted
> correctly),
> characters that services needs to send to the database.  The
> error checking
> is a simple matter of checking if return value is success or failure (7
> characters), so in total, the entire update procedure, will send
> and receive
> 57 characters.  Can this be said for the DBs currectly in use?

You have no clue how the database system works. Stop trying to improve
something you don't understand. Services does not issue commands to
anywhere - the commands come from its hub and are updated in memory. See the
explanation of the databases at the beginning of my reply.

[snip]
> In other words, by using something like SQL, you will once more, SAVE on
> performance and speed, because of the fact that the location of the record
> in the database does NOT need to be located...

Again, you don't seem to know what is happening in Services.

[snip]
> This read once / write many times in my books, is a pile of horse ....  It
> is of no relavence whether there is being read or written to the database.
> The question is about WHAT is written / read, and WHERE in the database is
> it being sent to, or read from, and if services actually KNOW where the
> information is that needs to be updated.

Okay, Chris, you have NO idea what's going on with the databases. You're
probably  confused by our references to "databases". Services does not use a
SQL style database. It uses a flat file database. Nor does Services update
the information in those databases (files) every time a change is made.
Therefore there is no "database querying" to improve.

As a side note, the term "database" does not imply SQL. SQL in turn does not
imply MS SQL, or even a RDBMS. It simply means "structured query language".
You could almost look at the Services commands as a type of "SQL".

> Thus, my closing questions are:
> 1) Can someone provide me with the structures of all the dbs so that I can
> properly port it to SQL.

See services.h

> 2) Can there be added SQL support in a non-public version of Services for
> speed analysis

I've got more pressing issues to address at the present moment.

> 3) How and where does services read / write / update information in its
> database

You may have wanted to find this out before writing this email. See the
comments at the beginning of my reply for details on how Services does its
database handling.

> 4) How does Services keep record of where in its databases information is
> stored.

It doesn't - it has no need to. Again, you may have wanted to find this out
before going off an a tangent.

> 5) Does anyone have specs (detailed) about the speed of the services
> database at the current moment?  (Note, I'm talking about accurate speed
> analysis of the databases IN USE by services, accessing the
> information the
> SAME way services does.)  Surely, if you are looking at the database, you
> CANT tell me SQL is not going to enhance performance if you DONT have
> information backing up your accusations...

This does not apply to Services because there are not database queries -
only full updates.

> If for some reason you still believe I am wrong about SQL Andrew, let us
> test this, and compare our results?  I'll bet you R100 :P  SQL is
> multithread aswell, which adds ALLOT more speed into services by
> allowing it
> to make more than one query at a time (for starters).

I think you owe me a lot more than R100 for making me read and reply to a
thesis on things that don't actually exist in Services. :P

Multi-threading will not improve a single threaded application.

A note to everyone:
		PLEASE SEND ALL REPLIES TO THE CODING MAILING LIST!

Later, Andrew


---------------------------------------------------------------
To unsubscribe, send email to majordomo at ender.shadowfire.org
with "unsubscribe ircservices" in the body, without the quotes.