[IRCServices] cheers. ideas.

[Jonathan Morton]

>     Okay, my two cents:
>
>     I'll be the first to admit that the current Services database format
>is less than ideal.  When I originally designed Services, I designed it
>the network I had started a bit earlier, which at the time had at most
>20-30 simultaneous users.  Services can write a database of a few thousand
>nicks quickly enough not to matter, but up that by an order of magnitude
>and you'll start seeing lag just from writing the database.  Moreover, a
>single bit error in the wrong place and you lose the entire database (and
>apologies to anyone who's experienced this).
>
>     There's also the issue of accessing, and particularly altering, the
>databases from outside of Services.  Although access alone isn't really a
>problem--if you want to access the Services databases now, you can do it
>by borrowing the database loading code; listchans and listnicks work like
>that--altering the databases from outside of Services' control brings in
>all the classic problems of multithreaded applications--races, deadlocks,
>that sort of stuff.  Which isn't to say it can't be done, of course, but
>as Andrew Kempe commented, it'll take a good deal of work.  My personal
>thought is that it would be better done as a multi-step process: first
>implement some manner of sending Services certain commands from outside
>(e.g. register/drop a nick), iron out the problems there, then look at
>going to complete external read/write support.

<snip>

OK, so there's two separate issues.  One: the reliability/performance of
the database file format (this *is* an issue, ChatCircuit experienced a
severe db corruption around 29th Feb / 1st March).  Two: the possibility of
external applications wishing to read/update Services data.

In terms of file format, I wonder if it's really necessary to have it all
in one file?  I've heard of many applications which need to handle lots of
data which use many files for storing it, in a systematic manner.  Squid
(the webcache) uses one or two files per cached object, and spreads the
files over a large number of directories in a single tree (this is an
extreme example).  I see no fundamental reason why there shouldn't be a
limit on the number of nicks per database file, and as many database files
as necessary.  A similar scheme for channels could exist, particularly if
there were one file per channel.

Thus corruption to the database is much more limited given corruption of a
single file, and Services' performance while writing files can be optimised
a *lot* because it only needs to write to files that have actually changed.
Also, since less data is being written at one time, it becomes feasible to
write the files in a more "robust" format, including one based on ASCII
such as XML.  Whether this is still a good idea is an entirely separate
issue which I won't try and get involved in.

In terms of external access, what kind of external access do we need?
Read-only?  Writable?  If we only need read-only access, then we can write
a daemon (or Apache module) which is capable of reading database file(s)
directly and interpreting them according to the required application.
Since it would be separate from Services, it shouldn't interfere with
performance too much, especially given an SMP box.  Writable access is more
of a problem, and requires essentially that Services get back in on the
act.  Opening another socket and allowing commands/responses to
retrieve/set data shouldn't be all that much of a problem, IMHO.  Whether
one makes that compatible with any existing protocol is left as a
discussion exercise for the reader.  :)

--------------------------------------------------------------
from:     Jonathan "Chromatix" Morton
mail:     chromi at cyberspace.org  (not for attachments)
uni-mail: j.d.morton at lancaster.ac.uk

The key to knowledge is not to rely on people to teach you it.

Get VNC Server for Macintosh from http://chromatix.autistics.org/vnc/

-----BEGIN GEEK CODE BLOCK-----
Version 3.12
GCS$/E/S dpu(!) s:- a19 C+++ UL++ P L+++ E W+ N- o? K? w--- O-- M++$ V? PS
PE- Y+ PGP++ t- 5- X- R !tv b++ DI+++ D G e+ h+ r- y+
-----END GEEK CODE BLOCK-----



---------------------------------------------------------------
To unsubscribe, send email to majordomo at ender.shadowfire.org
with "unsubscribe ircservices" in the body, without the quotes.