Agenda:

1. what is up
2. SoC
3. GeoTools contirbutor agreement
4. arcsde datastore command design

<jgarnett> How are you andrea?
<aaime> it's holiday in Italy, not sure about other countries
<aaime> sick, my throat hurts
<jgarnett> is it warm enough for you to play tennis these days?
<aaime> and sick, my hope in Struts2 was demolished by a couple of days investigation
<jgarnett> We just had a storm; so the sky is scrubbed clean
<aaime> jgarnett, it is all year, I play indoor during winter
<jgarnett> that is interesting; the struts 2 part ... were they not bold enough?
<sfarber> yo, meeting time?
<aaime> jgarnett, not really, it's just that there is no tool support
<aaime> would you write java code with notepad?
<jgarnett> you had to do struts 1 in notepad.
<jgarnett> but I agree it is nobodies preference.
<aaime> well, jsp alternatives (freemarker) lack editors that can do html and the templating at the same time
<jgarnett> did you ever try the eclipse Web Tools plug-ins ?
<aaime> I have it installed, but we need to move away from jsp in order to have a pluggable UI
<jgarnett> I have not looked into struts 2 enough to know what they did different this time around.
<jgarnett> hrm
<jgarnett> well lets start the meeting...
<jgarnett> floor is open for agenda topics.
<aaime> I cannot think of any
<jgarnett> Saul I wanted to chat with you or gabriel to see what was going on w/ arcsde datastore; is gabriel realling moving to a command based system like WFSDatastore?
<aaime> dunno
<jgarnett> I was working at the same time as him and kept tripping up.
<jgarnett> hard to know with out email from him...
<aaime> 3...2...1
<aaime> ...0
<jgarnett> 1) what is up

• aaime sick of java web frameworks (other than Wicket)
  <jgarnett> jgarnett - enjoying the holiday; looking into ESRI Web 2.0 about face; thinking about finishing up the user guide if we have some SoC students to read it.
• groldan enjoying holiday too
  <jgarnett> okay moving on ...
  <jgarnett> 1) SoC
  <jgarnett> Simone was nice enough to get us a page going
  <jgarnett> I wonder if there are any students here today?
  <jgarnett> So far the difference for me has been a bunch of private emails asking how to install geotools.
  <jgarnett> Anyone else run into a SoC student?
  <aaime> not me
  <groldan> nope
  <jgarnett> http://docs.codehaus.org/display/GEOTOOLS/Summer+of+Code+2008
  <sfarber> jgarnett, sure thing. Let's chat after meeting?
  <jgarnett> If last time is any indication we won't get much until the last couple hours before the deadline; If I can ask everyone to mind the user list this week and do not hesitate to fix any of the documentation.
  <jgarnett> this will be a quick meeting
  <jgarnett> 2) svn cut off
  <jgarnett> I have gotten emails returned by most of the active developer community now
  <aaime> mail sent, hope it gets to destination
  <jgarnett> and surprisingly a lot of the inactive devel community. It has been really nice to hear how people are doing in their various paths in in life.
  <aaime> any interesting story?
  <jgarnett> So the good news is we will still have enough developers to continue after the svn cut off :-D
  <jgarnett> the list is here
  <jgarnett> http://docs.codehaus.org/display/GEOTOOLS/Summer+of+Code+2008
  <bullenh> uhh..
  <jgarnett> oops
  <jgarnett> http://docs.codehaus.org/display/GEOTOOLS/Graduate+from+OSGeo
  <jgarnett> yeah!
  <jgarnett> (sorry new firefox beta is occasionally tripping up)
  <jgarnett> It was nice to hear that RobA is enjoyig his new position; that Sanjay still exists and so on...
  <aaime> (new firefox beta is working a lot better for me than the old stable )
  <jgarnett> I also got a lot of feedback from those that are really happy about the OSGeo direction; excited that they will find an easier time getting funding for GeoTools work etc...
  <bullenh> (I really find the new gui much better in it)
  <jgarnett> the kind of thing that we considered when we joined; but have since lost excitement over.
  <jgarnett> So next Monday we will try and restrict svn access; and then we can update our headers.
  <desruisseaux> Thanks for doing that Jody!
  <desruisseaux> (I means contacting all those peoples)
  <jgarnett> well thanks for waiting; feel I delayed everything a bit doing it myself.
  <jgarnett> a mistake I won't repeat with the headers
  <jgarnett> So that may be it for the meeting ...
  <jgarnett> shall I post the logs; or just open up the floor for a general chat?
  <desruisseaux> I can not stay toonight on my side...
  <sfarber> jgarnett: you've got gabriel and me here...wanna talk arcsde?
  <desruisseaux> Bye all
  <jgarnett> well thanks for staying thus far.
  <jgarnett> sure
  <aaime> bye
  <groldan> sure
  <jgarnett> gabriel what are your plans for arcsde?
  =-= jgarnett has changed the topic to "0) what is up 1) SoC 2) One week till svn cut off 3) arcsde chat"
  <groldan> right now it should be editing versioned tables
  <groldan> next step is
  <groldan> to improve the threading handling a big deal
  <sfarber> from my end the current list is:
  <groldan> since the current approach is error prone and hard to maintain
  <sfarber> * support lots more arcsde raster types
  <sfarber> (float, double, colormapped 1-byte, non-colormapped 1-byte, etc)
  <groldan> the general idea is to control the threads sde operations are run at, in order to be able to execute more granular units of work concurrently
  <groldan> instead of locking a connection since a FC iterator is open until its closed
  <groldan> with that, we're joy
  <sfarber> * re-do the raster test suite in junit 4 style and taking a cue from gabriel so that the test load their own data.
  <sfarber> that's all from me: groldan:
  <jgarnett> I have not looked at JUnit 4 yet; it seems Andrea was all gung-ho to switch.
  <groldan> what do you gain redoing in junit4?
  <jgarnett> the work I was trying to do gabriel may be related to what you are doing. gabriel.
  <jgarnett> I was trying to put arcsde datastore on a diet
  <sfarber> with the connection locking are you proposing to override the SeStreamOp locking methods?
  <jgarnett> and make it still work when only one thread is available.
  <aaime> jgarnett, switch to junit4 completed
  <groldan> jgarnett: indeed, what you're trying to achiece may only be possible with something like that
  <aaime> you can use it in your modules
  <aaime> (just needed a change of version in 2 dependencies)
  <jgarnett> so I was capturing this as a subclass of arcsdedatastore; and then look at all the little methods and make the read-only ones reuse the existing connection
  <jgarnett> aaime is the junit4 something you want to just do for the whole library? Or is it more serious than that.
  <groldan> sfarber: looking for sestreamop lock methods...
  <aaime> we already have it for the whole library
  <aaime> junit4 contains junit3 classes as well
  <sfarber> ok, so the thread synchronization is going to be something that's enforced at the gt2-arcsde code level?
  <aaime> you can use both at the same time
  <groldan> what locking methods are those?
  <sfarber> groldan: one sec...let me get a little bit out and I think it'll be clear
  <jgarnett> (gabriel I agree; I think I should wait until after your work is done? Or at least review your design rather than guess. For a moment I was thinking you were doing something like the WFSDataStore transaction state diff where it keeps a list of "commands" and then applies them on the result as it is sreamed...
  <jgarnett> Do you have any design stuff we can read? Or just read the code ...
  <groldan> jgarnett: yes, it'd be of help to get you aborad to get to use a single connection, once that's possible
  <sfarber> so, groldan, you're proposing that everyone writing code at the gt2-arcsde datastore code level use a set of methods internal to the the datastore code to coordinate and use connections correctly?
  <groldan> nope
  <groldan> that's all arcsde plugin business
  <groldan> the thing is as follows (in my understanding):
  <groldan> -sde connections are not thread safe, yet a single thread can run various operations over it, which's what jgarnet needs
  <groldan> -we're handling transactions with sde native transactions, ie, held in the connection
  <groldan> so any transactional operation plus queries over a FeatureStore need to use the same connection
  <groldan> and often they're called concurrently
  <groldan> so the plan is to run all the operations over a connection in a single thread, which will allow
  <groldan> for a more granular locking
  <groldan> and hopefully the thing is gonna be easy with the java.util.concurrent.Executors framwork
  <jgarnett> thinking
  <jgarnett> single thread per Transaction you mean? A little command queue for each thread ...
  <groldan> yup
  <jgarnett> okay I am starting to get it
  <groldan> that's the only way I can see to avoid the performance overhead and maintainance nightmare right now
  <jgarnett> so I wasted a couple of days last week But now at least I can try and plan to take part later on ... provided you can tell me when that is?
  <sfarber> groldan:
  <groldan> I'm waiting for cholmes' go ahead, once I get it it'll take three work days
  <groldan> sfarber?
  <jgarnett> gabriel / saul should we of asked for a proposal / design doc for this?
  <groldan> hmm... I guess not?
  <jgarnett> okay; if it will help I can be available for the QA side of things at the end of your 3 days.
  <groldan> sounds good, sure
  <sfarber> ok groldan, so how will this affect someone calling "ArcSDEPooledConnection.getConnection()" a lot
  <groldan> gonna keep communication open on the mailing list
  <groldan> you wondering how will it affect the gce side of the fence?
  <groldan> which reminds me, Saul, we need to figure out what to do about DataStore.dispose()
  <jgarnett> if I can guess; it would be come ArcSDEPooledConnection.executeWithConnection( new ConnectionRunnable(){
  <jgarnett> void run( Connection )
  Unknown macro: { <jgarnett> ... <jgarnett> }
  
  <jgarnett> );
  <jgarnett> or something like that ...
  <groldan> something like that
  <groldan> ideally ArcSDEPooledConnection should stop extending SeConnection
  <sfarber> Err, can you be really specific? I'm not really so worried about the gce stuff (I mean, que sera, sera) but I'm just curious about what the nature of getting/executing connections will be and how it will change.
  <groldan> and provide a ArcSDEPooledConnection.execute(ArcSDECommand);
  <groldan> interface ArcSDECommand
  Unknown macro: { void execute(SeConnection conn);}
  
  <groldan> as to speak
  <sfarber> ahh. Ok, just like HibernateTemplate in spring?
  <groldan> extending seconnection is something that may be worth to stop doing, as it'd simplify unit testing
  <groldan> I mean, unit testing
  <groldan> dunno about HibernateTemplate in Spring, Saul
  <sfarber> So what, specifically, does this solve? How does it solve the "featureiterator holds open connection" problem?
  <groldan> but as long as it starts to make sense for you, I'm glad
  <sfarber> yeah, it makes sense.
  <groldan> it solves the featureiterator problem
  <groldan> but phrase it like: "featureiterator holds lock over connection" problem, instead
  <groldan> think of the following:
  <groldan> uDig is our best test bench for this
  <groldan> you have 5 sde layers in uDig
  <groldan> edit one
  <groldan> udig sets the same Transaction for the 5 FeatureStores
  <groldan> the ArcTransactionState grabs a connection and holds it, since the connection manages the native transaction
  <groldan> udig calls one addFeatures and 5 (or 6) getFeatures, concurrently
  <groldan> right now, the addFeatures operation is an atomic operation
  <groldan> but querying the 5 layers and iterating over them are not
  <groldan> each iteration consists of a number of atomic operations, say, every next()
  <groldan> but right now we're forced to wait until one iterator is fully traversed until another one is open
  <groldan> with this change, we can still serve the 5 iterators concurrently, or interleaved
  <groldan> since we can call a fetch command on each next() invocation
  <groldan> rather than grabbing the lock when opening the iterator and releasing it when closing it
  <jgarnett> understood; that is why I wanted this to settle down before I try the "single connection mode" - since may of the "get metadata" requests I would like to handle as atomic operations.
  <groldan> makes sense?
  <sfarber> groldan: I'm still a bit confused. In the end you open a SeQuery against an SeConnection object, and until you've closed that SeOuery you can't use that SeConnection for anything else, right?
  <groldan> not right
  <jgarnett> so if you can leave some kind of .exectureReadonly( ... ) method in there it will help; since I can use a connection regardless of if a transaction has been started or not.
  <sfarber> ahh, ok.
  <groldan> the problem is you have to do that in the same thread
  <groldan> so the queue per thread commands
  <jgarnett> guys ping me when done and I will post the longs
  <sfarber> So you're saying that in the same thread (but ONLY in the same thread) you can open a number of SeQuery objects against ONE SeConnection object and read from each query out of order and it will work?
  <groldan> if get two concurrent getFeature requests and try to execute the SeQuery from the calling threads, you'll get a SeException
  <groldan> I mean, while an SeStreamOp is open by one thread, no other thread can open an SeStreamOp
  <groldan> BUT a single thread can have up till 24 SeStreamOp open at same time
  <sfarber> So the lock policies listed at http://edndoc.esri.com/arcsde/9.2/api/japi/docs/index.html
  <groldan> (24 is a server side configurable thing afaik)
  <sfarber> aren't really relevant? We can't just set SE_UNPROTECTED_POLICY and let ArcSDE trust us that we won't do anything too screwey?
  <sfarber> yup, I've definitely run across the streams/connection setting before.
  <groldan> about the policies
  <groldan> I guesss they don't solve our problem
  <groldan> trylock fails if another thread is using the connection
  <groldan> lock choses a random waiting one
  <groldan> we need a predictable queue
  <groldan> but I might be convinced otherwise
  <groldan> yet, one way or the other, the change applies in that it isolates this problem in a single point, rather than being a concern to deal with every time you use a connection?
  <sfarber> sure, ok that makes sense.
  <sfarber> So, in the end, the change is something like this:
  <sfarber> OLD WAY:
  <sfarber> ArcSDEPooledConnection con = pool.getConnection();
  <sfarber> //do stuff with con
  <sfarber> con.close();
  <sfarber> wait...
  <sfarber> I missed an important parT!
  <sfarber> OLD WAY:
  <sfarber> ArcSDEPooledConnection con = pool.getConnection();
  <sfarber> con.lock();
  <sfarber> SeQuery q = new SeQuery(con);
  <sfarber> q.execute();
  <sfarber> con.getLock().release(0;
  <sfarber> con.lock();
  <sfarber> q.close();
  <sfarber> con.unlock();
  <sfarber> con.close();
  <sfarber> and now the NEW WAY:
  <sfarber> // doing something in a method...
  <sfarber> ArcSDEPooledConnection con = pool.getConnection();
  <sfarber> con.execute( new ArcSDECommand()
  Unknown macro: { <sfarber> public void execute(SeConnection con)

  Unknown macro: { <sfarber> SeQuery q = new SeQuery(con); <sfarber> q.execute(); <sfarber> //blah blah <groldan> more or less <sfarber> q.close(); <sfarber> }

  <groldan> I'd better say}
  
  <groldan> conn = pool.getConnection();
  <jgarnett> ie this is what they mean when they say "closure" for Java; the same reason I keep trying to use FeatureVisitor
  <jgarnett> a couple more things gabriel
  <jgarnett> use an abstract super class for your ArcSDECommand
  <jgarnett> and provide methods to create SeQuery etc...
  <groldan> ah, sorry, you talking about SeQuery, me thinking of ArcSDEQuery
  <groldan> yes, that's right Saul
  <jgarnett> that way you can clean them up after.
  <jgarnett> less client code == better
  <groldan> yup
  <sfarber> so the implementation of con.execute() is that it can "store up" these ArcSDECommand objects and run them out of order, or synchronize them in a one-thread-per-actual-connection implementation?
  <groldan> if I were able to provide a full facade to arcsde that's be ideal, but wont have that much time I guess
  <groldan> sfarber: not planning to run them out of order
  <groldan> but in the invocation order, as seen by each calling thread
  <jgarnett> gabriel you are still going to have "fun" when having several queues going on different "Transactions" right?
  <sfarber> ok, makes sene. This is mostly purely to solve the problem of connection locking and getting all the commands executed in one thread, rather than every bit of "direct-to-sde" client code running at the SeConnection synchronization stuff willy-nilly.
  <groldan> different transactions means different connections
  <jgarnett> are you thinking of a single queue? ie to preserve order across transactions (ie why bother) or multiple queues one per transaction.
  <groldan> since the transaction state holds the connection
  <groldan> one per transaction, the connection handles the native transaction, a geotools transaction is tied to a sde transaction throught the transaction state
  <groldan> GTTransaction-->ArcTransactionState->ArcSdePooledConnection-->command queue
  <jgarnett> understood
  <jgarnett> so I get where you are going gabriel
  <jgarnett> saul how are you doing?
  <sfarber> eh, I'm still a bit fuzzy on how ArcTransactionState relates to arcsde native transactions, but I'm not too worried about it. I'm happy about the command-based callback system.
  <jgarnett> gabriel I am thinking hard about the single case now
  <jgarnett> basically set up a datastore with a single command queue
  <jgarnett> and have two kinds of commands; one for read only and one for read-write
  <sfarber> However, it's going to take a lot of re-coding! It'll take me a while to get everything back and working after the change.
  <jgarnett> and use that to figure out when to schedule them
  <jgarnett> it may just be a matter of building up the queues; and then letting them "go" as they have a connection made available.
  <groldan> you mean for the maxConnections=1 case?
  <jgarnett> yes I do
  <groldan> well we have a timeout parameter, how long to wait for an available connection
  <jgarnett> It is almost like these things are comming in with Transaction.ANY
  <jgarnett> I want them schedule on whomever is "next"; possibly budging in line since they do not represent modifications.
  <groldan> the thing would be, to make transactions as short lived as possible
  <groldan> hmmm I feel like in a trap here...
  <sfarber> hey guys, I gotta bail. I'm glad we talked this through. It would have been a shock otherwise!
  <sfarber> catch you all later.
  <groldan> ok
  <groldan> you want udig running with a single connection
  <groldan> but I wonder how does that play with the way udig handles transactions
  <groldan> lets see.. (just thinking)
  <groldan> if all the sde layers have the same transaction set (which is what happens right now I guess?)
  <groldan> we should be gold
  <groldan> then, say there's no transaction in course... and you hit refresh
  <groldan> the sde will get N concurrent getFeature requests, with Transaction.AUTO_COMMIT
  <groldan> that means they're free to ask for their own connections
  <groldan> so
  <jgarnett> hi gabriel; sorry was distracted building 2.2.x - I don't mean you to be trapped; and this goal may not be possible.
  <jgarnett> in udig all the arcsde layers for a map use the same transaction.
  <jgarnett> indeed all the arcsde layers ever should use Transaction.AUTO_COMMIT until they have need to modify data.
  <groldan> no, I don't say trapped in a bad sense
  <groldan> it should be possible indeed
  <groldan> I'm just trying to figure out the exact scenario
  <groldan> so, in in autocommit mode and the pool has a single connection
  <groldan> you're tied to have enough luck as the pool.timeOut parameter is high enough as to allow serializing all the getFeature requests
  <groldan> makes sense?
  <jgarnett> correct; single connection until you start editing
  <jgarnett> after that single connection for the Map
  <jgarnett> (but I still would like all the "book keeping" commands to go out on that connection .... even though right now they are AUTO_COMMIT
  <jgarnett> they really should be DO_NOT_CARE)
  <jgarnett> and then if the user opens a second map
  <jgarnett> that is when they are stuck; and run out of connections.
  <jgarnett> for my customer it is more important that the number of connections be low - and 1 if possible.
  <jgarnett> right now I think I am at 1 for AUTO_COMMIT, and 1 per map
  <groldan> mean you're being able to do that?
  <groldan> and your customer should consider using WFS to keep the connection count under control for multiple users
  <groldan> (kidding, I know we already talked about that)
  <jgarnett> heh
  <jgarnett> well make geoserver faster!
  <jgarnett> okay let me wrap up the logs; thanks for the chat.
  <groldan> thanks you
  <jgarnett> (did you understand what I was thinking you were up to? keep the queue of commands in memory and use it to postporcess the feature reader)
  <jgarnett> and then execute the entire set of modifications in one go
  <jgarnett> like wfsdatastore does.
  <groldan> I understand, but I'm not going to have the energy to do that I guess
  <jgarnett> fair enough!
  <jgarnett> enjoy the rest of your holiday
  <groldan> ditto
  <groldan> (though something tells me that could be a sort of stratagy object since the execution is gonna be isolated to a single point?)
  <jgarnett> better yet; subclass of DataStore
  <jgarnett> DataStoreFactory makes the decision
  <jgarnett> (I already started that side of it; but I will now delay further work until I hear from you)
  <jgarnett> consider it a BIG stratagy object.
  <groldan> and sets one or other executor, using composition instead of hineritance
  <jgarnett> either works ... I was just trying to poke fun at using stratagy everywhere; one of the main feedbacks on uDig is that code should "make up its mind" rather than delegate out responsibility all the time.
  <groldan> ok, I'll try to keep your goal in mind for the design, though won't be able to enforce your use case
  <jgarnett> make it easier to follow / debug.
  <jgarnett> good good