Blog from February, 2009

<simboss> ciao mbedward
<simboss> or michael (smile)
<aaime> Hi
<mbedward> hi simone, andrea
<mbedward> I'm not sure how to start - do you have something you would like to begin with
<aaime> Is this a meeting? (smile)
<mbedward> no just a chat (smile)
<mbedward> about jai-related things
<simboss> well, the floor is yours
<simboss> daniele is coming
<simboss> not sure if antonello is around
<simboss> looks like is not
<simboss> around
<simboss> checked on the udig channel
<mbedward> I can wait a few minutes - that's no problem
<simboss> if you did not sent around a reminder
<simboss> it might be that he has forgotten
<simboss> (smile)
<mbedward> yes - I'm afraid I forgot about reminding anyone (smile)
<mbedward> perhaps we can just start chatting - we can always go back if he joins us or rehash later
<aaime> if you want any chance of him showing up send a mail to him and Silvia
<mbedward> ok, I'll do that now
<moovida> morning folks
<moovida> thanks for the email
<mbedward> no worries
<moovida> I was working on a bug and forgot (smile)
<moovida> what did I miss?
<mbedward> we haven't started
<moovida> oh, great!
<mbedward> we couldn't start without you !
<moovida> (big grin)
<mbedward> perhaps I'll begin my just describing my immediate project needs that have led me down this path
<moovida> good for me
<mbedward> for an app that I'm working on at the moment I need to be able to do two things
<mbedward> 1. allow a user to create a grid coverage based on mathematical expressions
<mbedward> where the user just provides a relatively simple script
<mbedward> and the expressions have spatial position and/or other coverage data as inputs
<aaime> just for the record, Sextante has some raster algebra support built-in
<mbedward> ok - I'd be interested to hear about that in a minute
<moovida> mbedward: what do you mean by: the expressions have spatial position?
<--| ggesquiere has left #geotools ("Bye")
<mbedward> basically functions of x,y coords
<moovida> not row/column, but instead coordinates?
<mbedward> could be row/col, or proportion of width/height etc
<moovida> can you make an example in which you would want to use coordinates inside an expression?
<moovida> ah, ok, got it, wanted to be sure
<mbedward> sure: my app is about animals in landscapes
<mbedward> responding to various resources
<mbedward> imagine I want to create an artificial resource map
<mbedward> with a north-south gradient
<mbedward> I might do that with a simple linear function plus a noise component
<moovida> ok, nice
<moovida> so you are talking about functions really or more neighbour cell touching?
<mbedward> ???
<moovida> I mean you won't be able to embed a function based on position in a static approac
<moovida> you will have to use relative positions between cells?

  • moovida has difficulties to explain himself
    <moovida> so the gradient will be based for every cell on values of the cells around it
    <moovida> is that what you mean by function?
    <mbedward> that's not what I was think about in the example...
    <mbedward> although neighbourhood stuff certainly comes in lates
    <mbedward> later
    <moovida> ok, good, how would a hipotetic function look like?
    <moovida> you have an idea already?
    <moovida> the linear function for example
    <mbedward> value = xpos * alpha + beta + random(sigma)
    <mbedward> where xpos could be position along coverage axis scaled as 0 -> 1
    -->| mauricio (n=mauro@128.Red-80-36-0.staticIP.rima-tde.net) has joined #geotools
    <moovida> sounds good
    <mbedward> so very simple
    <mbedward> this was the idea behind 'jiffle'
    <mbedward> the ANTLR grammar that I'm fiddling with
    <mbedward> it's essentially a simple expression evaluator
    <mbedward> I've put the current ANTLR parser and tree grammars on to the project site
    <mbedward> they are very preliminary but maybe give you some idea when you look at them
    <moovida> ok
    <mbedward> but it isn't joined to images yet
    <mbedward> now that I've got basic arithmetic, trig functions etc in jiffle
    <mbedward> the next thing I'd like to do is work out how to plug input image data into the interpreter
    <mbedward> I was interested in the r.mapcalc page that you pointed me to yesterday
    <aaime> again for the record, what about Janino? http://www.janino.net/use.html It can be used to compile expressions into bytecode. Makes the evaluation very fast
    <moovida> r.mapcalc is what I will need and for which I could work
    <mbedward> I don't know this at all - would be very interested to know more - have you used it ?
    <aaime> nope
    <moovida> and I think it contains also what you need
    <moovida> so, how could I/we help you?
    <mbedward> you mentioned that you were using jep previously ?
    <aaime> sextante does too
    <aaime> (use jep)
    <moovida> yes, I did for a while
    <moovida> but it is now closed source
    <moovida> and so I would not want to use it any more
    <moovida> or better, I do not use it any more
    <aaime> yep, that's why I looked into janino a little bit, to suggest Sextante an escape route
    <mbedward> I understand jep was highly optimized (question)
    <moovida> yes, it was, but from that moment on they closed the source (smile)
    <mbedward> at the moment speed is less important for my needs than just ease of use and flexibility
    <mbedward> and having something that I can understand !
    <moovida> ok, but you would not want to use jep, right?
    <mbedward> what are your r.mapcalc needs ?
    <mbedward> no - I want to use open source
    <moovida> great
    <moovida> my needs are well inside yours
    <moovida> functions would be great
    <moovida> because they contain also functions based on several maps
    <moovida> and conditions are important
    <mbedward> yep
    <moovida> things like:
    <moovida> if in map1 the value is null, do that, else take value from map2
    <moovida> but I think on this we have the same ideas
    <mbedward> yes
    <moovida> so one question from an ignorant antrl person
    <moovida> antrl is used as a functional language "compiler"?
    <moovida> and then you need something to evaluate the transformed language?
    <mbedward> it is general parser building tool
    <mbedward> similar in aim to lex, yacc, bison...
    <mbedward> for example
    <moovida> ok, so thorught that you get your java language for the math you write
    <mbedward> you can
    <mbedward> or C, or python or other things
    <mbedward> it can be used to build a translator
    <mbedward> but what I am interested in is an interpreter
    <mbedward> so two steps...
    <mbedward> first provide a grammar that describes the language elements (similar to EBNF)
    <mbedward> from this ANTLR helps you to build a lexer and parser that can accept language statements
    <mbedward> detect syntax errors
    <mbedward> re-arrange things for faster processing
    <mbedward> the output of this first step is a tree representation of your input statements
    <mbedward> Abstract Syntax Tree in the jargon
    <mbedward> second step is to build a 'tree-walker'
    <mbedward> this knows how to take navigate the tree-representation of your 'program'
    <mbedward> and you can embed the code for actions into the walker
    <mbedward> that's a really terrible explanation (smile))
    <mbedward> I'll give an example
    <moovida> (smile) I understand, the JGrass console is built like that
    <mbedward> phew...
    <moovida> now I am very sorry we didn't use antrl
    <mbedward> antlr is really cool
    <mbedward> I'm only learning
    <mbedward> but it is possible to get quite a bit done in a small amount of time
    <moovida> the developer created an own compiler, and now every time I have to changes something, I get mad (smile)
    <moovida> yes, that was my feeling of antrl
    <moovida> sound really good
    <mbedward> in the past I've written parsers by hand and this is **much** easier
    <moovida> what timeline did you think of for this?
    <moovida> also, were are you based?
    <mbedward> I'm already very late (smile)
    <moovida> (big grin)
    <mbedward> I'm in Sydney
    <moovida> argh, I found one in London and thought we could meet for an initial push (sad)
    <mbedward> I think he works for the BBC
    <mbedward> I saw his name on TV once
    <moovida> hmmm, then let's get back to the timeline, Sydney is out of budget for this project (smile)
    <moovida> just to understand if we could coordinate
    <moovida> I also have to study antrl first
    <moovida> so it would be good to see if we are at least a bit in sync
    <mbedward> well, it's very fuzzy for me and there are many strands to my project that I can also work on when it's not convenient to be in sync
    <mbedward> sync in time I mean
    <mbedward> there are good intros on the antlr site
    <moovida> yep
    <moovida> great, I'll check them out
    <mbedward> I want to find a good starting approach for inputting image data into the interpreter...
    <mbedward> r.mapcalc had the simple convention of var names
    <mbedward> if not functions etc. they were assumed to be maps - is that right ?
    <moovida> yes, and keeping those would help me get JGrass -> GRASS compatibility
    <moovida> yes, exactly
    <mbedward> mmm...
    <mbedward> I was thinking of something slightly more indirect
    <mbedward> though it might still be compatible
    <moovida> users have huge scripts for mapcalc to do the most powerfull things
    <mbedward> I thought that would be the case
    <mbedward> so it would be great to be compatible
    <moovida> good you think so
    <moovida> at least to start like that (smile)
    <mbedward> I want to keep this all as modular as possible
    <moovida> then there are always possibilities to get out of compatibility
    <moovida> what do you mean?
    <mbedward> well the interpreter is one module
    <mbedward> it 'knows' (will know) how to access data from some provider interface
    <mbedward> but the provider implementations could be quite diverse
    <mbedward> my thinking is very fuzzy...
    <mbedward> in r.mapcalc is there any sort of definition section for variables / inputs
    <mbedward> or do they just appear in statements ?
    <moovida> I don't understand exactly what you mean
    <mbedward> is there anywhere in the script where you have to declare var names prior to using them ?
    <moovida> you can have also variables evaluated in steps and then used inside the script
    <moovida> no, var=eval() is enough
    <mbedward> I need to go study r.mapcalc while you look at ANTLR (smile)
    <moovida> nice
    <moovida> (smile) then we have a sort of plan
    <mbedward> yes !
    <moovida> even if I feel it will take me some longer
    <moovida> the mapcalc module can do very complex things
    <moovida> did you think to make a constatly usable progress?
    <mbedward> definitely !
    <moovida> I mean, start to work and get a version that supports some functions and son and on?
    <mbedward> otherwise giant mess !
    <moovida> great, that sounds good
    <moovida> same for me here, and also the project then can grow
    <moovida> slowly and stable
    <mbedward> there is also a second strand to this work - JAI operators
    <mbedward> especially for neighbourhood map calculations
    <mbedward> I need to be able to do the sort of things that, in the old, old days I might have done in arc-info (smile)
    <mbedward> esp. focal functions
    <mbedward> e.g. average neighbourhood value, dominant value, number of discrete values...
    <mbedward> I was a bit surprised that these were not already in JAI operators
    <moovida> are you sure there aren't?
    <mbedward> also being able to constrain these and other calculations with mask / ROI
    <moovida> sounds strange to me also
    <mbedward> I think some of these things could be useful to geotools etc. generally
    <mbedward> and I would like to open source my code for new operators
    <mbedward> so I plan to put them on the jai-tools site also
    <moovida> seems to be a lot of work to be done (smile)
    <moovida> what was your schedule again?
    <mbedward> I'm very very late (sad)
    <moovida> for yesterday or so?
    <mbedward> but I am also the project manager
    <moovida> (smile)
    <mbedward> unfortunately the project does not have enough money to employ a real programmer...
    <mbedward> so an ecologist is doing the job
    <mbedward> me
    <moovida> (big grin)
    <moovida> ecologist and environmental engineer (smile)
    <mbedward> mmm... (smile)
    <mbedward> obviously there is overlap between ANTLR side and jai-operators side
    <moovida> any other person that wants to join the brigade?
    <mbedward> I think / hope simone is interested in the jai operators strand
    <moovida> simboss, dany_r ? What about you guys?
    <dany_r> yes he's interested
    <moovida> mbedward: just to have an idea, you need a version that does x,y,z for date a,b,c
    <moovida> can you tell about x,y,z and a,b,c
    <moovida> if you are allowed, and just a more-or-less
    <moovida> is enough
    <mbedward> yep, roughly anyway...
    <moovida> this job is resources absorbing and if I help I want to be able to really do so
    <mbedward> in next month I would like to have the scripting working to the point where
    <dany_r> our aim would be allowing handling, chaining math operations and functions in JAI operators.
    <mbedward> it can do arithmetic and basic math/trig functions already in grammar
    <mbedward> plus conditional expressions
    <moovida> hmmm, I guess that will not be exactly the case
    <moovida> am I right?
    <mbedward> and access data from input images
    <mbedward> (will get to that in a sec)
    <moovida> doing it mbedward's way will mean that no chaining occurs?
    <mbedward> if I can get that working by end of Feb that would be very good for me
    <mbedward> ok - in terms of output
    <mbedward> I have thought about two approaches, each of which I'd like to implement
    <dany_r> in this right moment Simone is not here but he can provide you more details about its idea very soon.
    <mbedward> first is interpreter produces output that can be linked to a JAI image function
    <mbedward> great
    <moovida> brb
    <moovida> sorry guys, I have to run, could anybody post the logs, or send me the logs, if much is said after this moment?
    <mbedward> no worries
    <moovida> mbedward: I am very interested to join the efforth, but I have to understand if I am ablr to contribute that early
    <mbedward> sure
    <moovida> to do so, I will study antrl in the next days
    <moovida> and study what you already did
    <mbedward> you can then tell me all the mistakes
    <moovida> would be glad to (smile)
    <moovida> and let's keep the discussion in the list active please
    <mbedward> will do
    <moovida> thanks for the chat
    <mbedward> cheers
    <moovida> ciao
    <--| moovida has left #geotools
    <mbedward> dany would you like to keep going when simone comes back ? or I can pen some thoughts and send them to you...
    <simboss> ciao michael
    <simboss> I am back
    <mbedward> ciao simone
    <simboss> sorry, but I have someone here, so I had to talk to him
    <simboss> (smile)
    <simboss> I am all yours now (big grin)
    <mbedward> (smile)
    <mbedward> I think we were only just getting to teh things you are most interested in
    <mbedward> jai operators plus
    <simboss> ah great
    <mbedward> possibility of using an interpreter to help make image functions etc.
    <simboss> well, at this stage
    <simboss> my interest is in having a good project
    <simboss> where we can drop JAI operators
    <simboss> so that we can releases
    <simboss> have
    <simboss> releases
    <simboss> in a more stable manner
    <simboss> JAI is not dead
    <simboss> but it's a zombie
    <mbedward> why has that happened ?
    <mbedward> it's hard to understand
    <simboss> the members of the prject are heavily involved in other projeects
    <simboss> well, it's simpler than it seems
    <simboss> JAI was created under bih push from JPL
    <simboss> the Jet Propulsion lab fa NASA
    <mbedward> yep
    <simboss> they used in a few mission
    <simboss> and they are still using
    <simboss> and it's pretty good the way it is
    <mbedward> but there are some surprising gaps
    <simboss> they are not interested in moving it forward
    <simboss> or in babysitting
    <simboss> it
    <simboss> exact
    <simboss> but it's an empasse
    <simboss> they do not want to spend time on it
    <simboss> but they are not releasing it
    <simboss> as OS
    <simboss> that's why I started imageio-ext
    <simboss> and I am happy to see something like JAI-operators
    <mbedward> are there tricky licensing issues ?
    <mbedward> for instance
    <mbedward> I have written a modifed convolve operator
    <mbedward> that can use an ROI
    <mbedward> so it's really just a hack of the existing image op
    <simboss> my advice about code from JAI is use it as a template
    <simboss> but rewrite it all
    <mbedward> I see
    <simboss> the license is too messy to reuse that code
    <simboss> for imageio
    <simboss> you can reuse it, but that's another story ....
    <simboss> anyway
    <simboss> I am hoping that sooner or later SUN will OS JAI and Imageio
    <mbedward> I have a number of operators that I need for my current project
    <mbedward> so I would like to open source these
    <simboss> if we show them that there is a community able to ensure the continuity of the projects that might help them to make up their minds
    <mbedward> that would be great
    <mbedward> but surely they must know that jai is very actively used already ?
    <simboss> yeh they know very well
    -->| dany_r_ (n=chatzill@host177-41-dynamic.30-79-r.retail.telecomitalia.it) has joined #geotools
    <simboss> but their team, while very very good, is small
    <mbedward> ok
    <mbedward> I should compile a list of the operators that I will be needing soon
    <mbedward> and send it to you
    <mbedward> perhaps some are already available and I have overlooked them
    <mbedward> otherwise it will be the list of things I will be coding in the next month or two
    <simboss> there is one thing
    <simboss> that I might want to bring up
    <simboss> I did the same thing with antonello a while ago
    <simboss> are your operators geospatial-aware (e.g. they need some parts of geotools torun)
    <mbedward> not so far
    <simboss> or do they run in the pure raster space, where there is no notion of geospatial
    <mbedward> yes
    <simboss> I would recommed to be careful with this in order to not introduce circular dependencies between the teo projects
    <mbedward> I understand
    <simboss> usually I try, whenver possible, to clearly separate geostuff (geotools) from pure raster stuff (jai or imageio)
    <mbedward> yes
    <simboss> the key for doing this is usually the gridtoworld transform
    <simboss> (probably I am going to far, but I think it is important to say at least once)
    <mbedward> yes, that's good point to bring up early
    -->| acuster (n=acuster@mtd203.teledetection.fr) has joined #geotools
    <mbedward> so far
    <mbedward> for example with the convolve operator I mentioned
    <mbedward> that works in raster space
    <mbedward> then there is a wrapper for geo-spatial part
    <mbedward> but I agree with you that keeping those things separate is best
    <mbedward> and I also find it easiest
    <mbedward> anyway, regarding the project and site
    <simboss> following you...
    <mbedward> I am very open to
    <mbedward> approaches to managing it / handling releases etc
    <mbedward> I don't have fixed ideas or
    <mbedward> requirements in that regard
    <mbedward> and to begin with
    <mbedward> I will just be gradually uploading my operators
    <mbedward> and working on the ANTLR related strand
    <mbedward> we are probably done for now - yes ?
    INFO Preference "collapseMsgs" is "on".
    <simboss> I think
    we would like to port one operator
    that is right now inside geotools
    which we could probably somehow join with your stuff
    we should develop also a simple operator

    <-- acuster has left irc.freenode.net ("Leaving")
    <simboss> thatcan be used
    to transform a single color
    into a trasparent color
    <mbedward> which is the operator in geotools ?
    <simboss> it is buried inside the renderer since there was not complete consensu about where to put it
    <CIA-21> aaime * r32368 /trunk/modules/ (4 files in 3 dirs): GEOT-2316 jdbc-ng should advertise query capabilities
    <simboss> it can be used to create a piecewise transformation
    <mbedward> have to go in minute - sorry
    <simboss> np, I think we are done

    <-- dany_r has left irc.freenode.net (Read error: 113 (No route to host))
    <mbedward> thanks simone !
    ciao
    <simboss> thanks to you man
    ciao ciao

Some good news from the developers list today. The ECQL documentation is finished and the implementation is now available on trunk.

The OGC Common Query Language is defined as part of the catalog specification and provides a very readable text format for defining Filters. Filters are also use by the WFS specification to select data.

However the CQL format leaves out a few of the capabilies that WFS supports. The ECQL syntax documented above extends the CQL syntax allows you to do everything WFS is capable of.

Jody Garnett (PSC member) here - I have been unable to attend the formal IRC meeting time do to a change in timezone. As such I am setting up an informal IRC chat time for the month of February for Q&A and hopefully to work through a range of fascinating topics, ideas and directions that have emerged for the GeoTools library.

Click on the link to see the time in your area:

I have an idea for this week already:

  • Week 1: Review the state of Filter with Andrea, consider FilterCapabilities (ie function list) and Function

Here are some ideas to consider for the rest of the month (not each depends on hunting down a party I know is interested)

  • Go over the rendering module and clean it up for documentation in the user guide
  • Figure out OSGi bundle plan (requires Harald Wellmann)
  • Go over bzr (with Adrian Custer?)
  • Review jdbc-ng work and go over migration stratagy
  • Set up some good MemoryDataStore, MemoryFeatureCollection implementations (with Frank Gasdorf?)
  • Review the FeatureCollection / FeatureSource plans Justin wrote up last year

The idea is for some hands on community work; with a PSC that can review and commit patches if needed.

Happy Hacking and welcome to 2009.