<brett> ok, anyone up for dependency mediation?
<jdcasey> bring it on
<brett> jdcasey: you said you had some feedback on that?
<jdcasey> it's about the whole resolve-to-concrete thing on release
<brett> yep, that's important.
<brett> I had a couple of questions myself in the yellow box there ;)
<brett> what's your Q?
<jdcasey> what about creating either (a) a special release-POM with all deps resolved concretely, or (b) a special tagged version of the POM with that info, followed by another non-tagged commit of the user-defined POM
<jdcasey> this is something it seems that you're wrestling with in that document...
<jdcasey> I'm -1 on losing user info in favor of resolved deps...but that's an obvious statement
<brett> yeah, I didn't think of recreating the pom with resolved stuff, I was just going to mark it up
<brett> which also facilitated doing it in SCM
<jdcasey> mark it up how?
<brett> that extra resolvedVersion tag
<brett> maybe it could add a profile :)
<jdcasey> that's an idea, but it'll lead to some serious bloat, and for something that's essentially a snapshot in time
<jdcasey> IMO so would the resolvedVersion stuff
<jdcasey> resolvedVersion would get stale pretty quick, and probably would be removed before starting development of the next release anyway
<jason> what is scope=none? only resolved for the project?
<jason> the example on the top there
<jdcasey> but wow, this represents a big shift in maven philosophy wrt versioning...now we're starting to enforce a versioning scheme of sorts
<brett> jason: that was how I figured to exclude transitive deps you don't want
<jason> when would you want to do that?
<jason> is the jdbc thing a real example?
<brett> ideal world: never
<brett> but we can't leave projects at the mercy of their dependencies
<brett> so this would be how they get to have final say on their deps by overruling
<brett> rather than turning of transitivity altgoether
<brett> which would just be abused
<jdcasey> but if a dependency declares that it needs such-and-such, can we really second-guess that? it seems like it would lead to bugs/errors
<jason> did someone have a case where scope=none helped?
<brett> jason: you'd use that example on velocity, for example
<trygvis> the continuum plexus app
<brett> when you have a dep on velocity, and it gives you jdbc
<trygvis> and logkit+avalon framework api
<brett> jdcasey: we're not second guessing as such, its just making sure the project gets to have the final say
<brett> I think that's important, both on inclusion and version
<jason> yah, that's a crappy build
<brett> right, and there will -always- be crappy builds
<jdcasey> so if I define a dep with scope=none, it'll just blow it out of the deps list wherever it encounters it?
<brett> jdcasey: right
<brett> jason: I know, but...
<jason> i think some extra deps there might be better then scope=none
<jason> might it lead to some problem, where you did that and have another dep that needs jdbc
<brett> yes, it requires some intelligent use
<jason> just playing devil's advocate and what's better, some extra deps or something that potentially doesn't work
<brett> not when they get bundled
<brett> not when they don't exist in the repo
<jason> sorry, not following
<brett> not if they are not licensed appropriately (when we start checking that)
<brett> examples when it is not better to just get the deps
<jason> what do you mean "not when they get bundled"?
<brett> way I see it is that the project should always have the final say, but this is somewhat annoying enough that they will try and get the sourc eof the problem fixed
<brett> look at continuum's lib directory
<brett> I know they could be excluded in other ways
<brett> but that's the sort of thing we're trying to avoid
<jdcasey> brett: how will the scope=none stuff work when it's in a project that's brought in through transitive resolution?
<jason> yah, not ideal but i can live with that knowing that the velocity build will improve
<evenisse> If I use a dep that define a dep with scope = none, would I obtain it or is it excluded only for the project?
<brett> jason: you can. other's won't
<jason> i think the scope=none could be potentially cause problems
<brett> ok, alternatives to solve the same problem?
<brett> or is it solving the problem that causes problems?
<trygvis> a better application for managing the repository so stuff like this can be fixed in a better and more permanent way
<jdcasey> trygvis: the avg. user won't have access to such
<jason> i don't think getting some extra bits is a huge problem
<brett> ok, jason volunteers to answer that question on the user list every day ;)
<jason> i will get loggy to do it!
<jdcasey> brett: I think one thing that would help immensely is knowing what implies what...seeing the dependency graph, in other words
<trygvis> we'll just use bayesian filtering on the user list to point people to the FAQ
<trygvis> like we talked about jason
<brett> jdcasey: I agree
<jdcasey> a tool to do that, plus a local-only way to "fix" poms might do the trick
<brett> local fixes is problematic
<jdcasey> b/c of portability?
<trygvis> because it's local :)
<brett> rm -rf local_repo breaks stuff
<brett> repoman is better, but we really want to hear about it
<brett> everyone using scope=none emails the maven developers automatically :)
<jdcasey> you mean setup a trigger in maven to actually notify the list (or whatever) that this pom is broken? :)
<brett> ok, lets put it aside as problematic. I think we need a solution. I don't think it is acceptable to just drop in extra deps for the reasons given above. We'll come back after more thought
<jdcasey> brett: [repost] how does scope=none factor into poms resolved transitively?
<jason> it just gets taken out of the graph entirely
<jdcasey> ignored (I hope)?
<brett> not sure
<jdcasey> I mean resolved poms with a scope=none in it
<jdcasey> ok, just file it away then :)
<brett> depends where depMgmt is applied on transitive poms
<brett> will also be affected by versioning, so I'll just note it
<brett> I think "changes to dep mgmt" actually describes how it works now
<brett> except for the last point, which is just scope=none
<brett> changes to scope element can be skipped
<brett> everyone agree "resolve dependencies once only" is sane?
<jdcasey> I like that one, actually
<brett> ok, something more tricky
<brett> version ranges
<ben> oh that'll be fun
<jason> would be good to get one of the SAS guys here
<jason> i'm going to mail one of them, they probably have some ideas wrt versioning
<jason> having 300 builds
<jason> and if they would go for a standard versioning scheme
<brett> will gwaltney one of them? he was here 1/2 hr ago (I assume that was him)
<jdcasey> jstrachan might have a problem with it ;)
<jason> yah, he's one and jared
<jdcasey> active-* always causes problems on conversion with version names
<brett> what did you think of the versioning scheme?
<jdcasey> brett: what's the diff between '1.0' and '(1.0)'?
<jdcasey> okay, then
<brett> "Soft requirement, equivalent to 1.0 on its own
<brett> but [1.0] would be absolute
<brett> does the syntax suit people?
<brett> very math oriented :)
<brett> kind of ugly though
<jdcasey> nah, syntax is good for me
<jason> i think it's good, being math oriented we know it already works
<jason> set notation good
<jason> no one can argue with us
<brett> I bet someone has a version alpha(2) :)
<ben> not on windows they don't
<brett> ok, we'll stick with that for now
<brett> I like Jason's point too
<ben> oops I tell a lie, ( is valid on windows
<jdcasey> brett: will 1.0-RELEASE and 2.0-RELEASE be ok together?
<brett> no such thing
<jdcasey> you can't be maintaining two release cycles on a product?
<brett> I really prefer RELEASE be silent and behind the scenes
<brett> that's not what it was intended for...
<brett> we might need to re-evaluate, its mainly to support the plugins right now
<jdcasey> misunderstood the finer points of RELEASE, I guess
<brett> you're right, which is why I want it hidden
<brett> if that makes sense
<jdcasey> I think so
<brett> ok, are we happy with the technique of version comparison, and the version layout?
<jdcasey> I think those types of notations (like -SNAPSHOT) might be useful eventually, but for now, that's no biggie
<jdcasey> +1 here
<brett> shall we change our snapshots to have _build then?
<jason> i pinged walt to see if he wanted to show up
<brett> cool, we'll skip over the next bit until then as that's the important bit
<jdcasey> brett: I think so
<jason> as long as you stipulate deployment to one repository or else you won't be able to increment accurately
<jason> which we are doing now yes?
<jason> then that's cool
<brett> except under profiles, but that's intentional
<jdcasey> brett: we may have to move the artifact classifier too, just to make the whole thing more parseable
<brett> ie, you are incrementing different builds
<brett> jdcasey: not intending to have to parse it in the repo
<brett> you just need group ID and artifact ID
<jdcasey> actually, strike that anyways
<brett> that identifies the project, and that has all the info
<jdcasey> I meant for the repo mgt tools
<brett> yep, it can find the project.
<brett> next point was "preventing RELEASE dependencies", which I explained earlier. Let them specify an open ended version instead
<brett> so [2.0) is 2.0-RELEASE
<jdcasey> is resolvedVersion on hold pending further thought?
<brett> yeah, will do that last if SAS guys turn up
<brett> hmm, did I screw up set notation?
<brett> I think ) actually means "don't include this"
<jason> look on mathworld
<jdcasey> should've known it would be a wolfram site
<brett> ok, I'll check later
<brett> lot of good that maths degree did me
<brett> agree on parent versions?
<brett> "They do not need to incorporate ranges, and are generally treated as unversioned."
<jdcasey> brett: I suspect you're right on being wrong, tho... ;)
<jdcasey> what do you mean by unversioned?
<jdcasey> they can change, no?
<brett> yes, to the user they generally want the latest
<brett> but it gets resolved on deploy
<brett> they can specify it, but generally they don't want to
<brett> actually, that's part of the release mgmt discussion, and JAson and I spent a fair while on it before, so maybe we should come back to it
<brett> nearly to the end of this document, then might be time for another break :)
<brett> agree that version ranges wouldn't match a snapshot unless it is specified as a boundary?
<brett> ok, so all that is left is conflict resolution, and reproducibility
<jdcasey> I'm interested in what would create a warning message as being potentially workable, but outside the specified range(s)
<jdcasey> is that just any dep that is specified by group/artifactID, but where the version is incompatible?
<brett> that may be redundant, I had it written earlier
<brett> where is it again?
<jdcasey> maybe I made it up...can't find it
<jdcasey> disregard it, then
<brett> no, I said it
<brett> oh, its just in the original reqs
<brett> they are not part of the design, just the inspiration for it
<jdcasey> well, that would apply in "use-nearest" after a fashion
<brett> yeah, I think using soft requirements solved that
<jdcasey> if you use the nearest, and another transitive dep had that artifact with a different version spec, you might want to drop a warning
<jdcasey> ok, cool
<jdcasey> where is that strategy best defined?
<jdcasey> I might argue for the settings.xml, personally
<brett> needs to be consistent for different people building the project
<jdcasey> per-project might leave it open to changes injected by POMs of transitive deps, though...
<jdcasey> I agree on that
<brett> I don't think you'd honour what a transitive pom did
<jdcasey> guess you could specify to use the strategy in place, and ignore those specified by POMs of transitive deps
<brett> I'd like to start with just the one strategy so we get a feel for how it works, and see whether others are really needed
<brett> easier to give than to take away
<jdcasey> so start with the range-based?
<jason> just a note, if we're going to start using a graph proper then let's make a separate project of that tool because i want to be able to visually represent the graph with decorations on the nodes for all the items we're talking about
<jason> i want to use the exact same tool for visualization
<jason> jung.sf.net is very cool btw
<jason> i've settle on thta
<brett> ok, I'll take a look
<brett> do we think we'll take a crack at the reproducibility question?
<jdcasey> I think we have to, if we're going to get into compatibility ranges at all
<brett> yeah, I just mean now :)
<jdcasey> we have to have a concrete, reproducible build for release
<jason> if we're doing ranges are we implicity going to promote our versioning strategy
<brett> I think we only support our versionining strategy for ranges right now
<jason> i would like
<jason> i think if we have something reasonable, people will use it
<ben> could you not emit a file containing the decisions the automater took
<ben> then use that as an optional input at another point in the future
<jdcasey> ben: like a properties file or something of resolved versions, as a separate file?
<brett> jason: it encompasses most of the standards now, although changing build to _ differs from RPM
<jdcasey> the only problem with that might be portability
<brett> what do you mean jdcasey?
<jdcasey> was talking to ben :)
<brett> I know
<jdcasey> I mean the resolved-versions file would have to be deployed too...wouldn't it?
<jdcasey> unless we have a fully-resolved pom that we release to the repo
<jdcasey> that means a pom with all transitive deps included and pegged, no?
<brett> ok some objectives:
<jdcasey> actually that might be a really good thing to do...flatten it all out for a release
<brett> - capture all of the information used in a release
<brett> - retain the user's original specifications
<brett> - store at release point in SCM, and repo
<brett> the question I have is, when do we use the resolved versions, and when don't we?
<brett> hi jared, willgwal
<brett> brb, grabbing coffee. feel free to discuss
<jdcasey> I'd say that you're going to have a problem any time you're unable to depend on a release-quality POM...you don't know how it'll resolve those transitive deps
<brett> that's whats getting to me
<jdcasey> I think that on release you should build a release-quality POM that has all transitive deps included in it, with concrete versions for each. This POM shouldn't replace the pom.xml for the project, but should probably be put into version control for future reference
<brett> if you always take what the release used, you are losing the ability to make a smarter decision knowing all the factors it had available
<brett> if you always take the other factors, you might get a different result
<brett> jdcasey: sure, that's for building that project. I agree 100% on that
<jdcasey> but do you want to make a different decision? that's not reproducing the build, and could mean that debug/test builds fail to reproduce a bug
<Arnaud> +1 for jdcasey and POMs
<brett> here's another point: shouldn't it always make the same decision?
<brett> the released poms don't change
<jdcasey> aren't you assuming that all the transitive deps will be releases?
<brett> for a reproducible build, yes
<brett> the release plugin should just freak out at snapshots
<jdcasey> but as a user you don't always have that kind of control
<jason> releases or just non snapshots?
<jdcasey> jason: what's the difference?
<brett> is there a difference?
<ben> but lets say commons-foo depends on commons-clown [1.*]
<ben> and commons-clown 1.1 is out for your original build, but after than, commons-clown 1.2 came out
<evenisse> I'm sorry, but I must leave.
<jdcasey> what about the timestamped deploys that go with snapshots? can't you depend on those for a release?
<jason> jdcasey, that's what i mean
<jdcasey> yeah, just got that :)
<jason> it doesn't necessary have to be a proper release, just a timestamp
<jason> sorry, i have twins running around at my feet
<jdcasey> that's why we can't depend on all transitive deps to supply release-quality poms
<ben> jason: I have a newfound appreciation for the distraction that children cause :)
<jason> oh yah
<jason> that's why i borrow them :-)
<jason> i'm enough of a scatter brain
<brett> the problem with timestamped snapshots (at least now) is that the deployed poms may contain unresolved snapshots
<jdcasey> right, which means that when your project does a release, it has to resolve and then peg those snapshots for its own purposes
<jdcasey> so it has to collapse the transitive deps into its own pom
<jdcasey> it'd be nice to annotate them as to their origin
<jason> what if decided that anything deployed had to be resolved?
<brett> hold up
<brett> 3 questions I can answere here
<brett> actually, 2 I forgot what I had to say about clowns
<trygvis> they rock?
<brett> folding down transitivity:
<brett> this means you can't effectively use the ranges any more
<brett> unless you retain the spec to use later
<jdcasey> I see what you mean
<brett> [1.1) might resolve to the latest 2.0, but your project wants [1.2] exactly, no way to find out what was really wanted
<brett> jason: resolving snapshots on deploy means you don't get updates any more, basically same reason
<brett> you lose info
<brett> is that right?
<trygvis> this is something that all the linux distros have fixed, we should perhaps look more into what they've done
<jdcasey> maybe we need a built-with metadata file, then...more of an audit than anything else
<jason> what are you losing if the POM is fully resolved for deploy?
<brett> trygvis: have they?
<jdcasey> something that we could use as input
<jdcasey> jason: you lose the compatibility statement provided by that range
<jdcasey> for other projects to use
<jdcasey> range on the dep, I mean
<jdcasey> or snapshot, yeah
<brett> a SNAPSHOT depends on b SNAPSHOT (1), c depends on a SNAPSHOT and hence b
<brett> a gets published, is now (1) with dep on b (1)
<brett> new b (2) is published
<brett> c ->a (1) -> b(1)
<brett> but you want b(2)
<jason> yah but for a release how are you ending up with a snapshot depending on a snapshot?
<jdcasey> brett: I'd agree, with one thing: major version changes are meant to signal API incompatibility, no?
<jason> oh, the ranges
<jason> jdcasey, typically
<jdcasey> so maybe [1.0) would mean anything in the 1.x range, but not 2.0
<brett> I don't think so
<brett> they might be backwards compat
<jdcasey> [1.0,2.0) would be correct for that case, then?
<brett> interesting thought there
<brett> hard to retrofit onto something that used to declare [1.0)
<trygvis> on debian they would change the artifact id if they change stuff (radically)
<brett> typically a good practice, I think
<jdcasey> so it'd go from foo-1 to foo-2 artfactID?
<brett> but we can't rely on it. People break stuff between 1.1 and 1.2 :)
<trygvis> that's how they do it
<brett> though that was prettty ugly when you showed it before :)
<brett> ok, noting the retrofitting as an issue
<brett> actually, the dependency management would just be used to override the over-enthusiastic pom
<adc> Get an error running the m2 tutorial
<brett> adc: I forgot to update it I think
<brett> with the last release
<brett> which was dumb of me
<brett> will take a look later
<brett> ok, trygvis has a good point
<brett> I think the packagers take the approach of "one version to rule them all"
<trygvis> they do
<brett> and if they don't play together, its a different package
<brett> but that has come with its own set of problems, too
<jason> yes, that is one thing etrade specifically didn't like about gentoo
<jason> and why they didn't use it
<brett> right, we don't have to do that. we can have every version known to man around for use
<jdcasey> yeah, like try tracking an artifact across all the different artifactId's
<brett> trygvis: your point?
<trygvis> just showing how they specify deps
<brett> I'd like to use >= syntax, but not in XML :)
<trygvis> kaffe (>= 2:1.1.4.PRECVS8-2)
<jdcasey> I think we need to get the version-range syntax nailed down, but I think this stuff will work.
<trygvis> they also have some issues with they version stating although they're making it more flexible
<brett> the | is interesting, kind of like specification deps. I've ruled them out for this release though
<jdcasey> (having a syntax makes it a little easier to explore it all)
<trygvis> I like what brett proposed, yay mathematical synatx
<brett> as long as I get it right
<jdcasey> and we all learn it ;)
<jdcasey> I didn't have set theory in college
<brett> I did a bit of everything. A lot of pure maths is good for computer science, if useless in the real world :)
<jdcasey> yeah, been reading that in spare moments, trygvis ;)
<trygvis> I like the power set character
<trygvis> looks like a super fancy P
<jason> brett: that's not true. pure math is very handy
<jason> you wait till we play graph visualization :-)
<brett> yeah, it was tongue in cheek, mostly
<brett> ok, let me summarise where we are at...
<brett> I think we've agreed on everything in that doc
<brett> but with the following issues
<brett> - strategies are out for release 1, adding complexity to a problem not fully tested yet. likewise spec deps
<brett> we'll just use the range
<adc> brett, ping me when you're ready to help me out, tks.
<brett> adc: will do
<brett> - we need to get to the bottom of what gets resolved, how, and when it is used
<brett> did we all agree that resolving versions in the transitive poms outright would limit the ability to do the right selection later?
<brett> I'm thinking we operate in two modes, basically. use resolved versions (historical builds) and used everything
<brett> ok, I think we can keep this simple
<brett> I think we can ensure the reproducible build is self contained
<brett> that eliminates all of the guess work
<brett> I don't think that needs to go into the repo at all, only SCM/source distro
<jdcasey> might also put it in the jar itself, for reference
<brett> this is essentially what ben said at the start
<jdcasey> sort of a "here's how I was built" thing
<brett> jdcasey: right. We actually agreed to do that already last week, didn't get around to it. This takes it a step further.
<jason> so effecitvely a manifest for reproduction?
<brett> right. if you are using a source build, transitivity is turned off, and all the versions are resolved
<brett> not sure about the parent
<jason> so that pom would enumerate the entire list of artifacts required to build?
<brett> makes it immune from repo changes, changes in the app code, etc
<brett> jdcasey: good point :)
<brett> I can't see a use case for deploying fully populated poms to the repo - we went through that the other day and we agreed it was a bad idea
<jason> really with ranges there is no other way
<brett> and meanwhile, normal source code builds will always use the intelligent version
<brett> its nicely paralleled to the snapshots, so I like it
<jason> so for a source drop that POM could be placed in there so it would always work, what would we do with the SCM as far as the build being reproducable?
<brett> drop it in, tag, revert back and bump version.
<brett> no more commits than we have now
<brett> bigger diffs :)
<brett> I'm open to an alternate way of doing that though
<jason> so fully resolved pom gets checked in and revert back again
<brett> maybe a separate file
<brett> so, if we agree on that, outstanding issues are:
<jason> so tag with the full resolved POM and then the perform will build using that fully resolved POM?
<jason> that makes sense
<brett> ok, issues:
<brett> - how do we resolve the parent?
<brett> - how do we store the information? (in the pom, replacement pom, side by side file)
<brett> parent, I guess we also fold in.
<jason> fold in how?
<brett> I was thinking we needed to keep it dynamic because somethings can change (eg the company URL), but a release is a release
<jason> there won't be any ranges there and we could always retrieve it no?
<brett> fold in, I mean write out the model at the end of assembly, and keep that
<brett> the parent has deps that would affect the final pom
<jason> so fully resolve parent and all
<jdcasey> brett: I'd be +1 for making it totally flat for historical purposes...nothing to lose that way...
<brett> yeah, I think so
<brett> ok, last decision in this marathon topic... where do we store it
<brett> I'm in favour of a separate file actually
<jdcasey> yeah, me too
<jdcasey> seems cleaner
<jason> you mean the fully resolved POM?
<jason> the full-monty-pom.xml
<brett> well, the profiles doc already called for profiles.xml
<jdcasey> nice side effect is to be able to diff releases of that file for differences between releases
<jason> so in a source drop or tag it's just pom.xml but where is this uberpom going?
<brett> next to it
<brett> if it exists, use it instead (unless overridden by switch?)
<jason> the same file?
<brett> not sure what you mean?
<jdcasey> no, we'd leave the original pom.xml untouched, except for possibly incrementing the version (?)
<jason> if i just checkout a tag and go "m2 install"
<jdcasey> this is instead of replace-tag-revert,isn't it?
<jason> so the tag will have the original pom.xml and to reproduce the build the release-pom.xml would be used
<jdcasey> that's what I think
<brett> yep, added bonus is that it would remove the need for release.properties
<brett> I think
<jason> ok, but i think everyone will check out the tag and just "m2 install"
<jason> which won't use what we intend
<jason> ok, i'm confused
<jason> the pom.xml in a tag will be full resolved on or not?
<brett> I was thinking not, so it didn't have such a wild change history
<jason> that's fine, we could probably have the cli look for the release pom so people don't inadvertently build with the wrong pom
<jason> they will naturally just check out and go "m2 install"
<jason> or at least warn them there is a release pom there that can be used
<brett> do we want to keep the release pom there, or only have it on the tags?
<jdcasey> I'd think only on the tags
<jdcasey> since it's only valid for that release
<jason> i think only in the tags and source balls
<jdcasey> that file will go stale as soon as new development starts
<brett> ok, so release process is:
<brett> - set release version in pom.xml, write out release-pom.xml
<brett> - tag
<brett> - delete release-pom.xml, bump version in pom.xml